(nifi-site) branch main updated: exclude PMC from Committers list (#83)
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-site.git
The following commit(s) were added to refs/heads/main by this push:
new 9d6b27e1 exclude PMC from Committers list (#83)
9d6b27e1 is described below
commit 9d6b27e121aed433c57c674a2ce1d1b0c89ab8fb
Author: Márton Szász
AuthorDate: Thu Feb 22 19:30:08 2024 +0100
exclude PMC from Committers list (#83)
---
content/community/_index.md | 2 +-
themes/nifi/layouts/shortcodes/project-members.html | 8 +++-
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/content/community/_index.md b/content/community/_index.md
index 1a061298..0b6d68dc 100644
--- a/content/community/_index.md
+++ b/content/community/_index.md
@@ -31,7 +31,7 @@ members on a vote open for at least 72 hours. A Committer is
considered inactive
can be revoked by a unanimous vote of all the active PMC members, except the
member in question if they are a PMC
member.
-{{< project-members project="nifi" >}}
+{{< project-members project="nifi" exclude-project="nifi-pmc" >}}
## Contributors
diff --git a/themes/nifi/layouts/shortcodes/project-members.html
b/themes/nifi/layouts/shortcodes/project-members.html
index bb5c7cc3..33b4ad59 100644
--- a/themes/nifi/layouts/shortcodes/project-members.html
+++ b/themes/nifi/layouts/shortcodes/project-members.html
@@ -6,6 +6,13 @@
{{- $project = . -}}
{{- end -}}
+{{ $members := index $groups $project }}
+
+{{- with .Get "exclude-project" -}}
+ {{ $excluded := index $groups . }}
+ {{ $members = complement $excluded $members }}
+{{- end -}}
+
@@ -14,7 +21,6 @@
- {{ $members := index $groups $project }}
{{- range $username := $members -}}
{{ $name := index $people $username }}
(nifi-minifi-cpp) branch main updated: MINIFICPP-2265 Add AttributeRollingWindow and EL nextInt
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 4a4113713 MINIFICPP-2265 Add AttributeRollingWindow and EL nextInt
4a4113713 is described below
commit 4a41137135b318c9b11f3ed94bce4f1e5dfef765
Author: Marton Szasz
AuthorDate: Mon Nov 20 22:10:53 2023 +0100
MINIFICPP-2265 Add AttributeRollingWindow and EL nextInt
Signed-off-by: Marton Szasz
Signed-off-by: Arpad Boda
This closes #1703
---
EXPRESSIONS.md | 25 -
PROCESSORS.md | 39 +++
extensions/expression-language/Expression.cpp | 8 ++
extensions/standard-processors/CMakeLists.txt | 3 +-
extensions/standard-processors/RollingWindow.h | 72
.../processors/AttributeRollingWindow.cpp | 123 +
.../processors/AttributeRollingWindow.h| 117
.../tests/unit/AttributeRollingWindowTests.cpp | 101 +
.../tests/unit/RollingWindowTests.cpp | 108 ++
libminifi/include/core/AbstractProcessor.h | 58 ++
libminifi/src/core/extension/ExtensionManager.cpp | 7 +-
libminifi/test/unit/AbstractProcessorTest.cpp | 114 +++
12 files changed, 769 insertions(+), 6 deletions(-)
diff --git a/EXPRESSIONS.md b/EXPRESSIONS.md
index 00d6306a0..c61eaecd2 100644
--- a/EXPRESSIONS.md
+++ b/EXPRESSIONS.md
@@ -225,6 +225,7 @@ token, filename.
- [`UUID`](#uuid)
- [`literal`](#literal)
- [`reverseDnsLookup`](#reversednslookup)
+- [`nextInt`](#nextInt)
### Evaluating Multiple Attributes
@@ -258,7 +259,6 @@ token, filename.
### Subjectless Functions
-- `nextInt`
- `getStateValue`
## Unsupported Features
@@ -950,7 +950,7 @@ filename.txt".
**Arguments**: No arguments
-**Return Type**: String
+**Return Type**: Number
**Examples**: If the attribute "filename" has a value of "a brand new
filename.txt" and the attribute "hello" does not exist, then the Expression
@@ -1289,7 +1289,7 @@ found at the beginning of the Subject, the value returned
will be `0`, not `1`.
| - | - |
| value | The value to search for in the Subject |
-**Return Type**: Boolean
+**Return Type**: Number
**Examples**:
@@ -1320,7 +1320,7 @@ found at the beginning of the Subject, the value returned
will be `0`, not `1`.
| - | - |
| value | The value to search for in the Subject |
-**Return Type**: Boolean
+**Return Type**: Number
**Examples**:
@@ -1615,6 +1615,23 @@ more than 3 attributes whose names begin with the letter
a.
| `${reverseDnsLookup('::1')}` | `localhost` |
| `${reverseDnsLookup('2001:4860:4860::'), 100}` | `dns.google` |
+### nextInt
+
+**Description**: Returns a one-up value (starting at 0) and increasing over the
+lifetime of the running instance of MiNiFi. This value is not persisted across
restarts.
+This counter is shared across all MiNiFi components, so calling this function
multiple
+times from one Processor will not guarantee sequential values within the
context of a
+particular Processor.
+
+**Subject Type**: No subject
+
+**Arguments**: No arguments
+
+**Return Type**: Number
+
+**Examples**: If the previous value returned by nextInt was 5, the Expression
+${nextInt():divide(2)} obtains the next available integer (6) and divides the
result
+by 2, returning a value of 3.
## Evaluating Multiple Attributes
diff --git a/PROCESSORS.md b/PROCESSORS.md
index 6f1d93f1b..ee9c5bbd1 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -17,6 +17,7 @@ limitations under the License.
- [AppendHostInfo](#AppendHostInfo)
- [ApplyTemplate](#ApplyTemplate)
+- [AttributeRollingWindow](#AttributeRollingWindow)
- [AttributesToJSON](#AttributesToJSON)
- [BinFiles](#BinFiles)
- [CapturePacket](#CapturePacket)
@@ -147,6 +148,44 @@ In the list below, the names of required properties appear
in bold. Any other pr
| success | success operational on the flow record |
+## AttributeRollingWindow
+
+### Description
+
+Track a Rolling Window based on evaluating an Expression Language expression
on each FlowFile. Each FlowFile will be emitted with the count of FlowFiles and
total aggregate value of values processed in the current window.
+
+### Properties
+
+In the list below, the names of required properties appear in bold. Any other
properties (not in bold) are considered optional. The table also indicates any
default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value
(nifi-minifi-cpp) branch main updated: MINIFICPP-2271 update AWS SDK and regions, port patches
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new de5baead4 MINIFICPP-2271 update AWS SDK and regions, port patches
de5baead4 is described below
commit de5baead47490c1444dfa2b1d4b905f47ddac62a
Author: Marton Szasz
AuthorDate: Thu Dec 14 15:35:28 2023 +0100
MINIFICPP-2271 update AWS SDK and regions, port patches
Signed-off-by: Marton Szasz
Signed-off-by: Arpad Boda
This closes #1705
---
cmake/BundledAwsSdkCpp.cmake | 14 +--
extensions/aws/processors/S3Processor.h| 17 +--
thirdparty/aws-sdk-cpp/bundle-openssl.patch| 57 ++
.../aws-sdk-cpp/c++20-compilation-fixes.patch | 12 ---
thirdparty/aws-sdk-cpp/dll-export-injection.patch | 118 ++---
thirdparty/aws-sdk-cpp/openssl3-fix.patch | 54 --
6 files changed, 129 insertions(+), 143 deletions(-)
diff --git a/cmake/BundledAwsSdkCpp.cmake b/cmake/BundledAwsSdkCpp.cmake
index ccdcca6ab..74c90bf6a 100644
--- a/cmake/BundledAwsSdkCpp.cmake
+++ b/cmake/BundledAwsSdkCpp.cmake
@@ -16,17 +16,13 @@
# under the License.
function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
-set(PATCH_FILE1
"${SOURCE_DIR}/thirdparty/aws-sdk-cpp/c++20-compilation-fixes.patch")
-set(PATCH_FILE2
"${SOURCE_DIR}/thirdparty/aws-sdk-cpp/dll-export-injection.patch")
-set(PATCH_FILE3 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/shutdown-fix.patch")
-set(PATCH_FILE4
"${SOURCE_DIR}/thirdparty/aws-sdk-cpp/bundle-openssl.patch")
-set(PATCH_FILE5 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/openssl3-fix.patch")
+set(PATCH_FILE1
"${SOURCE_DIR}/thirdparty/aws-sdk-cpp/dll-export-injection.patch")
+set(PATCH_FILE2 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/shutdown-fix.patch")
+set(PATCH_FILE3
"${SOURCE_DIR}/thirdparty/aws-sdk-cpp/bundle-openssl.patch")
set(AWS_SDK_CPP_PATCH_COMMAND ${Bash_EXECUTABLE} -c "set -x &&\
(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i
\"${PATCH_FILE1}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE1}\") &&\
(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i
\"${PATCH_FILE2}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE2}\") &&\
-(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i
\"${PATCH_FILE3}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE3}\") &&\
-(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i
\"${PATCH_FILE4}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE4}\") &&\
-(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i
\"${PATCH_FILE5}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE5}\") ")
+(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i
\"${PATCH_FILE3}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE3}\") ")
if (WIN32)
set(LIBDIR "lib")
@@ -85,7 +81,7 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
ExternalProject_Add(
aws-sdk-cpp-external
GIT_REPOSITORY "https://github.com/aws/aws-sdk-cpp.git;
-GIT_TAG "1.10.48"
+GIT_TAG "1.11.219"
UPDATE_COMMAND git submodule update --init --recursive
SOURCE_DIR "${BINARY_DIR}/thirdparty/aws-sdk-cpp-src"
INSTALL_DIR "${BINARY_DIR}/thirdparty/libaws-install"
diff --git a/extensions/aws/processors/S3Processor.h
b/extensions/aws/processors/S3Processor.h
index 85e186eb2..19ff7b61e 100644
--- a/extensions/aws/processors/S3Processor.h
+++ b/extensions/aws/processors/S3Processor.h
@@ -48,6 +48,7 @@ inline constexpr std::string_view AP_NORTHEAST_1 =
"ap-northeast-1";
inline constexpr std::string_view AP_NORTHEAST_2 = "ap-northeast-2";
inline constexpr std::string_view AP_NORTHEAST_3 = "ap-northeast-3";
inline constexpr std::string_view AP_SOUTH_1 = "ap-south-1";
+inline constexpr std::string_view AP_SOUTH_2 = "ap-south-2";
inline constexpr std::string_view AP_SOUTHEAST_1 = "ap-southeast-1";
inline constexpr std::string_view AP_SOUTHEAST_2 = "ap-southeast-2";
inline constexpr std::string_view AP_SOUTHEAST_3 = "ap-southeast-3";
@@ -55,8 +56,10 @@ inline constexpr std::string_view CA_CENTRAL_1 =
"ca-central-1";
inline constexpr std::string_view CN_NORTH_1 = "cn-north-1";
inline constexpr std::string_view CN_NORTHWEST_1 = "cn-nor
(nifi-minifi-cpp) branch main updated: MINIFICPP-2274 Ship Markdown docs in binary packages
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 52ab84603 MINIFICPP-2274 Ship Markdown docs in binary packages
52ab84603 is described below
commit 52ab84603045c349fa0f2e232e06a3d33754b705
Author: Marton Szasz
AuthorDate: Wed Dec 13 17:10:26 2023 +0100
MINIFICPP-2274 Ship Markdown docs in binary packages
Signed-off-by: Marton Szasz
Signed-off-by: Arpad Boda
This closes #1709
---
CMakeLists.txt | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 5ed19b7a7..d73684e7b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -422,8 +422,11 @@ if(WIN32)
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/conf/" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/conf/")
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}")
-file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/README.md" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}")
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/NOTICE" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}")
+file(GLOB markdown_docs "${CMAKE_CURRENT_SOURCE_DIR}/*.md")
+foreach(mddocfile ${markdown_docs})
+file(COPY "${mddocfile}" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}")
+endforeach()
# Determine the path of the VC Redistributable Merge Modules
if (DEFINED ENV{VCToolsRedistDir})
@@ -531,7 +534,11 @@ if (NOT WIN32)
COMPONENT bin)
endif()
-install(FILES LICENSE README.md NOTICE
+install(FILES LICENSE NOTICE
+DESTINATION .
+COMPONENT bin)
+file(GLOB markdown_docs "*.md")
+install(FILES ${markdown_docs}
DESTINATION .
COMPONENT bin)
(nifi) branch support/nifi-1.x updated (0a67620c0c -> 44de1f2d67)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git from 0a67620c0c NIFI-12358 Corrected syntax for Java 8 add 44de1f2d67 NIFI-12377 - Deprecate support for Jython in NiFi 1.x line No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/nifi/script/ScriptingComponentHelper.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[nifi] branch support/nifi-1.x updated: NIFI-12095 This closes #7762. Increase default Xmx to 1g
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 5c1ca14d33 NIFI-12095 This closes #7762. Increase default Xmx to 1g 5c1ca14d33 is described below commit 5c1ca14d330c928703326e46a1e24eaa6ae9022b Author: Nandor Soma Abonyi AuthorDate: Wed Sep 20 11:06:38 2023 +0200 NIFI-12095 This closes #7762. Increase default Xmx to 1g Signed-off-by: Joseph Witt --- .../nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml index 7139d02838..4f55ff9040 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml @@ -25,8 +25,8 @@ Holds common resources used to build installers -512m -512m +1g +1g true
[nifi] branch main updated: NIFI-12114 Create separate test instance for python extensions
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new 4b0c8bf6af NIFI-12114 Create separate test instance for python
extensions
4b0c8bf6af is described below
commit 4b0c8bf6af6eee30b0c9fc851a82e0fab45d35ab
Author: Nandor Soma Abonyi
AuthorDate: Fri Sep 22 16:47:27 2023 +0200
NIFI-12114 Create separate test instance for python extensions
Signed-off-by: Arpad Boda
This closes #7780
---
.github/workflows/system-tests.yml | 1 +
nifi-system-tests/nifi-system-test-suite/pom.xml | 24 +++
.../org/apache/nifi/tests/system/NiFiSystemIT.java | 16 +-
.../tests/system/python/PythonProcessorIT.java | 5 +-
.../resources/conf/clustered/node1/nifi.properties | 2 +-
.../resources/conf/clustered/node2/nifi.properties | 2 +-
.../test/resources/conf/default/nifi.properties| 2 +-
.../test/resources/conf/pythonic/bootstrap.conf| 33
.../src/test/resources/conf/pythonic/logback.xml | 214 +
.../conf/{default => pythonic}/nifi.properties | 34 ++--
.../resources/conf/pythonic/state-management.xml | 32 +++
.../resources/conf/pythonic/zookeeper.properties | 45 +
12 files changed, 386 insertions(+), 24 deletions(-)
diff --git a/.github/workflows/system-tests.yml
b/.github/workflows/system-tests.yml
index d1e81806f9..f2c4c41748 100644
--- a/.github/workflows/system-tests.yml
+++ b/.github/workflows/system-tests.yml
@@ -56,6 +56,7 @@ env:
package
verify
-P integration-tests
+-D include-python-integration-tests=true
MAVEN_PROJECTS: >-
-pl :nifi-python-framework
-pl :nifi-python-extension-api
diff --git a/nifi-system-tests/nifi-system-test-suite/pom.xml
b/nifi-system-tests/nifi-system-test-suite/pom.xml
index 92906f1c89..f173a353af 100644
--- a/nifi-system-tests/nifi-system-test-suite/pom.xml
+++ b/nifi-system-tests/nifi-system-test-suite/pom.xml
@@ -24,6 +24,30 @@
nifi-system-test-suite
jar
+
+
+include-python-integration-tests
+
+
+include-python-integration-tests
+!true
+
+
+
+
+
+maven-failsafe-plugin
+
+
+PythonProcessorIT.java
+
+
+
+
+
+
+
+
diff --git
a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java
b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java
index f0ef3741e2..b927fae922 100644
---
a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java
+++
b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java
@@ -201,7 +201,7 @@ public abstract class NiFiSystemIT implements
NiFiInstanceProvider {
.bootstrapConfig("src/test/resources/conf/default/bootstrap.conf")
.instanceDirectory("target/standalone-instance")
.overrideNifiProperties(getNifiPropertiesOverrides())
-.unpackPythonExtensions(isUnpackPythonExtensions())
+.unpackPythonExtensions(false)
.build());
}
@@ -211,6 +211,16 @@ public abstract class NiFiSystemIT implements
NiFiInstanceProvider {
"src/test/resources/conf/clustered/node2/bootstrap.conf");
}
+public NiFiInstanceFactory createPythonicInstanceFactory() {
+return new SpawnedStandaloneNiFiInstanceFactory(
+new InstanceConfiguration.Builder()
+
.bootstrapConfig("src/test/resources/conf/pythonic/bootstrap.conf")
+.instanceDirectory("target/pythonic-instance")
+.overrideNifiProperties(getNifiPropertiesOverrides())
+.unpackPythonExtensions(true)
+.build());
+}
+
protected String getTestName() {
return testInfo.getDisplayName();
}
@@ -545,10 +555,6 @@ public abstract class NiFiSystemIT implements
NiFiInstanceProvider {
return node2Dto;
}
-protected boolean isUnpackPythonExtensions() {
-return false;
-}
-
/**
* Disconnects a node from the cluster
* @param nodeIndex the 1-based index of the node
diff --git
a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/python/PythonProcessorIT.java
b/nifi-system-tests/nifi-system-test
[nifi] branch support/nifi-1.x updated: NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch support/nifi-1.x
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/support/nifi-1.x by this push:
new af8128bb5c NIFI-12010: Handle auto-commit and commit based on driver
capabilities in SQL components
af8128bb5c is described below
commit af8128bb5c26e418d31ec38741648dd87607c33b
Author: Matt Burgess
AuthorDate: Wed Aug 30 23:30:22 2023 -0400
NIFI-12010: Handle auto-commit and commit based on driver capabilities in
SQL components
Signed-off-by: Arpad Boda
This closes #7663
---
.../org/apache/nifi/admin/AuditDataSourceFactoryBean.java | 10 +-
.../transaction/impl/StandardTransactionBuilder.java| 10 +-
.../apache/nifi/processors/groovyx/ExecuteGroovyScript.java | 13 +++--
.../java/org/apache/nifi/hive/metastore/ScriptRunner.java | 5 -
.../apache/nifi/processors/standard/AbstractExecuteSQL.java | 12 +++-
.../java/org/apache/nifi/processors/standard/PutSQL.java| 7 ++-
.../org/apache/nifi/processors/standard/TestExecuteSQL.java | 3 ++-
.../org/apache/nifi/record/sink/db/DatabaseRecordSink.java | 9 -
8 files changed, 60 insertions(+), 9 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
index 11d7c30e64..4bac1716c0 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
@@ -28,6 +28,7 @@ import java.io.File;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
+import java.sql.SQLFeatureNotSupportedException;
import java.sql.Statement;
/**
@@ -151,7 +152,14 @@ public class AuditDataSourceFactoryBean implements
FactoryBean {
try {
// get a connection
connection = connectionPool.getConnection();
-connection.setAutoCommit(false);
+final boolean isAutoCommit = connection.getAutoCommit();
+if (isAutoCommit) {
+try {
+connection.setAutoCommit(false);
+} catch (SQLFeatureNotSupportedException sfnse) {
+logger.debug("setAutoCommit(false) not supported by
this driver");
+}
+}
// create a statement for initializing the database
statement = connection.createStatement();
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
index 7d4a1fcc44..e4b12180c2 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
@@ -18,6 +18,7 @@ package org.apache.nifi.admin.service.transaction.impl;
import java.sql.Connection;
import java.sql.SQLException;
+import java.sql.SQLFeatureNotSupportedException;
import javax.sql.DataSource;
import org.apache.nifi.admin.service.transaction.Transaction;
import org.apache.nifi.admin.service.transaction.TransactionBuilder;
@@ -35,7 +36,14 @@ public class StandardTransactionBuilder implements
TransactionBuilder {
try {
// get a new connection
Connection connection = dataSource.getConnection();
-connection.setAutoCommit(false);
+final boolean isAutoCommit = connection.getAutoCommit();
+if (isAutoCommit) {
+try {
+connection.setAutoCommit(false);
+} catch (SQLFeatureNotSupportedException sfnse) {
+throw new TransactionException("setAutoCommit(false) not
supported by this driver");
+}
+}
// create a new transaction
return new StandardTransaction(connection);
diff --git
a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java
b/nifi-nar-bundles/nifi-groovyx
[nifi] branch main updated: NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new 932cfe22a3 NIFI-12010: Handle auto-commit and commit based on driver
capabilities in SQL components
932cfe22a3 is described below
commit 932cfe22a379ea83e11fe283a3f789115cce1607
Author: Matt Burgess
AuthorDate: Wed Aug 30 23:30:22 2023 -0400
NIFI-12010: Handle auto-commit and commit based on driver capabilities in
SQL components
Signed-off-by: Arpad Boda
This closes #7663
---
.../org/apache/nifi/admin/AuditDataSourceFactoryBean.java | 10 +-
.../transaction/impl/StandardTransactionBuilder.java| 10 +-
.../apache/nifi/processors/groovyx/ExecuteGroovyScript.java | 13 +++--
.../java/org/apache/nifi/hive/metastore/ScriptRunner.java | 5 -
.../apache/nifi/processors/standard/AbstractExecuteSQL.java | 12 +++-
.../java/org/apache/nifi/processors/standard/PutSQL.java| 7 ++-
.../org/apache/nifi/processors/standard/TestExecuteSQL.java | 3 ++-
.../org/apache/nifi/record/sink/db/DatabaseRecordSink.java | 9 -
8 files changed, 60 insertions(+), 9 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
index 22165d2b68..040a207dcb 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java
@@ -27,6 +27,7 @@ import java.io.File;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
+import java.sql.SQLFeatureNotSupportedException;
import java.sql.Statement;
/**
@@ -146,7 +147,14 @@ public class AuditDataSourceFactoryBean implements
FactoryBean {
try {
// get a connection
connection = connectionPool.getConnection();
-connection.setAutoCommit(false);
+final boolean isAutoCommit = connection.getAutoCommit();
+if (isAutoCommit) {
+try {
+connection.setAutoCommit(false);
+} catch (SQLFeatureNotSupportedException sfnse) {
+logger.debug("setAutoCommit(false) not supported by
this driver");
+}
+}
// create a statement for initializing the database
statement = connection.createStatement();
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
index 7d4a1fcc44..e4b12180c2 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java
@@ -18,6 +18,7 @@ package org.apache.nifi.admin.service.transaction.impl;
import java.sql.Connection;
import java.sql.SQLException;
+import java.sql.SQLFeatureNotSupportedException;
import javax.sql.DataSource;
import org.apache.nifi.admin.service.transaction.Transaction;
import org.apache.nifi.admin.service.transaction.TransactionBuilder;
@@ -35,7 +36,14 @@ public class StandardTransactionBuilder implements
TransactionBuilder {
try {
// get a new connection
Connection connection = dataSource.getConnection();
-connection.setAutoCommit(false);
+final boolean isAutoCommit = connection.getAutoCommit();
+if (isAutoCommit) {
+try {
+connection.setAutoCommit(false);
+} catch (SQLFeatureNotSupportedException sfnse) {
+throw new TransactionException("setAutoCommit(false) not
supported by this driver");
+}
+}
// create a new transaction
return new StandardTransaction(connection);
diff --git
a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java
b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-proc
[nifi-minifi-cpp] branch main updated: MINIFICPP-2010 Disable systemd on non-linux remove option from bootstrap UI
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new d6912551b MINIFICPP-2010 Disable systemd on non-linux remove option
from bootstrap UI
d6912551b is described below
commit d6912551b2e356db6ed55b7a0a58843e4b92fca8
Author: Marton Szasz
AuthorDate: Fri Dec 9 16:44:56 2022 +0100
MINIFICPP-2010 Disable systemd on non-linux
remove option from bootstrap UI
Signed-off-by: Arpad Boda
This closes #1470
---
bootstrap.sh | 10 +-
bstrp_functions.sh| 8 +---
extensions/systemd/CMakeLists.txt | 2 +-
3 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/bootstrap.sh b/bootstrap.sh
index 15a887ed5..cd4f9e8b3 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -163,6 +163,12 @@ export OS_MINOR
OS_REVISION=$(echo "$EVR" | cut -d. -f3)
export OS_REVISION
+if [[ "$OSTYPE" =~ .*linux.* ]]; then
+ LINUX=true
+else
+ LINUX=false
+fi
+
### Verify the compiler version
COMPILER_VERSION="0.0.0"
@@ -328,7 +334,9 @@ add_dependency OPC_ENABLED "mbedtls"
add_option AZURE_ENABLED ${TRUE} "ENABLE_AZURE"
-add_option SYSTEMD_ENABLED ${TRUE} "ENABLE_SYSTEMD"
+if $LINUX; then
+ add_option SYSTEMD_ENABLED ${TRUE} "ENABLE_SYSTEMD"
+fi
add_option NANOFI_ENABLED ${FALSE} "ENABLE_NANOFI"
set_dependency PYTHON_ENABLED NANOFI_ENABLED
diff --git a/bstrp_functions.sh b/bstrp_functions.sh
index 778c5dd51..7ce885e60 100755
--- a/bstrp_functions.sh
+++ b/bstrp_functions.sh
@@ -380,7 +380,9 @@ show_supported_features() {
echo "V. SQL Support..$(print_feature_status SQL_ENABLED)"
echo "W. Openwsman Support ...$(print_feature_status
OPENWSMAN_ENABLED)"
echo "X. Azure Support ...$(print_feature_status AZURE_ENABLED)"
- echo "Y. Systemd Support .$(print_feature_status
SYSTEMD_ENABLED)"
+ if $LINUX; then
+echo "Y. Systemd Support .$(print_feature_status
SYSTEMD_ENABLED)"
+ fi
echo "Z. NanoFi Support ..$(print_feature_status NANOFI_ENABLED)"
echo "AA. Splunk Support .$(print_feature_status SPLUNK_ENABLED)"
echo "AB. Kubernetes Support .$(print_feature_status
KUBERNETES_ENABLED)"
@@ -430,7 +432,7 @@ read_feature_options(){
ToggleFeature PYTHON_ENABLED
else
echo -e "${RED}Please ensure static linking is enabled for Python
Support...${NO_COLOR}" && sleep 2
- fi
+ fi
;;
n) ToggleFeature COAP_ENABLED ;;
o) ToggleFeature SFTP_ENABLED ;;
@@ -440,7 +442,7 @@ read_feature_options(){
v) ToggleFeature SQL_ENABLED ;;
w) ToggleFeature OPENWSMAN_ENABLED ;;
x) ToggleFeature AZURE_ENABLED ;;
-y) ToggleFeature SYSTEMD_ENABLED ;;
+y) if $LINUX; then ToggleFeature SYSTEMD_ENABLED; fi ;;
z) ToggleFeature NANOFI_ENABLED ;;
aa) ToggleFeature SPLUNK_ENABLED ;;
ab) ToggleFeature KUBERNETES_ENABLED ;;
diff --git a/extensions/systemd/CMakeLists.txt
b/extensions/systemd/CMakeLists.txt
index 9c1356569..9a77ee255 100644
--- a/extensions/systemd/CMakeLists.txt
+++ b/extensions/systemd/CMakeLists.txt
@@ -17,7 +17,7 @@
# under the License.
#
-if (NOT ENABLE_SYSTEMD)
+if (NOT ENABLE_SYSTEMD OR NOT CMAKE_SYSTEM_NAME STREQUAL "Linux")
return()
endif()
[nifi] 01/01: NIFI-10833 - Fix grammar error in ListenHTTP log msg
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch NIFI-10833
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit ffcfbc8dc73840c1e501cad94bbcf21e41a17008
Author: Arpad Boda
AuthorDate: Wed Nov 16 21:51:37 2022 +0100
NIFI-10833 - Fix grammar error in ListenHTTP log msg
---
.../src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
index a35b9edc73..14a4ff3145 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
@@ -562,7 +562,7 @@ public class ListenHTTP extends
AbstractSessionFactoryProcessor {
for (final String id : findOldFlowFileIds(context)) {
final FlowFileEntryTimeWrapper wrapper = flowFileMap.remove(id);
if (wrapper != null) {
-getLogger().warn("failed to received acknowledgment for HOLD
with ID {} sent by {}; rolling back session", id, wrapper.getClientIP());
+getLogger().warn("failed to receive acknowledgment for HOLD
with ID {} sent by {}; rolling back session", id, wrapper.getClientIP());
wrapper.session.rollback();
}
}
[nifi] branch NIFI-10833 created (now ffcfbc8dc7)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch NIFI-10833 in repository https://gitbox.apache.org/repos/asf/nifi.git at ffcfbc8dc7 NIFI-10833 - Fix grammar error in ListenHTTP log msg This branch includes the following new commits: new ffcfbc8dc7 NIFI-10833 - Fix grammar error in ListenHTTP log msg The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[nifi-minifi-cpp] 01/02: MINIFICPP-1963 improve exception logging
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 9cd6d868ffb1ca98ade9a0b3756d363985c300f6
Author: Marton Szasz
AuthorDate: Wed Oct 19 17:36:38 2022 +0200
MINIFICPP-1963 improve exception logging
more info in SchedulingAgent::onTrigger catch-all
and bump exception message log level from debug to warning
Signed-off-by: Arpad Boda
This closes #1436
---
extensions/libarchive/MergeContent.cpp | 5 ++-
libminifi/include/Exception.h | 16 +++-
libminifi/src/Exception.cpp| 41 ++
libminifi/src/FlowController.cpp | 5 ++-
libminifi/src/SchedulingAgent.cpp | 11 +++--
libminifi/src/c2/C2Agent.cpp | 4 +-
libminifi/src/c2/C2Client.cpp | 8 +++-
libminifi/src/core/ProcessGroup.cpp| 12 +++---
libminifi/src/core/ProcessSession.cpp | 48 +++---
libminifi/src/core/Processor.cpp | 7 ++--
libminifi/src/sitetosite/RawSocketProtocol.cpp | 4 +-
libminifi/src/sitetosite/SiteToSiteClient.cpp | 8 ++--
.../integration/StateTransactionalityTests.cpp | 6 +--
13 files changed, 111 insertions(+), 64 deletions(-)
diff --git a/extensions/libarchive/MergeContent.cpp
b/extensions/libarchive/MergeContent.cpp
index 4a2d69f7d..9af2194b7 100644
--- a/extensions/libarchive/MergeContent.cpp
+++ b/extensions/libarchive/MergeContent.cpp
@@ -253,8 +253,11 @@ bool MergeContent::processBin(core::ProcessContext
*context, core::ProcessSessio
try {
mergeBin->merge(context, session, bin->getFlowFile(), *serializer,
merge_flow);
session->putAttribute(merge_flow, core::SpecialFlowAttribute::MIME_TYPE,
mimeType);
+ } catch (const std::exception& ex) {
+logger_->log_error("Merge Content merge catch exception, type: %s, what:
%s", typeid(ex).name(), ex.what());
+return false;
} catch (...) {
-logger_->log_error("Merge Content merge catch exception");
+logger_->log_error("Merge Content merge catch exception, type: %s",
getCurrentExceptionTypeName());
return false;
}
session->putAttribute(merge_flow, BinFiles::FRAGMENT_COUNT_ATTRIBUTE,
std::to_string(bin->getSize()));
diff --git a/libminifi/include/Exception.h b/libminifi/include/Exception.h
index 634060d65..55fccefc4 100644
--- a/libminifi/include/Exception.h
+++ b/libminifi/include/Exception.h
@@ -17,8 +17,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#ifndef LIBMINIFI_INCLUDE_EXCEPTION_H_
-#define LIBMINIFI_INCLUDE_EXCEPTION_H_
+#pragma once
#include
#include
@@ -31,10 +30,7 @@
#include "utils/StringUtils.h"
-namespace org {
-namespace apache {
-namespace nifi {
-namespace minifi {
+namespace org::apache::nifi::minifi {
enum ExceptionType {
FILE_OPERATION_EXCEPTION = 0,
@@ -59,6 +55,8 @@ inline const char *ExceptionTypeToString(ExceptionType type) {
return nullptr;
}
+std::string getCurrentExceptionTypeName();
+
struct Exception : public std::runtime_error {
/*!
* Create a new exception
@@ -92,8 +90,4 @@ struct SystemErrorException : Exception {
std::error_condition error_condition_;
};
-} // namespace minifi
-} // namespace nifi
-} // namespace apache
-} // namespace org
-#endif // LIBMINIFI_INCLUDE_EXCEPTION_H_
+} // namespace org::apache::nifi::minifi
diff --git a/libminifi/src/Exception.cpp b/libminifi/src/Exception.cpp
new file mode 100644
index 0..c9e622077
--- /dev/null
+++ b/libminifi/src/Exception.cpp
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "Exception.h"
+
+#ifndef WIN32
+#include
+#endif // !WIN32
+
+#include
+
+namespace org::apache::nifi::minifi {
+std::string getCurrentExceptionTypeName() {
+#ifndef WIN32
+ const std::type_info* exception_type = abi::__cxa_current_exception_type();
+ if (exception_type) {
+
[nifi-minifi-cpp] branch main updated (b00db7438 -> c5d58e883)
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
from b00db7438 MINIFICPP-1952 Reset the callback when no longer needed
new 9cd6d868f MINIFICPP-1963 improve exception logging
new c5d58e883 MINIFICPP-1967 Add batch processing of lines in TailFile
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
PROCESSORS.md | 1 +
extensions/libarchive/MergeContent.cpp | 5 ++-
.../standard-processors/processors/TailFile.cpp| 14 ++-
.../standard-processors/processors/TailFile.h | 23 ---
.../tests/unit/TailFileTests.cpp | 30 ++
libminifi/include/Exception.h | 16 +++-
.../{include/SwapManager.h => src/Exception.cpp} | 38 -
libminifi/src/FlowController.cpp | 5 ++-
libminifi/src/SchedulingAgent.cpp | 11 +++--
libminifi/src/c2/C2Agent.cpp | 4 +-
libminifi/src/c2/C2Client.cpp | 8 +++-
libminifi/src/core/ProcessGroup.cpp| 12 +++---
libminifi/src/core/ProcessSession.cpp | 48 +++---
libminifi/src/core/Processor.cpp | 7 ++--
libminifi/src/sitetosite/RawSocketProtocol.cpp | 4 +-
libminifi/src/sitetosite/SiteToSiteClient.cpp | 8 ++--
.../integration/StateTransactionalityTests.cpp | 6 +--
17 files changed, 141 insertions(+), 99 deletions(-)
copy libminifi/{include/SwapManager.h => src/Exception.cpp} (63%)
[nifi-minifi-cpp] 02/02: MINIFICPP-1967 Add batch processing of lines in TailFile
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit c5d58e8836723ba232440d457c6ff1be811bd075
Author: Gabor Gyimesi
AuthorDate: Thu Oct 20 13:13:46 2022 +0200
MINIFICPP-1967 Add batch processing of lines in TailFile
Signed-off-by: Arpad Boda
This closes #1439
---
PROCESSORS.md | 1 +
.../standard-processors/processors/TailFile.cpp| 14 +-
.../standard-processors/processors/TailFile.h | 23 ++---
.../tests/unit/TailFileTests.cpp | 30 ++
4 files changed, 52 insertions(+), 16 deletions(-)
diff --git a/PROCESSORS.md b/PROCESSORS.md
index 8e7e6e328..8a0aac5de 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -2424,6 +2424,7 @@ In the list below, the names of required properties
appear in bold. Any other pr
| State File | TailFileState |
| Specifies the file that should be used for storing
state about what data has been ingested so that upon restart NiFi can resume
from where it left off
[...]
| tail-base-directory| |
| Base directory used to look for files to tail.
This property is required when using Multiple file mode. Can contain expression
language placeholders if Attribute Provider Service is set.**Supports
Expression Language: true**
[...]
| **tail-mode** | Single file | Single fileMultiple
file | Specifies the tail file mode. In 'Single file'
mode only a single file will be watched. In 'Multiple file' mode a regex may be
used. Note that in multiple file mode we will still continue to watch for
rollover on the initial set of watched files. The Regex used to locate multiple
files will be run during the schedule phrase. Note that if rotated files are
matched by the regex, thos [...]
+| **Batch Size** | 0 |
| Maximum number of flowfiles emitted in a single
trigger. If set to 0 all new content will be processed.
[...]
### Relationships
| Name| Description |
diff --git a/extensions/standard-processors/processors/TailFile.cpp
b/extensions/standard-processors/processors/TailFile.cpp
index 7a3f20194..abfd6784b 100644
--- a/extensions/standard-processors/processors/TailFile.cpp
+++ b/extensions/standard-processors/processors/TailFile.cpp
@@ -135,6 +135,13 @@ const core::Property TailFile::AttributeProviderService(
->asType()
->build());
+const core::Property TailFile::BatchSize(
+core::PropertyBuilder::createProperty("Batch Size")
+->withDescription("Maximum number of flowfiles emitted in a single
trigger. If set to 0 all new content will be processed.")
+->isRequired(true)
+->withDefaultValue(0)
+->build());
+
const core::Relationship TailFile::Success("success", "All files are routed to
success");
const char *TailFile::CURRENT_STR = "CURRENT.";
@@ -395,6 +402,11 @@ void TailFile::onSchedule(const
std::shared_ptr ,
context->getProperty(RollingFilenamePattern.getName(),
rolling_filename_pattern_glob);
rolling_filename_pattern_ =
utils::file::globToRegex(rolling_filename_pattern_glob);
initial_start_position_ =
InitialStartPositions{utils::parsePropertyWithAllowableValuesOrThrow(*context,
InitialStartPosition.getName(), InitialStartPositions::values())};
+
+ uint32_t batch_size = 0;
+ if (context->getProperty(BatchSize.getName(), batch_size) && batch_size !=
0) {
+batch_size_ = batch_size;
+ }
}
void TailFile::parseAttributeProviderServiceProperty(core::ProcessContext&
context) {
@@ -784,7 +796,7 @@ void TailFile::processSingleFile(const
std::shared_ptr
FileReaderCallback file_reader{full_file_name, state.position_, delim,
state.checksum_};
TailState state_copy{state};
-while (file_reader.hasMoreToRead()) {
+while (file_reader.hasMoreToRead() && (!batch_size_ |
[nifi] branch main updated: NIFI-10460 GetZendesk processor
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 0fa923159f NIFI-10460 GetZendesk processor 0fa923159f is described below commit 0fa923159f0d238e986a8dbe6edfc2acbe785f68 Author: Ferenc Kis AuthorDate: Tue Sep 6 17:10:18 2022 +0200 NIFI-10460 GetZendesk processor Signed-off-by: Arpad Boda This closes #6411 --- nifi-assembly/pom.xml | 6 + .../nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml | 46 +++ .../src/main/resources/META-INF/LICENSE| 204 .../src/main/resources/META-INF/NOTICE | 39 +++ .../nifi-zendesk-processors/pom.xml| 91 ++ .../apache/nifi/processors/zendesk/GetZendesk.java | 361 + .../zendesk/ZendeskAuthenticationType.java | 74 + .../processors/zendesk/ZendeskExportMethod.java| 85 + .../nifi/processors/zendesk/ZendeskResource.java | 97 ++ .../services/org.apache.nifi.processor.Processor | 15 + .../additionalDetails.html | 71 .../nifi/processors/zendesk/GetZendeskTest.java| 267 +++ .../zendesk/ZendeskAuthenticationTypeTest.java | 41 +++ nifi-nar-bundles/nifi-zendesk-bundle/pom.xml | 33 ++ nifi-nar-bundles/pom.xml | 1 + 15 files changed, 1431 insertions(+) diff --git a/nifi-assembly/pom.xml b/nifi-assembly/pom.xml index d923e3ae75..78f67698e8 100644 --- a/nifi-assembly/pom.xml +++ b/nifi-assembly/pom.xml @@ -892,6 +892,12 @@ language governing permissions and limitations under the License. --> 1.18.0-SNAPSHOT nar + +org.apache.nifi +nifi-zendesk-nar +1.18.0-SNAPSHOT +nar + org.apache.nifi nifi-dropbox-processors-nar diff --git a/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml new file mode 100644 index 00..b84a6fcdfc --- /dev/null +++ b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml @@ -0,0 +1,46 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-zendesk-bundle +1.18.0-SNAPSHOT + + +nifi-zendesk-nar +nar + +true +true + + + + +org.apache.nifi +nifi-zendesk-processors +1.18.0-SNAPSHOT + + +org.apache.nifi +nifi-standard-services-api-nar +1.18.0-SNAPSHOT +nar + + + + diff --git a/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/src/main/resources/META-INF/LICENSE b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/src/main/resources/META-INF/LICENSE new file mode 100644 index 00..de4b130f35 --- /dev/null +++ b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/src/main/resources/META-INF/LICENSE @@ -0,0 +1,204 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code,
[nifi] branch main updated (111c7ac0a4 -> 35fb66f50f)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git from 111c7ac0a4 NIFI-10444 FetchDropbox processor add 35fb66f50f NIFI-10403 Add processor supporting the new BigQuery Write API No new revisions were added by this update. Summary of changes: .../apache/nifi/serialization/record/Record.java | 2 +- .../nifi-gcp-bundle/nifi-gcp-processors/pom.xml| 14 + .../processors/gcp/bigquery/BigQueryUtils.java | 1 - .../nifi/processors/gcp/bigquery/PutBigQuery.java | 451 + .../processors/gcp/bigquery/PutBigQueryBatch.java | 6 +- .../gcp/bigquery/PutBigQueryStreaming.java | 20 +- .../processors/gcp/bigquery/proto/ProtoUtils.java | 66 +++ .../services/org.apache.nifi.processor.Processor | 1 + .../additionalDetails.html | 58 +++ .../processors/gcp/bigquery/AbstractBQTest.java| 26 +- .../gcp/bigquery/PutBigQueryBatchIT.java | 139 --- .../gcp/bigquery/PutBigQueryBatchTest.java | 39 +- .../processors/gcp/bigquery/PutBigQueryIT.java | 368 + .../gcp/bigquery/PutBigQueryStreamingIT.java | 306 -- .../processors/gcp/bigquery/PutBigQueryTest.java | 447 ...streaming-correct-data-with-date-formatted.json | 4 +- 16 files changed, 1439 insertions(+), 509 deletions(-) create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQuery.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/proto/ProtoUtils.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/resources/docs/org.apache.nifi.processors.gcp.bigquery.PutBigQuery/additionalDetails.html delete mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatchIT.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryIT.java delete mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryStreamingIT.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryTest.java
[nifi] branch main updated (4b0568e6fe -> 5303aadda3)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git from 4b0568e6fe NIFI-10242 Create QueryAirtableTable processor add 5303aadda3 NIFI-10500: Improved property ordering in MQTT processors No new revisions were added by this update. Summary of changes: .../apache/nifi/processors/mqtt/ConsumeMQTT.java | 77 -- .../apache/nifi/processors/mqtt/PublishMQTT.java | 50 -- .../mqtt/common/AbstractMQTTProcessor.java | 77 +++--- .../nifi/processors/mqtt/TestConsumeMQTT.java | 4 -- 4 files changed, 98 insertions(+), 110 deletions(-)
[nifi] branch main updated: NIFI-10395 Added Apache Xalan to banned dependencies
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new d9b3257e33 NIFI-10395 Added Apache Xalan to banned dependencies new 36ae680147 Merge pull request #6336 from exceptionfactory/NIFI-10395 d9b3257e33 is described below commit d9b3257e33da272370a625efcedfd7e46f003a53 Author: exceptionfactory AuthorDate: Thu Aug 25 10:50:15 2022 -0500 NIFI-10395 Added Apache Xalan to banned dependencies --- pom.xml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pom.xml b/pom.xml index a9f3a9de52..b560c7fc7c 100644 --- a/pom.xml +++ b/pom.xml @@ -859,6 +859,8 @@ org.apache.logging.log4j:log4j-core:* commons-logging:commons-logging:* + +xalan:xalan
[nifi] branch NIFI-10255 created (now d3ec916e0a)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch NIFI-10255 in repository https://gitbox.apache.org/repos/asf/nifi.git at d3ec916e0a NIFI-10255 - Decription of original relationship of ExecuteStreamCommand processor might be misleading This branch includes the following new commits: new d3ec916e0a NIFI-10255 - Decription of original relationship of ExecuteStreamCommand processor might be misleading The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[nifi] 01/01: NIFI-10255 - Decription of original relationship of ExecuteStreamCommand processor might be misleading
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch NIFI-10255
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit d3ec916e0ae98305d797752e36074d228e08d681
Author: Arpad Boda
AuthorDate: Thu Jul 21 01:08:52 2022 +0200
NIFI-10255 - Decription of original relationship of ExecuteStreamCommand
processor might be misleading
---
.../org/apache/nifi/processors/standard/ExecuteStreamCommand.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
index 2ebefcb1f1..00093bb972 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
@@ -173,7 +173,7 @@ public class ExecuteStreamCommand extends AbstractProcessor
{
public static final Relationship ORIGINAL_RELATIONSHIP = new
Relationship.Builder()
.name("original")
-.description("FlowFiles that were successfully processed.")
+.description("The original FlowFile will be routed. It will have
new attributes detailing the result of the script execution.")
.build();
public static final Relationship OUTPUT_STREAM_RELATIONSHIP = new
Relationship.Builder()
.name("output stream")
@@ -636,4 +636,4 @@ public class ExecuteStreamCommand extends AbstractProcessor
{
writerThread.setDaemon(true);
writerThread.start();
}
-}
\ No newline at end of file
+}
[nifi-minifi-cpp] branch main updated: MINIFICPP-1799 Fix PutGCSObject::Key property name
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 5ef439fd9 MINIFICPP-1799 Fix PutGCSObject::Key property name
5ef439fd9 is described below
commit 5ef439fd9f98cf4fe8199c513f18c4ca50e2fbeb
Author: Martin Zink
AuthorDate: Wed Apr 13 16:36:34 2022 +0200
MINIFICPP-1799 Fix PutGCSObject::Key property name
Signed-off-by: Arpad Boda
This closes #1303
---
extensions/gcp/processors/PutGCSObject.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/extensions/gcp/processors/PutGCSObject.cpp
b/extensions/gcp/processors/PutGCSObject.cpp
index 45d898d9b..9fadfedfe 100644
--- a/extensions/gcp/processors/PutGCSObject.cpp
+++ b/extensions/gcp/processors/PutGCSObject.cpp
@@ -46,7 +46,7 @@ const core::Property PutGCSObject::Bucket(
->build());
const core::Property PutGCSObject::Key(
-core::PropertyBuilder::createProperty("Name of the object.")
+core::PropertyBuilder::createProperty("Key")
->withDescription("Name of the object.")
->withDefaultValue("${filename}")
->supportsExpressionLanguage(true)
[nifi-minifi-cpp] branch main updated: MINIFICPP-1771: Reworked ListenSyslog
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 4a1d35b2c MINIFICPP-1771: Reworked ListenSyslog 4a1d35b2c is described below commit 4a1d35b2cc927be464031afbed32509013e0874d Author: Martin Zink AuthorDate: Thu Mar 10 09:27:58 2022 +0100 MINIFICPP-1771: Reworked ListenSyslog Signed-off-by: Arpad Boda This closes #1294 --- LICENSE| 28 ++ NOTICE | 1 + PROCESSORS.md | 57 ++- .../CMakeLists.txt => cmake/Asio.cmake | 27 +- .../integration/features/syslog_listener.feature | 26 ++ .../minifi/core/SingleNodeDockerCluster.py | 6 + .../minifi/core/SyslogTcpClientContainer.py| 23 + .../minifi/core/SyslogUdpClientContainer.py| 23 + .../integration/minifi/processors/ListenSyslog.py | 12 + docker/test/integration/steps/steps.py | 10 + extensions/gcp/tests/PutGCSObjectTests.cpp | 40 +- extensions/standard-processors/CMakeLists.txt | 3 +- .../processors/ListenSyslog.cpp| 511 + .../standard-processors/processors/ListenSyslog.h | 326 ++--- .../tests/unit/FetchFileTests.cpp | 51 +- .../tests/unit/ListenSyslogTests.cpp | 488 .../standard-processors/tests/unit/PutUDPTests.cpp | 10 +- libminifi/include/core/PropertyValidation.h| 14 +- ...ontroller.h => SingleProcessorTestController.h} | 14 +- libminifi/test/TestBase.cpp| 2 + 20 files changed, 1118 insertions(+), 554 deletions(-) diff --git a/LICENSE b/LICENSE index 8c3a9a21d..836615ad9 100644 --- a/LICENSE +++ b/LICENSE @@ -3181,6 +3181,34 @@ FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +-- + +This project bundles 'asio' under the Boost Software License 1.0 + +Boost Software License - Version 1.0 - August 17th, 2003 + +Permission is hereby granted, free of charge, to any person or organization +obtaining a copy of the software and accompanying documentation covered by +this license (the "Software") to use, reproduce, display, distribute, +execute, and transmit the Software, and to prepare derivative works of the +Software, and to permit third-parties to whom the Software is furnished to +do so, all subject to the following: + +The copyright notices in the Software and this entire statement, including +the above license grant, this restriction and the following disclaimer, +must be included in all copies of the Software, in whole or in part, and +all derivative works of the Software, unless such copies or derivative +works are solely in the form of machine-executable object code generated by +a source language processor. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT +SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE +FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + --- This project reuses test code from TartanLlama/expected from the Public Domain or under CC0 diff --git a/NOTICE b/NOTICE index ce26b3153..0e66d48fc 100644 --- a/NOTICE +++ b/NOTICE @@ -59,6 +59,7 @@ This software includes third party software subject to the following copyrights: - date (HowardHinnant/date) - notices below - range-v3 - Eric Niebler and other contributors - expected-lite - Copyright (C) 2016-2020 Martin Moene. +- asio - Copyright (c) 2003-2022 Christopher M. Kohlhoff (chris at kohlhoff dot com) - TartanLlama/expected - public domain, thanks to Sy Brand - libyaml - Copyright (c) 2006-2016 Kirill Simonov, Copyright (c) 2017-2020 Ingy döt Net - libwebsockets - Copyright (C) 2010 - 2020 Andy Green diff --git a/PROCESSORS.md b/PROCESSORS.md index 4290a8b25..dbe84270a 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -1,4 +1,4 @@ -
[nifi-minifi-cpp] branch main updated: MINIFICPP-1708 - Route flowfiles to Failure on decompression error
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 65fad0d02 MINIFICPP-1708 - Route flowfiles to Failure on decompression
error
65fad0d02 is described below
commit 65fad0d02d04463181470b21c8ea290ddd61c52a
Author: Adam Debreceni
AuthorDate: Mon Apr 11 11:15:07 2022 +0200
MINIFICPP-1708 - Route flowfiles to Failure on decompression error
Signed-off-by: Arpad Boda
This closes #1298
---
extensions/libarchive/CompressContent.cpp | 17 +-
.../test/archive-tests/CompressContentTests.cpp| 217 ++---
2 files changed, 116 insertions(+), 118 deletions(-)
diff --git a/extensions/libarchive/CompressContent.cpp
b/extensions/libarchive/CompressContent.cpp
index 421bc8d83..874eb72c0 100644
--- a/extensions/libarchive/CompressContent.cpp
+++ b/extensions/libarchive/CompressContent.cpp
@@ -175,7 +175,7 @@ void CompressContent::processFlowFile(const
std::shared_ptr& flo
fileExtension = search->second;
}
std::shared_ptr result = session->create(flowFile);
- bool success = false;
+ bool success = true;
if (encapsulateInTar_) {
std::function&, const
std::shared_ptr&)> transformer;
@@ -193,9 +193,15 @@ void CompressContent::processFlowFile(const
std::shared_ptr& flo
transformer = [&] (const std::shared_ptr& in, const
std::shared_ptr& out) -> int64_t {
io::ReadArchiveStreamImpl decompressor(in);
if (!decompressor.nextEntry()) {
- return -1;
+ success = false;
+ return 0; // prevents a session rollback
+}
+auto ret = internal::pipe(, out.get());
+if (ret < 0) {
+ success = false;
+ return 0; // prevents a session rollback
}
-return internal::pipe(, out.get());
+return ret;
};
}
session->write(result, FunctionOutputStreamCallback([&] (const auto& out) {
@@ -203,11 +209,6 @@ void CompressContent::processFlowFile(const
std::shared_ptr& flo
return transformer(in, out);
}));
}));
-// TODO(adebreceni): previous attempt to handle a malformed archive were
in vain
-//as the session->read threw anyway rolling back the flowfile, we
should correctly
-//forward a malformed archive to failure
-//https://issues.apache.org/jira/browse/MINIFICPP-1708
-success = true;
} else {
CompressContent::GzipWriteCallback callback(compressMode_, compressLevel_,
flowFile, session);
session->write(result, );
diff --git a/libminifi/test/archive-tests/CompressContentTests.cpp
b/libminifi/test/archive-tests/CompressContentTests.cpp
index bd1957797..6804e3a45 100644
--- a/libminifi/test/archive-tests/CompressContentTests.cpp
+++ b/libminifi/test/archive-tests/CompressContentTests.cpp
@@ -123,7 +123,7 @@ class CompressDecompressionTestController : public
TestController{
std::shared_ptr content_repo =
std::make_shared();
content_repo->initialize(std::make_shared());
-// connection from compress processor to log attribute
+// connection from compress processor to success
output = std::make_shared(repo, content_repo,
"Output");
output->addRelationship(core::Relationship("success", "compress successful
output"));
output->setSource(processor.get());
@@ -135,12 +135,47 @@ class CompressDecompressionTestController : public
TestController{
input->setDestinationUUID(processoruuid);
processor->addConnection(input.get());
-processor->setAutoTerminatedRelationships({{"failure", ""}});
+// connection from compress processor to failure
+failure_output = std::make_shared(repo, content_repo,
"FailureOutput");
+failure_output->addRelationship(core::Relationship("failure", "compress
failure output"));
+failure_output->setSource(processor.get());
+failure_output->setSourceUUID(processoruuid);
+processor->addConnection(failure_output.get());
processor->incrementActiveTasks();
processor->setScheduledState(core::ScheduledState::RUNNING);
context =
std::make_shared(std::make_shared(processor.get()),
nullptr, repo, repo, content_repo);
+helper_session = std::make_shared(context);
+ }
+
+ std::shared_ptr importFlowFile(const std::string&
content_path) {
+std::shared_ptr flow =
std::static_pointer_cast(helper_session->create());
+helper_session->import(content_path, flow, true, 0);
+helper_session->flushContent();
+input->put(flow);
+return flow;
+ }
+
+ template
+ std::shared_ptr importFlowFileFrom(T&& source) {
+std::shared_ptr flow =
std::static_pointer_cas
svn commit: r53789 - /release/nifi/KEYS
Author: aboda Date: Wed Apr 13 11:20:17 2022 New Revision: 53789 Log: Added new public key of Arpad Boda Modified: release/nifi/KEYS Modified: release/nifi/KEYS == --- release/nifi/KEYS (original) +++ release/nifi/KEYS Wed Apr 13 11:20:17 2022 @@ -1855,3 +1855,92 @@ RUN7+JEPRvIjpFFMOIlMB2Mkiwlo1Be/67zuAS0d TdbG6AMfmlsrDh8= =XV51 -END PGP PUBLIC KEY BLOCK- +pub rsa2048 2019-10-24 [SC] [expired: 2021-10-23] + EE78BB8A82A7851477AAAD0A390C1B5ADE978835 +uid [ expired] Arpad Boda +sig 3390C1B5ADE978835 2019-10-24 Arpad Boda + +pub rsa4096 2022-04-13 [SC] [expires: 2024-04-12] + D6C8A7149D47EF7AA08B35D1065668F2A58F097F +uid [ultimate] Arpad Boda +sig 3065668F2A58F097F 2022-04-13 Arpad Boda +sub rsa4096 2022-04-13 [E] [expires: 2024-04-12] +sig 065668F2A58F097F 2022-04-13 Arpad Boda + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQENBF2xedcBCADhvZxpFnZ37r3LuC4ZBNJu5OX4ORkXSWOgo7JwGb0pDGWrQlcq +ZU57fjeGvyciLPF7ICAOKDGCF2YYrRfCG2IhI4S6B+pYpIs9H9XnfoVLUOYgCQAl +VsugMlMXAnLb5MNNFEV6XLQooqvFkTtsbRcEsES6DogBeYlMoF2koJkY5W/Evpcs +96nwVH9cTtdahOx0Er3U5vNoflSsWqMXQPK+IRviaJP9f3e+pLnV2179e2TX59PV +bBwXKGwXAoKq19TFwmLAUjuHDKccynwwWenEzQ5CUiUTwtwxzp85hiVZkG07WWST +aOyGYIUnYUGRQiis4/ElXoBpS6mmld4zlbV7ABEBAAG0HUFycGFkIEJvZGEgPGFi +b2RhQGFwYWNoZS5vcmc+iQFUBBMBCgA+FiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUF +Al2xedcCGwMFCQPCZwAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQOQwbWt6X +iDXu/Qf8CS1g3ErUBwUkRPegJTjwTAxTEwyxMz2h8rYZMW2Tx55KWyRIqt0FxVqm +T7NtEeSoJbkIOcokaaOX673UmM5GJo2SaKqiBOzmYWPDiFYLHnvpfaDV9kDOSJG5 +ohhfSPGQX4x2L2GNO75h9bVSd6gITCSqq9FlyLAoaNZj5nS+1Px9ltAHpli4CQiP ++uCe3alFF7/WsYipUI5FyJ+2aUrHyua7MxNRx1SI0vky+ywqcb6O7+2BSOCkJsZS +5elSN43RfWq2+RABtSBv6kKtjxaC+FW6oCTCp6vJJvVEOA7I+rGUpP5mQHRJMtwW +2MfYBa0Byw+M+1ASE6orIjgyZPTn47kBDQRdsXnXAQgAz+nnewzYZcPEBVUQqObD +91g3sKO2PNrmW+Xql0V5li1D3s9JeufFHlkekcH6EtuvEskHrXBsqLRrCL+TlRf+ +uS+S6/+TPW8Bt0NX2UfEfV9DEKDzSnpxuzt6jDGYM6xLtDrgrJYFo/O/WGx69bnM +829Mg5DCBwLlMZ5x1gqEwFm6AKbsrqILLQSAnGP3N/FVI100aZSLe9dlTEvx/6uh +pRAUdJVNbwNr4GtQkc8PnwJkPNP8hU1yxngo613asQXSLcGOxs1TH5KPJIZq1oan +GTpyCgNYIDSSIjW6SvYEd8I6o3zbKhOGEuuRL3gJxoBWe2Y/vYO7dODXosGSk1tH +owARAQABiQE8BBgBCgAmFiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUFAl2xedcCGwwF +CQPCZwAACgkQOQwbWt6XiDXhAAf/S7RThbngay8xsqtsURpVBDEpzb0a9S3HtzgM +tH0FRvqJdgGUimXoYKI6aF9cBu6d3UQx7i24pAYaMWHYXfsUg9GKXFA5UeKNfm5I +XvAljxB/dize3lAXm09DWeZR/x6eFfqixsVAAenxZz0+r3EM+H14k1HbmzZTHNtE +SPTTDAUii7UKZPuzOj0gjM9QSqY04KCyBoKzi4KXNjQM7yznpfVd0H/Gq2YZt8pd +a57suDGU8qrQdTI01RhizHCNnJWmIVLe4OcNOHQF4XnEWX1Meg24cbGjMocqLJkf +9tVXeGYnwosk65MO6kBnE98Bdsey7T1rH+PyN1ZLYP1Yiq/rbpkCDQRiVq4bARAA +tL77Y7J52fPxk0iyjeJfN826kZ+B+7CBBOPFoGQw19U5y6wimIiqg3+287YJ8k6X +AKzqSRRbyuCb6JCA2M4nOumiCBU65+m8ZfeGSLpCcmhzaInK3tFXkAdZnMzNq0V7 +QnDE4BXGXGbEmLEVorU8H9aaKuAOygDGR2ucEMzgqufsveeFiqdguMZy4bK7CBom +vjThVhMaEDwXERpMJ3d/i2LQkZXT3rGRM/uO4Pdqyb6ewM7brrAEJogWRsL8NXHR ++o/b49QwGBRqGLplBK8EUoJoSqA+q4Rkdd3wDxtOXxHRtyK8RJzrzijkxLkH4zdF +D14EgI9xs/txnVoacOS2iszkFLJYgj9fSFxu78Mp/x7Q9MpiypJSYoGPjSihinL9 +ahhcYGM3p/s0XP0LpTM+A4R8pKXNMPfdJqiR176vn3AGfYmJzSNzRenvkuZF4gFy +WdWbkBISh8XQmxNafccwN8nde9FyjIbfeiMbuHWlJL1q4YsTWTc4ZpLgWvJ1+au0 +VH/sq8Yvsf/rzu0osw/hPizqgOlNpPPZoACepQ+9jmWEcuwjr6KG6Xvg7xZPMDV/ +QnNprC7p+ouuDu5abqoDErsg2ZG749bb249TvAmdJyEug3DLKUVlCKEg3K+icSE6 +MYzX5jojT3WbxwyZAFu3orWSypfCsr7Hpy55dLu2wLUAEQEAAbQdQXJwYWQgQm9k +YSA8YWJvZGFAYXBhY2hlLm9yZz6JAlgEEwEIAEIWIQTWyKcUnUfveqCLNdEGVmjy +pY8JfwUCYlauGwIbAwUJA8JnAAULCQgHAgMiAgEGFQoJCAsCBBYCAwECHgcCF4AA +CgkQBlZo8qWPCX+PkA//feSFzxnb5VW3SrowSwWSwZ4GwhfNebxKiltlr2MpvNXi +EnSmtSX88Ir9wTxdgDPP0pyhxTQIMVn1KltX6a62h0VtNZlFsECDMFDLDapTti3J +BTfzkw2Xjw/mU5WMl/sQzn/dM+1LdVEnt/k15njpaXdxd8mxctUU3PH2p5JMbdQ+ +MpaLPeX7IPv7+cahCaBL40Q/OBCbx8SWrxjnKurmdxOvYOtRq4Y7qzV1YmAk5Nh8 +Pp0P9Us0ihJZ0/AyN0cWNp9wvuTuDubN/omkD2DknnPxh6O5g+PgjY4CHxgRCPzc +IKgBMzeuus8t7oavO1/Hpx7HRUNRZqBXd7b9WWRZ+BV3ZJLP3khaJKVR5wyLY/To +uA87Qx6DjYEabw8JAOXG+SshVTg+TpF0uBghF7dutZ1ERiA8ZA/y09CugByKbT2o +K+2RqNk8B8KMppKV0g82x/8dcVqdf7XZnRA2ckYQfUwNwGBghdEda728tfwfxzRd +XyZRYqiH885/9bAfqTKIib8yJBbCsSw+IjB3d3tCnlBxFFRPH3nK5Vv/FtAq1RbT +AFBN4PC0PxWizayAme0TB2f7iy5Ov/KvXbOUdEQqxEnhFTa5C6aTu8WBFznk23HD +aw+3KMQatOw5uSZ+xNBJfJsMbxntGOEH4kkq8DNdlNT+NoUI1lGXVewwC/22AEC5 +Ag0EYlauGwEQANMmmOFnp9peYWudW65WftqrWOtx5o4uzjJTzGe0ZUdqMlTxcxDV +I2gvqA7iAoVn4V+VKBVcEs2A9sirfYJK3XeN07SPzOzHzVhZN8ygqQ91NAQhAX+C +gkrE2BUq8/0GUGsXXK35P7aHI7ZYH0+a1k7U8Ah44qq4hp1FxWw7K6oO7/vOm9K+ +VhtjSRbPS+PT7T/15GQ0KEhaffOaP/vOC+NGVoUZPA671l+3vRef2y6OoQyGPdLi +Lu0EBSfGN94uEh4stiufZWDPY0Hxl2ZGiX2sAXDcnwzaG6bcnaz2wvKpMRfexMhJ +kQFl5KZPy77f49f4qyas3JUnp7qBstoyHw0GO+gimZridm4G3rXBpjtuR6UCp+op ++TNaUEU4FSvPN3OUHeEoN6xEl8amHTaa3NZlsW9VJFLN+U081oMUwOpmGy62zEAN +pUAkSzDn7qdHbqcZnWCtQnIwDsP4tVYCUzvVzAAqSdpfqnnrppWSbi1Vj4nK+hB9 +tiqgcGOBnuUUH24vIk6IsLuUU/zoaFzhvsux4HRrupTNpd6RIP+/CJjlF6IyE7c/ +XkmLWK0cuhOGYlWv1KEOhShLeQKVAo1/nJ0q4G/eIHjXVa559RBmZRsLjyBZ0Fp9 +9
[nifi-site] branch main updated: move szaszm to pmc
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/main by this push: new f2adfdd move szaszm to pmc new 4726411 Merge pull request #58 from szaszm/szaszm-pmc-member f2adfdd is described below commit f2adfdd3d2962ab73ed76ec0bedcb7a14f068f73 Author: Marton Szasz AuthorDate: Thu Mar 24 16:55:00 2022 + move szaszm to pmc --- src/pages/html/people.hbs | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/pages/html/people.hbs b/src/pages/html/people.hbs index dab5b66..f6d696d 100644 --- a/src/pages/html/people.hbs +++ b/src/pages/html/people.hbs @@ -194,6 +194,11 @@ title: Apache NiFi Team David Handermann + +szaszm +Márton Szász + + @@ -290,11 +295,6 @@ title: Apache NiFi Team -szaszm -Márton Szász - - - adebreceni Adam Debreceni
svn commit: r51359 - /dev/nifi/nifi-minifi-cpp/0.11.0/ /release/nifi/nifi-minifi-cpp/0.11.0/
Author: aboda Date: Mon Dec 13 10:06:29 2021 New Revision: 51359 Log: MINIFICPP-1689 Added: release/nifi/nifi-minifi-cpp/0.11.0/ - copied from r51358, dev/nifi/nifi-minifi-cpp/0.11.0/ Removed: dev/nifi/nifi-minifi-cpp/0.11.0/
[nifi-minifi-cpp] branch main updated: MINIFICPP-1666 Add Azure extension to docker minimal image
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new c0d3bb1 MINIFICPP-1666 Add Azure extension to docker minimal image c0d3bb1 is described below commit c0d3bb104f4048e7fb9499caaae6c82a50cd8866 Author: Gabor Gyimesi AuthorDate: Mon Oct 11 18:30:11 2021 +0200 MINIFICPP-1666 Add Azure extension to docker minimal image Signed-off-by: Arpad Boda This closes #1199 --- cmake/DockerConfig.cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake index 5d3c086..6520200 100644 --- a/cmake/DockerConfig.cmake +++ b/cmake/DockerConfig.cmake @@ -71,6 +71,7 @@ add_custom_target( -c ENABLE_PYTHON=OFF -c ENABLE_LIBRDKAFKA=ON -c ENABLE_AWS=ON +-c ENABLE_AZURE=ON -c DISABLE_CONTROLLER=ON -c ENABLE_SCRIPTING=OFF -c DISABLE_PYTHON_SCRIPTING=ON
[nifi-minifi-cpp] branch main updated: MINIFICPP-1633 Add ccache to docker CI builds
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 48acd26 MINIFICPP-1633 Add ccache to docker CI builds
48acd26 is described below
commit 48acd26e3c07ea1bb05f91fecad601ec67bbae59
Author: Gabor Gyimesi
AuthorDate: Mon Jul 26 10:17:55 2021 +0200
MINIFICPP-1633 Add ccache to docker CI builds
Signed-off-by: Arpad Boda
This closes #1164
---
.github/workflows/ci.yml | 15 ---
.gitignore | 1 +
CMakeLists.txt | 1 +
cmake/DockerConfig.cmake | 1 +
docker/DockerBuild.sh| 29 +++--
docker/Dockerfile| 11 +++
6 files changed, 45 insertions(+), 13 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 29d024b..79dae16 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -258,15 +258,24 @@ jobs:
docker_integration_tests:
name: "Docker integration tests"
runs-on: ubuntu-20.04
-timeout-minutes: 90
+timeout-minutes: 120
steps:
- id: checkout
uses: actions/checkout@v2
+ - id: cache
+uses: actions/cache@v2
+with:
+ path: ~/.ccache
+ key: docker-ccache-${{github.ref}}-${{github.sha}}
+ restore-keys: |
+docker-ccache-${{github.ref}}-
+docker-ccache-refs/heads/main
- id: build
run: |
- ./bootstrap.sh -e -t
+ if [ -d ~/.ccache ]; then mv ~/.ccache .; fi
+ mkdir build
cd build
- cmake -DUSE_SHARED_LIBS= -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_JNI=OFF
-DDISABLE_JEMALLOC=ON -DDISABLE_SCRIPTING=ON -DENABLE_AWS=ON
-DENABLE_LIBRDKAFKA=ON -DENABLE_AZURE=ON -DENABLE_SQL=ON ..
+ cmake -DUSE_SHARED_LIBS= -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_JNI=OFF
-DDISABLE_JEMALLOC=ON -DDISABLE_SCRIPTING=ON -DENABLE_AWS=ON
-DENABLE_LIBRDKAFKA=ON -DENABLE_AZURE=ON -DENABLE_SQL=ON -DDOCKER_BUILD_ONLY=ON
-DDOCKER_CCACHE_DUMP_LOCATION=$HOME/.ccache ..
make docker
- id: install_deps
run: |
diff --git a/.gitignore b/.gitignore
index bcc7486..9dd9de9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -77,6 +77,7 @@ __pycache__/
.vs/slnx.sqlite
/.ccls-cache
/.vscode
+.ccache
# generated files from WEL test
extensions/windows-event-log/tests/custom-provider/unit-test-provider.cs
diff --git a/CMakeLists.txt b/CMakeLists.txt
index fd1de67..355bb6d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -28,6 +28,7 @@ set(BUILD_NUMBER "" CACHE STRING "Build number")
# Base Alpine image to be used in the Docker build instead of the default
Alpine image
set(DOCKER_BASE_IMAGE "" CACHE STRING "Docker build Alpine base image")
+set(DOCKER_CCACHE_DUMP_LOCATION "" CACHE STRING "Directory to dump ccache to
after docker build for later reuse")
include(CMakeDependentOption)
include(CheckIncludeFile)
diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake
index c6059eb..d841c49 100644
--- a/cmake/DockerConfig.cmake
+++ b/cmake/DockerConfig.cmake
@@ -56,6 +56,7 @@ add_custom_target(
-c DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING}
-c DISABLE_CONTROLLER=${DISABLE_CONTROLLER}
-c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE}
+-c DOCKER_CCACHE_DUMP_LOCATION=${DOCKER_CCACHE_DUMP_LOCATION}
-c BUILD_NUMBER=${BUILD_NUMBER}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/)
diff --git a/docker/DockerBuild.sh b/docker/DockerBuild.sh
index 397fa8c..d6e7734 100755
--- a/docker/DockerBuild.sh
+++ b/docker/DockerBuild.sh
@@ -28,6 +28,7 @@ IMAGE_TAG=
DUMP_LOCATION=
DISTRO_NAME=
BUILD_NUMBER=
+DOCKER_CCACHE_DUMP_LOCATION=
function usage {
echo "Usage: ./DockerBuild.sh -v [additional options]"
@@ -58,10 +59,10 @@ while [[ $# -gt 0 ]]; do
shift
;;
-v|--minifi-version)
-MINIFI_VERSION="$2"
-shift
-shift
-;;
+ MINIFI_VERSION="$2"
+ shift
+ shift
+ ;;
-t|--tag)
IMAGE_TAG="$2"
shift
@@ -84,6 +85,8 @@ while [[ $# -gt 0 ]]; do
BUILD_NUMBER="${ARR[1]}"
elif [ "${ARR[0]}" == "DOCKER_BASE_IMAGE" ]; then
BUILD_ARGS="${BUILD_ARGS} --build-arg BASE_ALPINE_IMAGE=${ARR[1]}"
+elif [ "${ARR[0]}" == "DOCKER_CCACHE_DUMP_LOCATION" ]; then
+ DOCKER_CCACHE_DUMP_LOCATION="${ARR[1]}"
else
BUILD_ARGS="${BUILD_ARGS} --build-arg ${ARR[0]}=${ARR[1]}"
fi
@@ -133,14 +136,28 @@ if [ -n "${BUILD_NUMBER}" ]; then
TARGZ_TAG="${TARGZ_TAG}-${BUILD_NUMBER}"
fi
-DOCKER_COMMAND="docker build "
+DOCKER_BUILD_
[nifi-minifi-cpp] branch main updated (b37d398 -> 1fb7540)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from b37d398 MINIFICPP-1607 Implement AttributesToJson processor add 1fb7540 MINIFICPP-1508 Remove the legacy Regex wrapper and use std::regex instead No new revisions were added by this update. Summary of changes: .../integration/minifi/core/DockerTestCluster.py | 6 +- encrypt-config/tests/ConfigFileEncryptorTests.cpp | 5 +- extensions/aws/s3/S3Wrapper.cpp| 11 +- extensions/http-curl/client/HTTPClient.cpp | 8 +- extensions/librdkafka/PublishKafka.cpp | 8 +- extensions/librdkafka/PublishKafka.h | 4 +- extensions/librdkafka/tests/ConsumeKafkaTests.cpp | 1 - extensions/sftp/processors/ListSFTP.cpp| 14 +- extensions/sftp/processors/ListSFTP.h | 6 +- .../standard-processors/processors/ExtractText.cpp | 16 +- .../standard-processors/processors/GetFile.cpp | 6 +- .../standard-processors/processors/TailFile.cpp| 14 +- .../tests/unit/RetryFlowFileTests.cpp | 6 +- libminifi/include/utils/HTTPUtils.h| 22 ++- libminifi/include/utils/RegexUtils.h | 85 -- libminifi/src/utils/RegexUtils.cpp | 175 - libminifi/test/unit/RegexUtilsTests.cpp| 83 -- 17 files changed, 63 insertions(+), 407 deletions(-) delete mode 100644 libminifi/include/utils/RegexUtils.h delete mode 100644 libminifi/src/utils/RegexUtils.cpp delete mode 100644 libminifi/test/unit/RegexUtilsTests.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1605 Always refresh AWS credentials through default credentials chain
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 2d2f300 MINIFICPP-1605 Always refresh AWS credentials through default
credentials chain
2d2f300 is described below
commit 2d2f3004ae42af6aa375d90ecba7a64a266e91ad
Author: Gabor Gyimesi
AuthorDate: Fri Jun 25 11:30:34 2021 +0200
MINIFICPP-1605 Always refresh AWS credentials through default credentials
chain
Signed-off-by: Arpad Boda
This closes #1130
---
extensions/aws/AWSCredentialsProvider.cpp | 4 +++
extensions/aws/AWSCredentialsProvider.h| 1 +
.../controllerservices/AWSCredentialsService.cpp | 2 +-
.../test/aws-tests/AWSCredentialsServiceTest.cpp | 37 ++
4 files changed, 43 insertions(+), 1 deletion(-)
diff --git a/extensions/aws/AWSCredentialsProvider.cpp
b/extensions/aws/AWSCredentialsProvider.cpp
index 9f9f171..7bc4dbd 100644
--- a/extensions/aws/AWSCredentialsProvider.cpp
+++ b/extensions/aws/AWSCredentialsProvider.cpp
@@ -44,6 +44,10 @@ void AWSCredentialsProvider::setUseDefaultCredentials(bool
use_default_credentia
use_default_credentials_ = use_default_credentials;
}
+bool AWSCredentialsProvider::getUseDefaultCredentials() const {
+ return use_default_credentials_;
+}
+
void AWSCredentialsProvider::setAccessKey(const std::string _key) {
access_key_ = access_key;
}
diff --git a/extensions/aws/AWSCredentialsProvider.h
b/extensions/aws/AWSCredentialsProvider.h
index a011887..aef2c80 100644
--- a/extensions/aws/AWSCredentialsProvider.h
+++ b/extensions/aws/AWSCredentialsProvider.h
@@ -46,6 +46,7 @@ class AWSCredentialsProvider {
void setAccessKey(const std::string _key);
void setSecretKey(const std::string _key);
void setCredentialsFile(const std::string _file);
+ bool getUseDefaultCredentials() const;
minifi::utils::optional getAWSCredentials();
private:
diff --git a/extensions/aws/controllerservices/AWSCredentialsService.cpp
b/extensions/aws/controllerservices/AWSCredentialsService.cpp
index 4525420..b1960fa 100644
--- a/extensions/aws/controllerservices/AWSCredentialsService.cpp
+++ b/extensions/aws/controllerservices/AWSCredentialsService.cpp
@@ -77,7 +77,7 @@ void AWSCredentialsService::onEnable() {
}
minifi::utils::optional
AWSCredentialsService::getAWSCredentials() {
- if (!aws_credentials_ || aws_credentials_->IsExpiredOrEmpty()) {
+ if (aws_credentials_provider_.getUseDefaultCredentials() ||
!aws_credentials_ || aws_credentials_->IsExpiredOrEmpty()) {
aws_credentials_ = aws_credentials_provider_.getAWSCredentials();
}
return aws_credentials_;
diff --git a/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
index e6d8aba..3f6167e 100644
--- a/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
+++ b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
@@ -45,6 +45,18 @@ class AWSCredentialsServiceTestAccessor {
std::shared_ptr
aws_credentials_service;
};
+namespace {
+
+void setEnvironmentCredentials(const std::string& key, const std::string&
secret_key) {
+ #ifdef WIN32
+ _putenv_s("AWS_ACCESS_KEY_ID", key.c_str());
+ _putenv_s("AWS_SECRET_ACCESS_KEY", secret_key.c_str());
+ #else
+ setenv("AWS_ACCESS_KEY_ID", key.c_str(), 1);
+ setenv("AWS_SECRET_ACCESS_KEY", secret_key.c_str(), 1);
+ #endif
+}
+
TEST_CASE_METHOD(AWSCredentialsServiceTestAccessor, "Test expired credentials
are refreshed", "[credentialRefresh]") {
plan->setProperty(aws_credentials_service, "Access Key", "key");
plan->setProperty(aws_credentials_service, "Secret Key", "secret");
@@ -65,3 +77,28 @@ TEST_CASE_METHOD(AWSCredentialsServiceTestAccessor, "Test
expired credentials ar
// Check for credential refresh
REQUIRE_FALSE(aws_credentials_impl->getAWSCredentials()->IsExpired());
}
+
+TEST_CASE_METHOD(AWSCredentialsServiceTestAccessor, "Test credentials from
default credential chain are always refreshed", "[credentialRefresh]") {
+ setEnvironmentCredentials("key", "secret");
+ plan->setProperty(aws_credentials_service, "Use Default Credentials",
"true");
+ aws_credentials_service->enable();
+ assert(aws_credentials_service->getControllerServiceImplementation() !=
nullptr);
+ auto aws_credentials_impl =
std::static_pointer_cast(aws_credentials_service->getControllerServiceImplementation());
+
+ // Check intial credentials
+ REQUIRE(aws_credentials_impl->getAWSCredentials());
+ REQUIRE(aws_credentials_impl->getAWSCredentials()->GetAWSAccessKeyId() ==
"key");
+ REQUIRE(aws_credentials_impl->getAWSCredentials()->Get
[nifi-minifi-cpp] branch main updated: MINIFICPP-1561 - Allow rocksdb encryption
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 5af180f MINIFICPP-1561 - Allow rocksdb encryption
5af180f is described below
commit 5af180fd964b6564045b146df8473602a0ec789c
Author: Adam Debreceni
AuthorDate: Thu Jul 8 10:30:08 2021 +0200
MINIFICPP-1561 - Allow rocksdb encryption
Signed-off-by: Arpad Boda
This closes #1090
---
CONFIGURE.md | 21
encrypt-config/ConfigFile.h| 2 +-
encrypt-config/ConfigFileEncryptor.h | 2 +-
encrypt-config/Utils.h | 2 +-
extensions/http-curl/tests/C2ConfigEncryption.cpp | 2 +-
extensions/rocksdb-repos/CMakeLists.txt| 2 +-
.../rocksdb-repos/DatabaseContentRepository.cpp| 16 ++-
.../rocksdb-repos/DatabaseContentRepository.h | 2 +
extensions/rocksdb-repos/FlowFileRepository.cpp| 41 +--
extensions/rocksdb-repos/FlowFileRepository.h | 16 ++-
.../RocksDbPersistableKeyValueStoreService.cpp | 13 +-
.../RocksDbPersistableKeyValueStoreService.h | 2 +
extensions/rocksdb-repos/database/OpenRocksDb.cpp | 9 ++
extensions/rocksdb-repos/database/OpenRocksDb.h| 2 +
.../rocksdb-repos/database/RocksDatabase.cpp | 2 +-
extensions/rocksdb-repos/database/RocksDatabase.h | 2 +-
.../rocksdb-repos/database/RocksDbInstance.cpp | 18 ++-
.../rocksdb-repos/database/RocksDbInstance.h | 4 +
extensions/rocksdb-repos/database/RocksDbUtils.h | 11 +-
extensions/rocksdb-repos/database/StringAppender.h | 11 +-
.../encryption/RocksDbEncryptionProvider.cpp | 127
.../RocksDbEncryptionProvider.h} | 28 ++---
libminifi/CMakeLists.txt | 2 +-
libminifi/include/properties/Decryptor.h | 4 +-
libminifi/include/properties/PropertiesFile.h | 2 +-
.../EncryptionManager.h} | 29 +++--
.../utils/{ => crypto}/EncryptionProvider.h| 15 ++-
.../include/utils/{ => crypto}/EncryptionUtils.h | 0
libminifi/include/utils/crypto/ciphers/Aes256Ecb.h | 78
.../ciphers/XSalsa20.h}| 13 +-
libminifi/include/utils/file/FileSystem.h | 2 +-
libminifi/src/utils/EncryptionProvider.cpp | 58 -
libminifi/src/utils/crypto/EncryptionManager.cpp | 88 ++
.../utils/crypto/EncryptionProvider.cpp} | 31 ++---
.../src/utils/{ => crypto}/EncryptionUtils.cpp | 2 +-
libminifi/src/utils/crypto/ciphers/Aes256Ecb.cpp | 133 +
libminifi/src/utils/file/FileSystem.cpp| 2 +-
libminifi/test/TestBase.h | 7 ++
libminifi/test/rocksdb-tests/EncryptionTests.cpp | 108 +
.../test/rocksdb-tests/RocksDBStreamTests.cpp | 2 +-
libminifi/test/rocksdb-tests/RocksDBTests.cpp | 49 +++-
libminifi/test/unit/EncryptionUtilsTests.cpp | 2 +-
42 files changed, 788 insertions(+), 174 deletions(-)
diff --git a/CONFIGURE.md b/CONFIGURE.md
index c93352d..13c859d 100644
--- a/CONFIGURE.md
+++ b/CONFIGURE.md
@@ -166,6 +166,8 @@ folder. You may specify your own path in place of these
defaults.
nifi.flowfile.repository.directory.default=${MINIFI_HOME}/flowfile_repository
nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_repository
+ Shared database
+
It is also possible to use a single database to store multiple repositories
with the `minifidb://` scheme.
This could help with migration and centralize agent state persistence. In the
scheme the final path segment designates the
column family in the repository, while the preceding path indicates the
directory the rocksdb database is
@@ -189,6 +191,25 @@ Moreover the `"default"` name is restricted and should not
be used.
nifi.state.manangement.provider.local.path=minifidb://${MINIFI_HOME}/agent_state/default
^ error: "default" is restricted
+### Configuring Repository encryption
+
+It is possible to provide rocksdb-backed repositories a key to request their
+encryption.
+
+in conf/bootstrap.conf
+
nifi.flowfile.repository.encryption.key=805D7B95EF44DC27C87FFBC4DFDE376DAE604D55DB2C5496DEEF5236362DE62E
+nifi.database.content.repository.encryption.key=
+# nifi.state.management.provider.local.encryption.key=
+
+In the above configuration the first line will cause `FlowFileRepository` to
use the specified `256` bit key.
+The second line will trigger the generation of a random (`256` bit) key
persisted back into `conf/bootstrap.conf`, which `DatabaseContentRepository`
will then use for encryption.
+(This way one can request e
[nifi-minifi-cpp] branch main updated (1ce101f -> 9173029)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 1ce101f MINIFICPP-1548 - Use the same rocksdb instance for multiple repositories add 9173029 MINIFICPP-1426 MSVC: Use /std:c++latest /permissive- and drop VS 2017 No new revisions were added by this update. Summary of changes: .github/workflows/ci.yml | 45 + .gitignore | 1 + CMakeLists.txt | 10 +--- README.md | 2 +- Windows.md | 11 ++-- cmake/BundledAzureSdkCpp.cmake | 8 ++- cmake/CppVersion.cmake | 42 +++ controller/CMakeLists.txt | 16 +- extensions/aws/s3/S3Wrapper.cpp| 4 +- extensions/script/CMakeLists.txt | 19 +-- extensions/sql/CMakeLists.txt | 5 +- extensions/windows-event-log/wel/JSONUtils.cpp | 2 +- extensions/windows-event-log/wel/JSONUtils.h | 2 +- libminifi/CMakeLists.txt | 59 ++ libminifi/include/utils/GeneralUtils.h | 2 +- main/CMakeLists.txt| 22 +--- main/MiNiFiWindowsService.cpp | 4 +- nanofi/CMakeLists.txt | 23 + nanofi/examples/CMakeLists.txt | 58 +++-- python/library/CMakeLists.txt | 2 +- .../fix-illegal-qualified-name-in-member.patch | 14 + win_build_vs.bat | 3 +- 22 files changed, 124 insertions(+), 230 deletions(-) create mode 100644 cmake/CppVersion.cmake create mode 100644 thirdparty/azure-sdk-cpp-for-cpp/fix-illegal-qualified-name-in-member.patch
[nifi-minifi-cpp] branch main updated (f2a0a42 -> 1ce101f)
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git.
from f2a0a42 MINIFICPP-1586-`make linter` gives errors when it is run in
an unpacked release tarball
add 1ce101f MINIFICPP-1548 - Use the same rocksdb instance for multiple
repositories
No new revisions were added by this update.
Summary of changes:
CONFIGURE.md | 23 +++
conf/minifi.properties | 1 +
extensions/rocksdb-repos/CMakeLists.txt| 2 +-
.../rocksdb-repos/DatabaseContentRepository.cpp| 24 ++-
.../rocksdb-repos/DatabaseContentRepository.h | 37 +---
extensions/rocksdb-repos/FlowFileRepository.cpp| 19 +-
extensions/rocksdb-repos/FlowFileRepository.h | 25 +--
extensions/rocksdb-repos/RocksDatabase.cpp | 149 ---
extensions/rocksdb-repos/RocksDbStream.cpp | 2 +-
extensions/rocksdb-repos/RocksDbStream.h | 6 +-
.../RocksDbPersistableKeyValueStoreService.cpp | 21 +--
.../RocksDbPersistableKeyValueStoreService.h | 2 +-
.../rocksdb-repos/database/ColumnHandle.cpp| 16 +-
.../rocksdb-repos/database/ColumnHandle.h | 22 +--
extensions/rocksdb-repos/database/OpenRocksDb.cpp | 114
.../{RocksDatabase.h => database/OpenRocksDb.h}| 77 +++-
.../rocksdb-repos/database/RocksDatabase.cpp | 91 +
.../rocksdb-repos/database/RocksDatabase.h | 52 +++---
.../rocksdb-repos/database/RocksDbInstance.cpp | 169 +
.../rocksdb-repos/database/RocksDbInstance.h | 71 +++
extensions/rocksdb-repos/database/RocksDbUtils.h | 80
.../rocksdb-repos/database/StringAppender.cpp | 59 ++
.../rocksdb-repos/database/StringAppender.h| 42 +++--
.../rocksdb-repos/database/WriteBatch.cpp | 25 +--
.../rocksdb-repos/database/WriteBatch.h| 32 ++--
libminifi/include/core/ProcessContext.h| 9 +-
libminifi/include/properties/Configuration.h | 1 +
libminifi/include/utils/GeneralUtils.h | 9 +
libminifi/test/TestBase.cpp| 5 +-
libminifi/test/integration/IntegrationBase.h | 5 +-
.../test/rocksdb-tests/RocksDBStreamTests.cpp | 18 +-
libminifi/test/rocksdb-tests/RocksDBTests.cpp | 206 +
32 files changed, 1015 insertions(+), 399 deletions(-)
delete mode 100644 extensions/rocksdb-repos/RocksDatabase.cpp
copy libminifi/include/utils/tls/TLSUtils.h =>
extensions/rocksdb-repos/database/ColumnHandle.cpp (76%)
copy libminifi/include/utils/requirements/EqualityComparable.h =>
extensions/rocksdb-repos/database/ColumnHandle.h (71%)
create mode 100644 extensions/rocksdb-repos/database/OpenRocksDb.cpp
rename extensions/rocksdb-repos/{RocksDatabase.h => database/OpenRocksDb.h}
(57%)
create mode 100644 extensions/rocksdb-repos/database/RocksDatabase.cpp
copy libminifi/include/core/logging/WindowsEventLogSink.h =>
extensions/rocksdb-repos/database/RocksDatabase.h (52%)
create mode 100644 extensions/rocksdb-repos/database/RocksDbInstance.cpp
create mode 100644 extensions/rocksdb-repos/database/RocksDbInstance.h
create mode 100644 extensions/rocksdb-repos/database/RocksDbUtils.h
create mode 100644 extensions/rocksdb-repos/database/StringAppender.cpp
copy libminifi/include/core/TraceableResource.h =>
extensions/rocksdb-repos/database/StringAppender.h (56%)
copy libminifi/include/utils/requirements/EqualityComparable.h =>
extensions/rocksdb-repos/database/WriteBatch.cpp (68%)
copy libminifi/include/utils/tls/WindowsCertStoreLocation.h =>
extensions/rocksdb-repos/database/WriteBatch.h (67%)
create mode 100644 libminifi/test/rocksdb-tests/RocksDBTests.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked release tarball
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new f2a0a42 MINIFICPP-1586-`make linter` gives errors when it is run in
an unpacked release tarball
f2a0a42 is described below
commit f2a0a42c55d5befcf7cf55b692987dbffa9d161b
Author: aminadinari19
AuthorDate: Tue Jun 15 11:59:03 2021 +0200
MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked
release tarball
MINIFICPP-1586-keeping script_dir variable
MINIFICPP-1586-Removed '/' from sub-directories
MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked
released tarball
MINIFICPP-1586-new line at the end
Signed-off-by: Arpad Boda
---
thirdparty/google-styleguide/run_linter.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/thirdparty/google-styleguide/run_linter.py
b/thirdparty/google-styleguide/run_linter.py
index 1cf03a0..7f7f6fd 100644
--- a/thirdparty/google-styleguide/run_linter.py
+++ b/thirdparty/google-styleguide/run_linter.py
@@ -2,9 +2,7 @@ import argparse
import os
import cpplint
-script_dir = os.path.dirname(os.path.realpath(__file__))
parser = argparse.ArgumentParser()
-
parser.add_argument('-i', '--includePaths', nargs="+", help='Run linter check
in these directories')
parser.add_argument('-q', '--quiet', action='store_true', help='Don\'t print
anything if no errors are found.')
args = parser.parse_args()
@@ -16,7 +14,8 @@ for include_path in args.includePaths:
if (".h" in file_name) or (".cpp" in file_name):
list_of_files += [os.path.join(dir_path, file_name)]
-repository_path = script_dir + "../../"
+script_dir = os.path.dirname(os.path.realpath(__file__))
+repository_path = os.path.abspath(os.path.join(script_dir, os.pardir,
os.pardir))
arg_list = list()
arg_list.append("--linelength=200")
[nifi-site] branch main updated: Add Ferenc Gerlits to the list of committers
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/main by this push: new e652912 Add Ferenc Gerlits to the list of committers new 658f90a Merge pull request #46 from fgerlits/add-fgerlits-to-the-list-of-committers e652912 is described below commit e652912591e2f9bafe9befc2c631096df2d74a73 Author: Ferenc Gerlits AuthorDate: Fri Jun 11 12:48:47 2021 +0200 Add Ferenc Gerlits to the list of committers --- src/pages/html/people.hbs | 5 + 1 file changed, 5 insertions(+) diff --git a/src/pages/html/people.hbs b/src/pages/html/people.hbs index 30ec17e..8bf093c 100644 --- a/src/pages/html/people.hbs +++ b/src/pages/html/people.hbs @@ -289,6 +289,11 @@ title: Apache NiFi Team Adam Debreceni + +fgerlits +Ferenc Gerlits + +
svn commit: r48291 - /dev/nifi/nifi-minifi-cpp/0.10.0/ /release/nifi/nifi-minifi-cpp/0.10.0/
Author: aboda Date: Fri Jun 11 09:17:11 2021 New Revision: 48291 Log: MINIFICPP-1574 Promote NiFi MiNiFi C++ 0.10.0 vote artifacts to release Added: release/nifi/nifi-minifi-cpp/0.10.0/ - copied from r48290, dev/nifi/nifi-minifi-cpp/0.10.0/ Removed: dev/nifi/nifi-minifi-cpp/0.10.0/
[nifi-minifi-cpp] branch main updated: MINIFICPP-1585 Rename enum values in ConsumeKafkaTests::PublishEvent
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 4d12658 MINIFICPP-1585 Rename enum values in
ConsumeKafkaTests::PublishEvent
4d12658 is described below
commit 4d12658952f553f1a8285ec5224698a72e708451
Author: Ferenc Gerlits
AuthorDate: Mon Jun 7 10:03:05 2021 +0200
MINIFICPP-1585 Rename enum values in ConsumeKafkaTests::PublishEvent
TRANSACTION_COMMIT is defined in winnt.h as a macro, so the build fails
on Windows. I have renamed it, and a few other enum values for
consistency.
Also added /K to the VS2019 Windows CI job, to make sure bugs like this
get caught in the CI stage in the future.
Signed-off-by: Arpad Boda
This closes #1102
---
.github/workflows/ci.yml | 2 +-
extensions/librdkafka/tests/ConsumeKafkaTests.cpp | 30 +++
2 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7cafa6f..c8399dc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -154,7 +154,7 @@ jobs:
run: |
PATH %PATH%;C:\Program Files (x86)\Windows
Kits\10\bin\10.0.19041.0\x64
PATH %PATH%;C:\Program Files (x86)\Microsoft Visual
Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn
- win_build_vs.bat build /2019 /64 /CI /S /A /PDH /L /R
+ win_build_vs.bat build /2019 /64 /CI /S /A /PDH /K /L /R
shell: cmd
- name: test
run: cd build && ctest --timeout 300 --parallel 8 -C Release
--output-on-failure
diff --git a/extensions/librdkafka/tests/ConsumeKafkaTests.cpp
b/extensions/librdkafka/tests/ConsumeKafkaTests.cpp
index c1d50f4..b011ea1 100644
--- a/extensions/librdkafka/tests/ConsumeKafkaTests.cpp
+++ b/extensions/librdkafka/tests/ConsumeKafkaTests.cpp
@@ -43,9 +43,9 @@ class KafkaTestProducer {
public:
enum class PublishEvent {
PUBLISH,
-TRANSACTION_START,
-TRANSACTION_COMMIT,
-CANCEL
+BEGIN_TRANSACTION,
+END_TRANSACTION,
+CANCEL_TRANSACTION
};
KafkaTestProducer(const std::string& kafka_brokers, const std::string&
topic, const bool transactional) :
logger_(logging::LoggerFactory::getLogger()) {
@@ -88,15 +88,15 @@ class KafkaTestProducer {
publish_message(*next_message, message_key, message_headers);
std::advance(next_message, 1);
break;
-case PublishEvent::TRANSACTION_START:
+case PublishEvent::BEGIN_TRANSACTION:
logger_->log_debug("Starting new transaction...");
rd_kafka_begin_transaction(producer_.get());
break;
-case PublishEvent::TRANSACTION_COMMIT:
+case PublishEvent::END_TRANSACTION:
logger_->log_debug("Committing transaction...");
rd_kafka_commit_transaction(producer_.get(),
TRANSACTIONS_TIMEOUT_MS.count());
break;
-case PublishEvent::CANCEL:
+case PublishEvent::CANCEL_TRANSACTION:
logger_->log_debug("Cancelling transaction...");
rd_kafka_abort_transaction(producer_.get(),
TRANSACTIONS_TIMEOUT_MS.count());
}
@@ -147,15 +147,15 @@ class ConsumeKafkaTest {
using ExtractText = org::apache::nifi::minifi::processors::ExtractText;
const KafkaTestProducer::PublishEvent PUBLISH=
KafkaTestProducer::PublishEvent::PUBLISH;
- const KafkaTestProducer::PublishEvent TRANSACTION_START =
KafkaTestProducer::PublishEvent::TRANSACTION_START;
- const KafkaTestProducer::PublishEvent TRANSACTION_COMMIT =
KafkaTestProducer::PublishEvent::TRANSACTION_COMMIT;
- const KafkaTestProducer::PublishEvent CANCEL =
KafkaTestProducer::PublishEvent::CANCEL;
+ const KafkaTestProducer::PublishEvent BEGIN_TRANSACTION =
KafkaTestProducer::PublishEvent::BEGIN_TRANSACTION;
+ const KafkaTestProducer::PublishEvent END_TRANSACTION=
KafkaTestProducer::PublishEvent::END_TRANSACTION;
+ const KafkaTestProducer::PublishEvent CANCEL_TRANSACTION =
KafkaTestProducer::PublishEvent::CANCEL_TRANSACTION;
const std::vector
NON_TRANSACTIONAL_MESSAGES { PUBLISH, PUBLISH };
- const std::vector
SINGLE_COMMITTED_TRANSACTION { TRANSACTION_START, PUBLISH, PUBLISH,
TRANSACTION_COMMIT };
- const std::vector TWO_SEPARATE_TRANSACTIONS
{ TRANSACTION_START, PUBLISH, TRANSACTION_COMMIT, TRANSACTION_START,
PUBLISH, TRANSACTION_COMMIT };
- const std::vector NON_COMMITTED_TRANSACTION
{ TRANSACTION_START, PUBLISH, PUBLISH };
- const std::vector CANCELLED_TRANSACTION
{ TRANSACTION_START, PUBLISH, CANCEL };
+ const std::vector
SINGLE_COMMITTED_TRANSACTION { BEGIN_TRANSACTION, PUBLISH, PUBLISH,
END_TRANSACTION };
+ const std::vector TWO_SEPARATE_TRANSACTIONS
{ BEGIN_TRANSACTIO
[nifi-minifi-cpp] branch main updated: MINIFICPP-1581: set cmake policy CMP0096 to NEW to preserve the leading zeros in project version
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 6020496 MINIFICPP-1581: set cmake policy CMP0096 to NEW to preserve
the leading zeros in project version
6020496 is described below
commit 6020496ad1cd2fc236815ee78b3d9f2cee640a0c
Author: Martin Zink
AuthorDate: Fri Jun 4 10:24:47 2021 +0200
MINIFICPP-1581: set cmake policy CMP0096 to NEW to preserve the leading
zeros in project version
MINIFICPP-1581: set cmake policy for nanofi and libminifi
use the buster-backports repo to install cmake 3.16.3 in debian ci job
updated README.md and all cmake requirements to 3.16
use java 11 on debian buster
Signed-off-by: Arpad Boda
This closes #1098
---
CMakeLists.txt | 3 ++-
README.md | 2 +-
controller/CMakeLists.txt | 2 +-
debian.sh | 4 ++--
docker/debian/Dockerfile | 6 --
extensions/ExtensionHeader.txt | 2 +-
libminifi/CMakeLists.txt | 4 ++--
main/CMakeLists.txt| 2 +-
nanofi/CMakeLists.txt | 3 ++-
nanofi/ecu/CMakeLists.txt | 2 +-
nanofi/examples/CMakeLists.txt | 2 +-
11 files changed, 18 insertions(+), 14 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 8f4d496..1ac5db7 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,7 +17,8 @@
# under the License.
#
-cmake_minimum_required(VERSION 3.11)
+cmake_minimum_required(VERSION 3.16)
+cmake_policy(SET CMP0096 NEW) # policy to preserve the leading zeros in
PROJECT_VERSION_{MAJOR,MINOR,PATCH,TWEAK}
cmake_policy(SET CMP0065 OLD) # default export policy, required for self-dlopen
project(nifi-minifi-cpp VERSION 0.10.0)
set(PROJECT_NAME "nifi-minifi-cpp")
diff --git a/README.md b/README.md
index 4e8a5f6..ecf16c4 100644
--- a/README.md
+++ b/README.md
@@ -111,7 +111,7 @@ Through JNI extensions you can run NiFi processors using
NARs. The JNI extension
### To build
Utilities
-* CMake 3.11 or greater
+* CMake 3.16 or greater
* gcc 4.8.4 or greater
* g++ 4.8.4 or greater
* bison 3.0.x (3.2 has been shown to fail builds)
diff --git a/controller/CMakeLists.txt b/controller/CMakeLists.txt
index 6610973..5533ba7 100644
--- a/controller/CMakeLists.txt
+++ b/controller/CMakeLists.txt
@@ -17,7 +17,7 @@
# under the License.
#
-cmake_minimum_required(VERSION 3.11)
+cmake_minimum_required(VERSION 3.16)
include_directories(../main/ ../libminifi/include ../libminifi/include/c2
../libminifi/include/c2/protocols/ ../libminifi/include/core/state
./libminifi/include/core/statemanagement/metrics
../libminifi/include/core/yaml ../libminifi/include/core)
diff --git a/debian.sh b/debian.sh
index 0df18e7..52836e9 100644
--- a/debian.sh
+++ b/debian.sh
@@ -67,8 +67,8 @@ build_deps(){
elif [ "$FOUND_VALUE" = "lua" ]; then
INSTALLED+=("liblua5.1-0-dev")
elif [ "$FOUND_VALUE" = "jnibuild" ]; then
-INSTALLED+=("openjdk-8-jdk")
-INSTALLED+=("openjdk-8-source")
+INSTALLED+=("openjdk-11-jdk")
+INSTALLED+=("openjdk-11-source")
INSTALLED+=("maven")
elif [ "$FOUND_VALUE" = "automake" ]; then
INSTALLED+=("automake")
diff --git a/docker/debian/Dockerfile b/docker/debian/Dockerfile
index 1691606..e353b8b 100644
--- a/docker/debian/Dockerfile
+++ b/docker/debian/Dockerfile
@@ -26,9 +26,11 @@ ARG MINIFI_VERSION
ENV MINIFI_BASE_DIR /opt/minifi
ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION
-ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
+ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/
-RUN apt-get update && apt-get install -y openjdk-11-jdk openjdk-11-source
libpython3.7-dev sudo git maven libarchive13 cmake
+RUN echo "deb http://deb.debian.org/debian buster-backports main" | tee -a
/etc/apt/sources.list \
+&& apt-get update && apt-get install -y openjdk-11-jdk openjdk-11-source
libpython3.7-dev sudo git maven libarchive13 libtool \
+&& apt-get -t buster-backports install -y cmake
RUN mkdir -p $MINIFI_BASE_DIR
COPY . ${MINIFI_BASE_DIR}
diff --git a/extensions/ExtensionHeader.txt b/extensions/ExtensionHeader.txt
index d7ac551..db240d1 100644
--- a/extensions/ExtensionHeader.txt
+++ b/extensions/ExtensionHeader.txt
@@ -17,7 +17,7 @@
# under the License.
#
-cmake_minimum_required(VERSION 3.11)
+cmake_minimum_required(VERSION 3.16)
include_directories(../../libminifi/include ../../libminifi/include/core)
diff --git a/libminifi/CMakeLists.txt b/libminifi/CMakeLists.txt
index 26bd3b3..7a4b751 100644
--- a/libminifi/CMakeLists.txt
+++
[nifi-minifi-cpp] 01/02: MINIFICPP-1582 fixing build falure in centos ci job
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit e51192ed9639c04d69252fa033b444373173a00b
Author: Martin Zink
AuthorDate: Fri Jun 4 15:56:04 2021 +0200
MINIFICPP-1582 fixing build falure in centos ci job
centos cmake symbol lookup error fix
Signed-off-by: Arpad Boda
This closes #1099
---
docker/centos/Dockerfile | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docker/centos/Dockerfile b/docker/centos/Dockerfile
index e54ab4c..b6284c1 100644
--- a/docker/centos/Dockerfile
+++ b/docker/centos/Dockerfile
@@ -27,7 +27,7 @@ ARG MINIFI_VERSION
ENV MINIFI_BASE_DIR /opt/minifi
ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION
-RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel python36-devel
gcc gcc-c++ sudo git which maven make cmake
+RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel python36-devel
gcc gcc-c++ sudo git which maven make cmake libarchive
RUN mkdir -p $MINIFI_BASE_DIR
COPY . ${MINIFI_BASE_DIR}
@@ -40,6 +40,6 @@ RUN cd $MINIFI_BASE_DIR \
&& rm -rf build \
&& mkdir build \
&& cd build \
- && cmake3 -DUSE_SHARED_LIBS= -DENABLE_MQTT=ON -DENABLE_LIBRDKAFKA=ON
-DPORTABLE=ON -DENABLE_COAP=ON -DCMAKE_BUILD_TYPE=Release -DSKIP_TESTS=true
-DENABLE_JNI=$ENABLE_JNI .. \
+ && cmake -DUSE_SHARED_LIBS= -DENABLE_MQTT=ON -DENABLE_LIBRDKAFKA=ON
-DPORTABLE=ON -DENABLE_COAP=ON -DCMAKE_BUILD_TYPE=Release -DSKIP_TESTS=true
-DENABLE_JNI=$ENABLE_JNI .. \
&& make -j$(nproc) package
[nifi-minifi-cpp] branch main updated (72e9c9e -> 33d65ed)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 72e9c9e Squashed commit of the following: new e51192e MINIFICPP-1582 fixing build falure in centos ci job new 33d65ed MINIFICPP-1583 windows build fix The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docker/centos/Dockerfile | 4 ++-- extensions/pdh/tests/PerformanceDataMonitorTests.cpp | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
[nifi-minifi-cpp] 02/02: MINIFICPP-1583 windows build fix
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 33d65edb3f005a17f3edbfd6fdcf9401605da568
Author: Martin Zink
AuthorDate: Fri Jun 4 17:59:39 2021 +0200
MINIFICPP-1583 windows build fix
Signed-off-by: Arpad Boda
This closes #1100
---
extensions/pdh/tests/PerformanceDataMonitorTests.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
index b3d5acd..cb49bab 100644
--- a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
+++ b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
@@ -250,16 +250,16 @@
TEST_CASE("PerformanceDataMonitorDecimalPlacesPropertyTest", "[performancedatamo
{
PerformanceDataMonitorTester tester;
tester.setPerformanceMonitorProperty(PerformanceDataMonitor::DecimalPlaces,
"asd");
-REQUIRE_THROWS_WITH(tester.runProcessors(), "General Operation: Invalid
conversion to int64_t for asd");
+REQUIRE_THROWS_WITH(tester.runWithRetries([]{ return true; }, 1), "General
Operation: Invalid conversion to int64_t for asd");
}
{
PerformanceDataMonitorTester tester;
tester.setPerformanceMonitorProperty(PerformanceDataMonitor::DecimalPlaces,
"1234586123");
-REQUIRE_THROWS_WITH(tester.runProcessors(), "Process Schedule Operation:
PerformanceDataMonitor Decimal Places is out of range");
+REQUIRE_THROWS_WITH(tester.runWithRetries([]{ return true; }, 1), "Process
Schedule Operation: PerformanceDataMonitor Decimal Places is out of range");
}
{
PerformanceDataMonitorTester tester;
tester.setPerformanceMonitorProperty(PerformanceDataMonitor::DecimalPlaces, "");
-REQUIRE_NOTHROW(tester.runProcessors());
+REQUIRE_NOTHROW(tester.runWithRetries([]{ return true; }, 1));
}
}
[nifi] branch main updated: [NIFI-8610] Do not reuse generic records in convertAvroToORC
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new f07e17b [NIFI-8610] Do not reuse generic records in convertAvroToORC
f07e17b is described below
commit f07e17ba74da2759213d52431e00f9d0ba5d39a5
Author: Dominik Przybysz
AuthorDate: Tue May 18 10:26:58 2021 +0200
[NIFI-8610] Do not reuse generic records in convertAvroToORC
[NIFI-8610] Simplify decimal test for convertAvroToORC
Signed-off-by: Arpad Boda
This closes #1081
---
.../nifi/processors/hive/ConvertAvroToORC.java | 5 +-
.../nifi/processors/hive/TestConvertAvroToORC.java | 65 +-
2 files changed, 66 insertions(+), 4 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java
b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java
index b323633..d918365 100644
---
a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java
+++
b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java
@@ -234,9 +234,8 @@ public class ConvertAvroToORC extends AbstractProcessor {
try {
int recordCount = 0;
-GenericRecord currRecord = null;
while (reader.hasNext()) {
-currRecord = reader.next(currRecord);
+GenericRecord currRecord = reader.next();
List fields =
currRecord.getSchema().getFields();
if (fields != null) {
Object[] row = new Object[fields.size()];
@@ -284,7 +283,7 @@ public class ConvertAvroToORC extends AbstractProcessor {
flowFile = session.putAttribute(flowFile,
CoreAttributes.MIME_TYPE.key(), ORC_MIME_TYPE);
flowFile = session.putAttribute(flowFile,
CoreAttributes.FILENAME.key(), newFilename.toString());
session.transfer(flowFile, REL_SUCCESS);
-session.getProvenanceReporter().modifyContent(flowFile, "Converted
"+totalRecordCount.get()+" records", System.currentTimeMillis() - startTime);
+session.getProvenanceReporter().modifyContent(flowFile, "Converted
" + totalRecordCount.get() + " records", System.currentTimeMillis() -
startTime);
} catch (ProcessException | IllegalArgumentException e) {
getLogger().error("Failed to convert {} from Avro to ORC due to
{}; transferring to failure", new Object[]{flowFile, e});
diff --git
a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java
b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java
index 9248b27..5fe6752 100644
---
a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java
+++
b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java
@@ -49,9 +49,9 @@ import org.junit.Test;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
-import java.io.InputStream;
import java.io.File;
import java.io.FileOutputStream;
+import java.io.InputStream;
import java.math.BigDecimal;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
@@ -321,6 +321,69 @@ public class TestConvertAvroToORC {
}
@Test
+public void test_onTrigger_complex_records_with_bigdecimals() throws
Exception {
+
+Map mapData1 = new TreeMap() {{
+put("key1", 1.0);
+put("key2", 2.0);
+}};
+
+
+BigDecimal sampleBigDecimal1 = new BigDecimal("3500.12");
+BigDecimal sampleBigDecimal2 = new BigDecimal("0.01");
+
+GenericData.Record record1 =
TestNiFiOrcUtils.buildComplexAvroRecord(null, mapData1, "XYZ", 4L,
Arrays.asList(100, 200), toByteBuffer(sampleBigDecimal1));
+DatumWriter writer = new
GenericDatumWriter<>(record1.getSchema());
+DataFileWriter fileWriter = new
DataFileWriter<>(writer);
+ByteArrayOutputStream out = new ByteArrayOutputStream();
+fileWriter.create(record1.getSchema(), out);
+fileWriter.append(record1);
+fileWriter.append(TestNiFiOrcUtils.buildComplexAvroRecord(null,
mapData1, "XYZ", 4L, Arrays.asList(100, 200), toByteBuffer(sampleBigDecimal2)));
+fileWriter
[nifi-minifi-cpp] branch main updated: MINIFICPP-1568 Retry PDH processor run to fix test flakiness
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 05ae25c MINIFICPP-1568 Retry PDH processor run to fix test flakiness
05ae25c is described below
commit 05ae25c1aae280767206b641f8c35d485aa85245
Author: Gabor Gyimesi
AuthorDate: Wed May 26 17:05:47 2021 +0200
MINIFICPP-1568 Retry PDH processor run to fix test flakiness
Signed-off-by: Arpad Boda
This closes #1085
---
.../pdh/tests/PerformanceDataMonitorTests.cpp | 283 +++--
1 file changed, 148 insertions(+), 135 deletions(-)
diff --git a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
index 7160bc5..b3d5acd 100644
--- a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
+++ b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp
@@ -20,6 +20,7 @@
#include
#include
#include
+#include
#include "TestBase.h"
#include "processors/PutFile.h"
@@ -43,12 +44,18 @@ class PerformanceDataMonitorTester {
plan_->setProperty(putfile_, PutFile::Directory.getName(), dir_);
}
- void runProcessors() {
-plan_->runNextProcessor(); // PerformanceMonitor
-std::this_thread::sleep_for(std::chrono::milliseconds(200));
-plan_->runCurrentProcessor(); // PerformanceMonitor
-plan_->runNextProcessor(); // PutFile
-plan_->runCurrentProcessor(); // PutFile
+ bool runWithRetries(std::function&& assertions, uint32_t max_tries =
10) {
+uint32_t tries = 0;
+for (uint32_t tries = 0; tries < max_tries; ++tries) {
+ test_controller_.runSession(plan_);
+ if (assertions()) {
+return true;
+ }
+ plan_->reset();
+ LogTestController::getInstance().reset();
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+}
+return false;
}
void setPerformanceMonitorProperty(const core::Property& property, const
std::string& value) {
@@ -65,8 +72,7 @@ class PerformanceDataMonitorTester {
TEST_CASE("PerformanceDataMonitorEmptyPropertiesTest",
"[performancedatamonitoremptypropertiestest]") {
PerformanceDataMonitorTester tester;
- tester.runProcessors();
-
+ tester.test_controller_.runSession(tester.plan_);
REQUIRE(tester.test_controller_.getLog().getInstance().contains("No valid
counters for PerformanceDataMonitor", std::chrono::seconds(0)));
auto created_flow_files = utils::file::FileUtils::list_dir_all(tester.dir_,
tester.plan_->getLogger(), false);
@@ -77,61 +83,64 @@
TEST_CASE("PerformanceDataMonitorPartiallyInvalidGroupPropertyTest", "[performan
PerformanceDataMonitorTester tester;
tester.setPerformanceMonitorProperty(PerformanceDataMonitor::PredefinedGroups,
"Disk,CPU,Asd");
tester.setPerformanceMonitorProperty(PerformanceDataMonitor::CustomPDHCounters,
"\\Invalid\\Counter,\\System\\Processes");
- tester.runProcessors();
-
- REQUIRE(tester.test_controller_.getLog().getInstance().contains("Asd is not
a valid predefined group", std::chrono::seconds(0)));
- REQUIRE(tester.test_controller_.getLog().getInstance().contains("Error
adding \\Invalid\\Counter to query", std::chrono::seconds(0)));
-
- uint32_t number_of_flowfiles = 0;
-
- auto lambda = [_of_flowfiles](const std::string& path, const
std::string& filename) -> bool {
-++number_of_flowfiles;
-FILE* fp = fopen((path + utils::file::FileUtils::get_separator() +
filename).c_str(), "r");
-REQUIRE(fp != nullptr);
-char readBuffer[500];
-rapidjson::FileReadStream is(fp, readBuffer, sizeof(readBuffer));
-rapidjson::Document document;
-document.ParseStream(is);
-fclose(fp);
-REQUIRE(document.IsObject());
-REQUIRE(document.HasMember("LogicalDisk"));
-REQUIRE(document["LogicalDisk"].HasMember("_Total"));
-REQUIRE(document["LogicalDisk"]["_Total"].HasMember("Free Megabytes"));
-REQUIRE(document["System"].HasMember("Processes"));
-return true;
+ const auto assertions = []() {
+REQUIRE(tester.test_controller_.getLog().getInstance().contains("Asd is
not a valid predefined group", std::chrono::seconds(0)));
+REQUIRE(tester.test_controller_.getLog().getInstance().contains("Error
adding \\Invalid\\Counter to query", std::chrono::seconds(0)));
+
+bool ff_contains_all_data = false;
+
+const auto lambda = [_contains_all_data](const std::string& path, const
std::string& filename) -> bool {
+ FILE* fp = fopen((path + utils::file::FileUtils::get_separator() +
filename).c_str(), "r");
+ REQUIRE(fp != nullptr);
+
[nifi-minifi-cpp] branch main updated: MINIFICPP-1577: Separate build and test steps in ubuntu-20.04 ci job
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 3c9fd89 MINIFICPP-1577: Separate build and test steps in ubuntu-20.04 ci job 3c9fd89 is described below commit 3c9fd895d413efce2e77cfcbe599ba15162d5df6 Author: Martin Zink AuthorDate: Tue Jun 1 14:23:25 2021 +0200 MINIFICPP-1577: Separate build and test steps in ubuntu-20.04 ci job Signed-off-by: Arpad Boda This closes #1095 --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 073350e..66efcfd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -239,7 +239,7 @@ jobs: echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts > /dev/null - name: build -run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DCMAKE_BUILD_TYPE=Release -DENABLE_BUSTACHE=ON -DENABLE_SQL=ON -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4 VERBOSE=1 && make test ARGS="--timeout 300 -j2 --output-on-failure" +run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DCMAKE_BUILD_TYPE=Release -DENABLE_BUSTACHE=ON -DENABLE_SQL=ON -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4 VERBOSE=1 - name: test run: cd build && make test ARGS="--timeout 300 -j2 --output-on-failure" - name: linter
[nifi-minifi-cpp] branch main updated: MINIFICPP-1578 Upgrade OS versions in docker builds
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 876906f MINIFICPP-1578 Upgrade OS versions in docker builds
876906f is described below
commit 876906faffe274268a8ea417ee28b4b44f0ce8d8
Author: Ferenc Gerlits
AuthorDate: Tue Jun 1 18:48:53 2021 +0200
MINIFICPP-1578 Upgrade OS versions in docker builds
Signed-off-by: Arpad Boda
This closes #1093
---
.gitignore | 2 +-
README.md | 12 ++--
centos.sh | 4 ++--
cmake/DockerConfig.cmake| 4 ++--
docker/DockerBuild.sh | 17 +
docker/centos/Dockerfile| 5 ++---
docker/debian/Dockerfile| 8 ++--
docker/fedora/Dockerfile| 2 +-
docker/{xenial => focal}/Dockerfile | 5 +++--
9 files changed, 28 insertions(+), 31 deletions(-)
diff --git a/.gitignore b/.gitignore
index 5926a60..60cce20 100644
--- a/.gitignore
+++ b/.gitignore
@@ -67,8 +67,8 @@ __pycache__/
# Ignore source files that have been placed in the docker directory during
build
docker/minificppsource
-docker/xenial/minificppsource
docker/bionic/minificppsource
+docker/focal/minificppsource
docker/debian/minificppsource
docker/centos/minificppsource
docker/fedora/minificppsource
diff --git a/README.md b/README.md
index 101280b..4e8a5f6 100644
--- a/README.md
+++ b/README.md
@@ -490,17 +490,17 @@ $ make docker-verify
```
### Building For Other Distros
-If you have docker installed on your machine you can build for CentOS 7,
Fedora 29, Ubuntu 16, Ubuntu 18, and Debian 9 via our make docker commands. The
following table
+If you have docker installed on your machine you can build for CentOS 8,
Fedora 34, Ubuntu 18.04, Ubuntu 20.04, and Debian 10 via our make docker
commands. The following table
provides the command to build your distro and the output file in your build
directory. Since the versions are limited ( except for Ubuntu ) we output the
archive based on the distro's name.
| Distro | command | Output File |
| - |:-| :-|
-| CentOS 7 | make centos | nifi-minifi-cpp-centos-$VERSION-bin.tar.gz
-| Debian 9 | make debian | nifi-minifi-cpp-debian-$VERSION-bin.tar.gz
-| Fedora 29 | make fedora | nifi-minifi-cpp-fedora-$VERSION-bin.tar.gz
-| Ubuntu 16 | make u16 | nifi-minifi-cpp-xenial-$VERSION-bin.tar.gz
-| Ubuntu 18 | make u18 | nifi-minifi-cpp-bionic-$VERSION-bin.tar.gz
+| CentOS 8 | make centos | nifi-minifi-cpp-centos-$VERSION-bin.tar.gz
+| Debian 10 (buster) | make debian |
nifi-minifi-cpp-debian-$VERSION-bin.tar.gz
+| Fedora 34 | make fedora | nifi-minifi-cpp-fedora-$VERSION-bin.tar.gz
+| Ubuntu 18.04 (bionic) | make u18 |
nifi-minifi-cpp-bionic-$VERSION-bin.tar.gz
+| Ubuntu 20.04 (focal) | make u20 | nifi-minifi-cpp-focal-$VERSION-bin.tar.gz
### Snapcraft
diff --git a/centos.sh b/centos.sh
index 7eef91e..84b0f03 100644
--- a/centos.sh
+++ b/centos.sh
@@ -138,9 +138,9 @@ build_deps(){
INSTALLED+=("java-1.8.0-openjdk-devel")
INSTALLED+=("maven")
elif [ "$FOUND_VALUE" = "python" ]; then
-INSTALLED+=("python34-devel")
+INSTALLED+=("python36-devel")
elif [ "$FOUND_VALUE" = "lua" ]; then
-INSTALLED+=("lua-devel")
+INSTALLED+=("lua-libs")
elif [ "$FOUND_VALUE" = "gpsd" ]; then
INSTALLED+=("gpsd-devel")
elif [ "$FOUND_VALUE" = "libarchive" ]; then
diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake
index e6d2040..5bb4107 100644
--- a/cmake/DockerConfig.cmake
+++ b/cmake/DockerConfig.cmake
@@ -124,7 +124,7 @@ add_custom_target(
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/)
add_custom_target(
-u16
+u20
COMMAND ${CMAKE_SOURCE_DIR}/docker/DockerBuild.sh
-u 1000
-g 1000
@@ -132,7 +132,7 @@ add_custom_target(
-i release
-c ENABLE_JNI=${ENABLE_JNI}
-l ${CMAKE_BINARY_DIR}
--d xenial
+-d focal
-c BUILD_NUMBER=${BUILD_NUMBER}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/)
diff --git a/docker/DockerBuild.sh b/docker/DockerBuild.sh
index 5f954d7..4641072 100755
--- a/docker/DockerBuild.sh
+++ b/docker/DockerBuild.sh
@@ -36,7 +36,7 @@ function usage {
echo "-i, --image-type Can be release or minimal (default: release)"
echo "-u, --uid User id to be used in the Docker image (defa
svn commit: r48033 - /release/nifi/KEYS
Author: aboda Date: Tue Jun 1 10:57:14 2021 New Revision: 48033 Log: Adding Ferenc Gerlits and Adam Debreceni Modified: release/nifi/KEYS Modified: release/nifi/KEYS == --- release/nifi/KEYS (original) +++ release/nifi/KEYS Tue Jun 1 10:57:14 2021 @@ -1722,3 +1722,136 @@ J2LnyOSBJXeH2UleGTF/QNxXW1FzhDN75/I0EAY9 UkE= =5IZc -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2021-01-20 [SC] [expires: 2023-01-20] + EF3757EEF7D8FBD652284EC5576CB454DDD6500D +uid [ultimate] Adam Debreceni +sig 3576CB454DDD6500D 2021-01-21 Adam Debreceni +uid [ultimate] Adam Debreceni +sig 3576CB454DDD6500D 2021-01-20 Adam Debreceni +sub rsa4096 2021-01-20 [E] [expires: 2023-01-20] +sig 576CB454DDD6500D 2021-01-20 Adam Debreceni + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBGAH/CABEADM+g8UUTuAgXEqKY0jYme4SH6PmBWRbkTHIh8l6SpKBMPATOl7 +hX6Tlzlv/8JchS89CgqBSwAFAmifO7uBht3rICMnZSHzgZDHD73duC3YZpHjXxmj +zyT6CIUL2LviRU/sVUO2oVcSWPXt9rDQUJo1u+7ZrLa5zjJtW6VMceYFYlnXa21w +emEjd0oooPW9s/KOXBBTR0qGjJo6ukiiZFWH6fSOmSuzqyhrwgDZsktAsF3GsAz3 +QpR5zgEwHxBZdthGiJrrsa5I53ULTvDY6CtgwIomgKf+M6lZGKiccCZGrEIlISOG +854XKqt9Dw+vlK77unkgkQmJe/0EBcGCjIkAAQvBM/WE0yYFj9AFVciRSHj3whqa +xwOhdFUykMuNtDQSdpKd/0vyWGhGM2KweSo/7C4JAOlGE1WiiE0v1eJZNFS6vN00 +5d/O4Da0QQQpLWvkuXtsoXM02fvfUSDyBj7nks5+23cOJO5pTvG1a6oq3kQH/hVL +bpy0x/g57vYpFfy1jn0glkdTypvDCKZ3xiJScDpCr4dM0VsGWB7AxZIgULzZXOWk +5XF0VQ5dJFpciL0Zy+8Mbn7N3bUGiKOrawGyb3B3FWykpzSMW2Y9HLMZjcj60d+0 +u+kR6A+varjKDjOTAHHpL3neNJ7LTbtBFc4hnMr8oQmbD+X6J6unCoWL6wARAQAB +tC5BZGFtIERlYnJlY2VuaSA8YWRhbS5kZWJyZWNlbmlAcHJvdG9ubWFpbC5jb20+ +iQJUBBMBCAA+FiEE7zdX7vfY+9ZSKE7FV2y0VN3WUA0FAmAH/CACGwMFCQPCZwAF +CwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQV2y0VN3WUA3AJA//SVIquVis/ZxL +Kgg1tLTKY/DgTVphlDjtS4klxf9vDmyC0QmXuKIdmunmAVhijFKc/hup7NikYuSz +0+MEpeDoiCV31BW3AqVhEoX9jbHYIHCZNn0UlnZlg/EYZT3LcaN7w33pwRLT/Hzu +uOoJQIdK3WpDaTpnBehk9LaeLPJDnElgwUSjCZpp/3fBMWCA9HmuVY1UbRxEcYyP +yfTNNgq/lfmVTvjeZSdhL6ohs8AaL1S/hFMRQj3AvYUgFlvFc9zO06xIynwTRw9i +qjm4SUbJZfcmRzmPOMumWyExLK+fr2R8Gom0L028L4L8Ds787MIHlKtSOIRvxN2k +G1pTpzl9I+Vmx9oG3YdfPRa3aE3CNooEO9gPquT8kJWEsItrmqbS9YA2d/E4uYws +KZSyBoTQRf4ADvmCkXrlJwJfyfxtDZQGTC49ZucQmeqL5CVctoFvanYQeyq19t4w +v8D8L7wlCD+d9qninJzahaES1BUCNHd5/4oDKFhLJH3pLqzw08r2KLL5whsfEEdH ++Z2lZKUoWIkYqsFNV1EWib8u+lEnR2l19pWWgUEjlrSamo1RbNPAZbMX9VClyMqh +HA1K33Aq08yx5JEV5kBBtIRt5E0tUCv85JyZ7tX1cwyfSlGD66wO1gBfjgb3uemm +yATBprzJGrbvY4TXon5QqAABWOZr+Py0JkFkYW0gRGVicmVjZW5pIDxhZGVicmVj +ZW5pQGFwYWNoZS5vcmc+iQJUBBMBCAA+FiEE7zdX7vfY+9ZSKE7FV2y0VN3WUA0F +AmAJMLwCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQV2y0VN3W +UA1JGhAAjjRrZNaO4tqfJOPD3u0TOEfwjNFR2BTASMGEk7bJYYW4M131aTBUWFn/ +aBhRTgGjACwV50Hpiss05ka1F4QDBE00xj80Uk6rEld6tVxO+j38IV8wDSGmAR/H +GCLv2FGzOFB6QqvBLelIisns5Nts3+40hJx8gsH+kTLmQcPSybkySSsu2bkcNJa8 +WKONP39nqZAL9DhWqRHnSl+wwL1jwQdXpu6pjnhUu6u2LB3EKSiW3LpFIaHrL2qk +zn+02kNeeKPaRRGOVQRvyqYuL36K5Wfg1j+V1yft8QywczquRAISP/E1RkifPSZ0 +Z6CkPbFvm/FwHLfyVxbYhI0V/KXt2NMzK2Ux0P1x/dvRh0eK+EyvmYHTRQa8J05B +hmF5BL0TIkZJOx5ZvQQkS0PHGtEasCnEEHhfFzuNGA0ZIRaCzmzAFa18mSHTSLRv +tZTO0TVkA84FoNLJ4gdIiIeJFlK7Xc//GvFlcD819UdyXxmZzWYt77MC78Jfvku+ +v8lvqzq1U8ihnhUzHHicXRQrcyDAxEyC39LtDYLxjE3Svy3osKDgr6XKPJvBXfil +HvNcFXx5KRrHvdnhyQN7KjTgdZMw4r1PcUfh5oNIGh5t3Hrcgx2BZL4RXFFIQJcl +PBcQdVaH3HlTLSECZnp0mWKmLNTZGhA9yaGJfj3cuecfj+m/bb25Ag0EYAf8IAEQ +ANOdFTyfj5Snnk/pJCi9QeF+2Zndt2DV0ROdltm3xY6Gi42zxxCjn1WUkZdOwDfo +RMSUnMvNWM6/KPxa3I8ldh7ie1eiNvCagw6i6P/fy6nVrFSvmAOSNkqflNGZ2H+e +1LlhXQsmcqWlyHevSWF8vSEeFRVFdLzJOBQAO3IkWXUnY/LHA2vnq9VX4mGrLTmG ++oxO+fYYvAFKVv0EFiqNfcHepA51RrP92ETEac3seKzvcbyH0YwFWbiT7vNyjAhY +ZCKTOQXcEYYJ3/N/0Q45KeMvYwhYFP0F0XdF2igi8vLNjmpB9lOKqIrPbTouI6ER +gasMXeBxMkZQ2NNWzfP3AetjEMZJjbU2vjye9L8hJR2HJTXqBajICCKB1VIiIMwQ +AIXOCzogC6/iLDhQTiflfcmHlzeEt1yhoGtN3DsOKht8LG5VA6XV5y0LS1PTGLx2 +KVNqz9pxju8yt59Ribw9nkDBH5vA7dJgcWiOIlTHt/3A+ab78ce/iptWqyYqM4Fm +x0fQGzkFhzZzJYxMF5ofJxS7DyNW0lNGg3b7l8/LpsVR86p3gSEOeGOHQTPnLQLL +ThTpd+uLMp+zMlm/bnoUGjQNuYk7COV2owqwsqgYPIL/Q5GL0mnJ0dVJCFgjDhLk +ZrL7MU8ndClu2SuaNWljCc7UnJDXKlDLBIbdjF/dxMJxABEBAAGJAjwEGAEIACYW +IQTvN1fu99j71lIoTsVXbLRU3dZQDQUCYAf8IAIbDAUJA8JnAAAKCRBXbLRU3dZQ +DcJAD/9nK8bOALnYuqUTjb7AEY0O8R8mWdxfye8ybvm8LRy5r4MUNvdAw7Cpa5EL +32PkmQ94XA+pM2NnRyo6Q6SWBnwDdueH0JRRBScqHAp8/P3W2Xa+N4OFqw55fwBS +fxLW9PltHM7ejrU5AH65Dvs5T6MDhdrEITRwzCTt8hzRHsH2X4YxK5H8LS9WCsa+ +oG5HnX6Y2P7Buyfa1ScqlAM8CGj2gULDPkZcTpgJf11YlNS0o/430RalQyi7+r/L +F+46T8LzxfRRZrw064meLNse3BBJfamGLnS2lJmwdNgjHbL6KH1Z+2Vl6/F9i7mT +bYXWtVpG0t5kvqKBTu7XCGLOwTmVeR3FtGz6+hZPInRSBlrJ8pSNLCLVWLPQwqQd +qA0W6UdWyRMbv2egs0GmMib5iJlMkdyfXlea6d5Ha1EFp/ZEOmLCKBhgkqlxWxol +8OoFvO7EsoGiS1VpwYfXDbRV1jUy+sAM637CfNYaWFhsl/KsjOUxuhLzcI8A5dK2 +WoLJFe0eVD7F5fMv2AmlMVTsu90YwDCi3RmaYfXMqInzoInjZ31P0rqm0loJAAMm +CRzjcdrKutLHQ6z7nSayCz2cpZaN9ogoTsBc4X6Y/WlJN6XdJlYp6QDmJYp+rtoP +2rSElbZPmVnCpbKnQlp5EtPNAv9YBC744D7Efgj2uI8Q7se7CA== +=yOyR +-END PGP PUBLIC KEY BLOCK- +pub rsa4096 2021-03-22 [SC
[nifi] branch main updated (e0a1cb8 -> 96d5211)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git. from e0a1cb8 NIFI-8133 Add ability to supress null/empty values in ElasticSearchClientService for PutElasticsearchRecord output add 96d5211 Fixing keys of Adam Debreceni and Ferenc Gerlits No new revisions were added by this update. Summary of changes: KEYS | 4 1 file changed, 4 insertions(+)
[nifi-minifi-cpp] branch main updated: MINIFICPP-1575 - Make linter creation configurable
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 88deb7a MINIFICPP-1575 - Make linter creation configurable
88deb7a is described below
commit 88deb7a0e4672ca3a66d9698022ae21906e1f817
Author: Adam Debreceni
AuthorDate: Mon May 31 13:41:12 2021 +0200
MINIFICPP-1575 - Make linter creation configurable
Signed-off-by: Arpad Boda
This closes #1091
---
.github/workflows/ci.yml | 2 +-
CMakeLists.txt | 1 +
cmake/Extensions.cmake | 10 ++
win_build_vs.bat | 4 +++-
4 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d330b4a..489fed1 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -152,7 +152,7 @@ jobs:
run: |
PATH %PATH%;C:\Program Files (x86)\Windows
Kits\10\bin\10.0.19041.0\x64
PATH %PATH%;C:\Program Files (x86)\Microsoft Visual
Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn
- win_build_vs.bat build /2019 /64 /CI /S /A /PDH /T
+ win_build_vs.bat build /2019 /64 /CI /S /A /PDH /T /L
shell: cmd
- name: test
run: cd build && ctest --timeout 300 --parallel 8 -C Release
--output-on-failure
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 8c80382..ab21b88 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -37,6 +37,7 @@ include(ExternalProject)
list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")
include(WholeArchive)
+option(ENABLE_LINTER "Create linter components" ON)
option(SKIP_TESTS "Skips building all tests." OFF)
option(PORTABLE "Instructs the compiler to remove architecture specific
optimizations" ON)
option(USE_SHARED_LIBS "Builds using shared libraries" ON)
diff --git a/cmake/Extensions.cmake b/cmake/Extensions.cmake
index 774a001..2475906 100644
--- a/cmake/Extensions.cmake
+++ b/cmake/Extensions.cmake
@@ -68,10 +68,12 @@ endfunction()
macro(register_extension_linter target-name)
- get_property(extensions GLOBAL PROPERTY EXTENSION-LINTERS)
- set_property(GLOBAL APPEND PROPERTY EXTENSION-LINTERS "${target-name}")
- add_custom_target(${target-name}
- COMMAND python
${CMAKE_SOURCE_DIR}/thirdparty/google-styleguide/run_linter.py -q -i
${CMAKE_CURRENT_LIST_DIR}/)
+ if (ENABLE_LINTER)
+get_property(extensions GLOBAL PROPERTY EXTENSION-LINTERS)
+set_property(GLOBAL APPEND PROPERTY EXTENSION-LINTERS "${target-name}")
+add_custom_target(${target-name}
+COMMAND python
${CMAKE_SOURCE_DIR}/thirdparty/google-styleguide/run_linter.py -q -i
${CMAKE_CURRENT_LIST_DIR}/)
+ endif()
endmacro()
# ARGN WILL be the
diff --git a/win_build_vs.bat b/win_build_vs.bat
index 952e3af..b710792 100755
--- a/win_build_vs.bat
+++ b/win_build_vs.bat
@@ -36,6 +36,7 @@ set cpack=OFF
set installer_merge_modules=OFF
set strict_gsl_checks=
set redist=
+set build_linter=OFF
set arg_counter=0
for %%x in (%*) do (
@@ -58,12 +59,13 @@ for %%x in (%*) do (
if [%%~x] EQU [/DD] set cmake_build_type=Debug
if [%%~x] EQU [/CI] set
"strict_gsl_checks=-DSTRICT_GSL_CHECKS=AUDIT" & set test_custom_wel_provider=ON
if [%%~x] EQU [/NONFREEUCRT] set "redist=-DMSI_REDISTRIBUTE_UCRT_NONASL=ON"
+if [%%~x] EQU [/L] set build_linter=ON
)
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform%
-DINSTALLER_MERGE_MODULES=%installer_merge_modules%
-DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL%
-DCMAKE_BUILD_TYPE_INIT=%cmake_build_type%
-DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32
-DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF
-DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH%
-DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_L [...]
+cmake -G %generator% -A %build_platform%
-DINSTALLER_MERGE_MODULES=%installer_merge_modules%
-DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL%
-DCMAKE_BUILD_TYPE_INIT=%cmake_build_type%
-DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32
-DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF
-DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH%
-DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_L [...]
IF %ERRORLEVEL% NEQ 0 EXIT /b %ERRORLEVEL%
if [%cpack%] EQU [ON] (
cpack -C %cmake_build_type%
[nifi-minifi-cpp] branch main updated: MINIFICPP-1559 Add new redistributable file to the msi in VS2019 builds
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new ff31971 MINIFICPP-1559 Add new redistributable file to the msi in
VS2019 builds
ff31971 is described below
commit ff3197165122b91b3f77a42c3036d355f3844ac6
Author: Ferenc Gerlits
AuthorDate: Thu May 13 18:42:30 2021 +0200
MINIFICPP-1559 Add new redistributable file to the msi in VS2019 builds
add msvcp140_atomic_wait.dll and msvcp140_codecvt_ids.dll
... and remove vcruntime140_1.dll in 32-bit VS2019 builds
Signed-off-by: Arpad Boda
This closes #1080
---
CMakeLists.txt | 18 +-
msi/WixWin.wsi | 49 +
2 files changed, 26 insertions(+), 41 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 5fc3273..8c80382 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -671,8 +671,8 @@ if(WIN32)
list(SORT VCRUNTIME_REDIST_VERSIONS)
list(REVERSE VCRUNTIME_REDIST_VERSIONS)
list(GET VCRUNTIME_REDIST_VERSIONS 0 VCRUNTIME_REDIST_DIR)
- message("Using redist directory: ${VCRUNTIME_REDIST_DIR}")
endif()
+ message("Using redist directory: ${VCRUNTIME_REDIST_DIR}")
if (MSI_REDISTRIBUTE_UCRT_NONASL)
set(UCRT_DIR_NAT
"$ENV{WindowsSdkDir}Redist\\ucrt\\DLLs\\$ENV{Platform}")
@@ -733,20 +733,12 @@ if(WIN32)
if(CMAKE_SIZEOF_VOID_P EQUAL 8)
message("Using ${VCRUNTIME_X64_REDIST_CRT_DIR} VC
Redistributables")
- file(COPY
"${VCRUNTIME_X64_REDIST_CRT_DIR}/concrt140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X64_REDIST_CRT_DIR}/msvcp140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X64_REDIST_CRT_DIR}/msvcp140_1.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X64_REDIST_CRT_DIR}/msvcp140_2.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X64_REDIST_CRT_DIR}/vccorlib140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X64_REDIST_CRT_DIR}/vcruntime140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
+ file(GLOB VCRUNTIME_X64_REDIST_CRT_FILES
"${VCRUNTIME_X64_REDIST_CRT_DIR}/*.dll")
+ file(COPY ${VCRUNTIME_X64_REDIST_CRT_FILES} DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
elseif(CMAKE_SIZEOF_VOID_P EQUAL 4)
message("Using ${VCRUNTIME_X86_REDIST_CRT_DIR} VC
Redistributables")
- file(COPY
"${VCRUNTIME_X86_REDIST_CRT_DIR}/concrt140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X86_REDIST_CRT_DIR}/msvcp140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X86_REDIST_CRT_DIR}/msvcp140_1.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X86_REDIST_CRT_DIR}/msvcp140_2.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X86_REDIST_CRT_DIR}/vccorlib140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
- file(COPY
"${VCRUNTIME_X86_REDIST_CRT_DIR}/vcruntime140.dll" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
+ file(GLOB VCRUNTIME_X86_REDIST_CRT_FILES
"${VCRUNTIME_X86_REDIST_CRT_DIR}/*.dll")
+ file(COPY ${VCRUNTIME_X86_REDIST_CRT_FILES} DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}/redist")
else()
message(FATAL_ERROR "Could not determine architecture,
CMAKE_SIZEOF_VOID_P is unexpected: ${CMAKE_SIZEOF_VOID_P}")
endif()
diff --git a/msi/WixWin.wsi b/msi/WixWin.wsi
index fbac160..411e52c 100644
--- a/msi/WixWin.wsi
+++ b/msi/WixWin.wsi
@@ -44,22 +44,16 @@ Licensed to the Apache Software Foundation (ASF) under one
or more
-
-
-
+
+
+
+
@@ -70,7 +64,6 @@ Licensed to the Apache Software Foundation (ASF) under one or
more
-
@@ -129,8 +122,6 @@ Licensed to the Apache Software Foundation (ASF) under one
or more
-
-
@@ -254,11 +245,8 @@ Licensed to the Apache Software Foundation (ASF) under one
or more
-->
-
-
-
@@ -318,12 +306,6 @@ Licensed to the Apache Software Foundation (ASF) under one
or more
-
-
-
-
-
-
[nifi-minifi-cpp] branch main updated (cb4b2be -> d3b149c)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from cb4b2be MINIFICPP-1564 - Remove agent update add d3b149c MINIFICPP-1565 - Minor improvements to PerformanceDataMonitor No new revisions were added by this update. Summary of changes: PROCESSORS.md | 103 + extensions/pdh/PDHCounters.cpp | 12 +-- extensions/pdh/PerformanceDataMonitor.cpp | 100 extensions/pdh/PerformanceDataMonitor.h| 14 ++- .../pdh/tests/PerformanceDataMonitorTests.cpp | 23 - libminifi/include/utils/JsonCallback.h | 25 - 6 files changed, 248 insertions(+), 29 deletions(-)
[nifi-minifi-cpp] branch main updated (cde679f -> cb4b2be)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from cde679f MINIFICPP-1563 fix -Wmaybe-uninitialized warnings add cb4b2be MINIFICPP-1564 - Remove agent update No new revisions were added by this update. Summary of changes: extensions/http-curl/tests/C2UpdateAgentTest.cpp | 40 --- extensions/http-curl/tests/CMakeLists.txt| 1 - extensions/http-curl/tests/HTTPIntegrationBase.h | 22 --- libminifi/include/c2/C2Agent.h | 10 --- libminifi/src/c2/C2Agent.cpp | 84 5 files changed, 157 deletions(-) delete mode 100644 extensions/http-curl/tests/C2UpdateAgentTest.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1532: Create a processor to capture resource consumption metrics on Windows
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 98b9055 MINIFICPP-1532: Create a processor to capture resource
consumption metrics on Windows
98b9055 is described below
commit 98b9055fda485f87cdc6e161c29ed557f063846a
Author: Martin Zink
AuthorDate: Wed Mar 24 12:29:15 2021 +0100
MINIFICPP-1532: Create a processor to capture resource consumption metrics
on Windows
Updated documentation to include information about the new processor
Signed-off-by: Arpad Boda
This closes #1066
---
.github/workflows/ci.yml | 4 +-
CMakeLists.txt | 18 +-
PROCESSORS.md | 22 ++
README.md | 1 +
Windows.md | 1 +
extensions/pdh/CMakeLists.txt | 32 +++
extensions/pdh/MemoryConsumptionCounter.h | 70 +
extensions/pdh/PDHCounters.cpp | 132 ++
extensions/pdh/PDHCounters.h | 99 +++
extensions/pdh/PerformanceDataCounter.h| 51
extensions/pdh/PerformanceDataMonitor.cpp | 284 +
extensions/pdh/PerformanceDataMonitor.h| 90 +++
extensions/pdh/tests/CMakeLists.txt| 37 +++
.../pdh/tests/PerformanceDataCounterTests.cpp | 187 ++
.../pdh/tests/PerformanceDataMonitorTests.cpp | 231 +
libminifi/include/utils/JsonCallback.h | 54
libminifi/include/utils/OsUtils.h | 5 +
libminifi/src/utils/OsUtils.cpp| 10 +
libminifi/test/unit/MemoryUsageTest.cpp| 35 ++-
win_build_vs.bat | 3 +-
20 files changed, 1347 insertions(+), 19 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 83e7e86..916e6fa 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -102,7 +102,7 @@ jobs:
run: |
PATH %PATH%;C:\Program Files (x86)\Windows
Kits\10\bin\10.0.17763.0\x86
PATH %PATH%;C:\Program Files (x86)\Microsoft Visual
Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn
- win_build_vs.bat build /CI /S /A /Z
+ win_build_vs.bat build /CI /S /A /Z /PDH
shell: cmd
windows_VS2019:
name: "windows-vs2019"
@@ -141,7 +141,7 @@ jobs:
run: |
PATH %PATH%;C:\Program Files (x86)\Windows
Kits\10\bin\10.0.19041.0\x64
PATH %PATH%;C:\Program Files (x86)\Microsoft Visual
Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn
- win_build_vs.bat build /2019 /64 /CI /S /A
+ win_build_vs.bat build /2019 /64 /CI /S /A /PDH
shell: cmd
ubuntu_16_04:
name: "ubuntu-16.04"
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6a222db..136796d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -493,7 +493,7 @@ if (ENABLE_TENSORFLOW)
createExtension(TENSORFLOW-EXTENSIONS "TENSORFLOW EXTENSIONS" "This
enables TensorFlow support" "extensions/tensorflow"
"${TEST_DIR}/tensorflow-tests")
endif()
-## AWS Extentions
+## AWS Extensions
option(ENABLE_AWS "Enables AWS support." OFF)
if (ENABLE_ALL OR ENABLE_AWS)
include(BundledAwsSdkCpp)
@@ -501,7 +501,15 @@ if (ENABLE_ALL OR ENABLE_AWS)
createExtension(AWS-EXTENSIONS "AWS EXTENSIONS" "This enables AWS
support" "extensions/aws" "${TEST_DIR}/aws-tests")
endif()
-## OpenCV Extesions
+## PDH Extentsions
+if (WIN32)
+ option(ENABLE_PDH "Enables PDH support." OFF)
+ if (ENABLE_PDH)
+ createExtension(PDH-EXTENSIONS "PDH EXTENSIONS" "This enables
PDH support" "extensions/pdh" "extensions/pdh/tests")
+ endif()
+endif()
+
+## OpenCV Extensions
option(ENABLE_OPENCV "Disables the OpenCV extensions." OFF)
if (ENABLE_OPENCV)
createExtension(OPENCV-EXTENSIONS "OPENCV EXTENSIONS" "This enabled
OpenCV support" "extensions/opencv" "extensions/opencv/tests")
@@ -515,7 +523,7 @@ if (ENABLE_BUSTACHE)
createExtension(BUSTACHE-EXTENSIONS "BUSTACHE EXTENSIONS" "This enables
bustache functionality including ApplyTemplate." "extensions/bustache"
"${TEST_DIR}/bustache-tests")
endif()
-## OPC Extentions
+## OPC Extensions
if (ENABLE_OPC)
include(BundledMbedTLS)
use_bundled_mbedtls(${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR})
@@ -546,7 +554,7 @@ if ((ENABLE_OPENWSMAN AND NOT DISABLE_CIVET AND NOT
[nifi-minifi-cpp] branch main updated (4fc6f49 -> cf4dbd9)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 4fc6f49 MINIFICPP-1504 followup review changes new f285b88 MINIFICPP-1553 Support credential refresh in AWSCredentialsService new cf4dbd9 MINIFICPP-1554 - Log c2 response code The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../controllerservices/AWSCredentialsService.cpp | 26 ++--- .../aws/controllerservices/AWSCredentialsService.h | 14 ++--- extensions/http-curl/protocols/RESTSender.cpp | 9 ++- extensions/mqtt/protocol/MQTTC2Protocol.cpp| 2 +- .../test/aws-tests/AWSCredentialsServiceTest.cpp | 66 ++ 5 files changed, 100 insertions(+), 17 deletions(-) create mode 100644 libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
[nifi-minifi-cpp] 02/02: MINIFICPP-1554 - Log c2 response code
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit cf4dbd926b70fb61826c77290ea12211c151555b
Author: Adam Debreceni
AuthorDate: Tue May 4 16:09:36 2021 +0200
MINIFICPP-1554 - Log c2 response code
Signed-off-by: Arpad Boda
This closes #1068
---
extensions/http-curl/protocols/RESTSender.cpp | 9 -
extensions/mqtt/protocol/MQTTC2Protocol.cpp | 2 +-
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/extensions/http-curl/protocols/RESTSender.cpp
b/extensions/http-curl/protocols/RESTSender.cpp
index 0f831e3..7f41c4a 100644
--- a/extensions/http-curl/protocols/RESTSender.cpp
+++ b/extensions/http-curl/protocols/RESTSender.cpp
@@ -38,7 +38,7 @@ namespace c2 {
RESTSender::RESTSender(const std::string , const utils::Identifier )
: C2Protocol(name, uuid),
- logger_(logging::LoggerFactory::getLogger()) {
+ logger_(logging::LoggerFactory::getLogger()) {
}
void RESTSender::initialize(core::controller::ControllerServiceProvider*
controller, const std::shared_ptr ) {
@@ -139,6 +139,13 @@ const C2Payload RESTSender::sendPayload(const std::string
url, const Direction d
}
bool isOkay = client.submit();
int64_t respCode = client.getResponseCode();
+ const bool clientError = 400 <= respCode && respCode < 500;
+ const bool serverError = 500 <= respCode && respCode < 600;
+ if (clientError || serverError) {
+logger_->log_error("Error response code '" "%" PRId64 "' from '%s'",
respCode, url);
+ } else {
+logger_->log_debug("Response code '" "%" PRId64 "' from '%s'", respCode,
url);
+ }
auto rs = client.getResponseBody();
if (isOkay && respCode) {
if (payload.isRaw()) {
diff --git a/extensions/mqtt/protocol/MQTTC2Protocol.cpp
b/extensions/mqtt/protocol/MQTTC2Protocol.cpp
index 79acf77..d50f8cb 100644
--- a/extensions/mqtt/protocol/MQTTC2Protocol.cpp
+++ b/extensions/mqtt/protocol/MQTTC2Protocol.cpp
@@ -25,7 +25,7 @@ namespace c2 {
MQTTC2Protocol::MQTTC2Protocol(const std::string& name, const
utils::Identifier& uuid)
: C2Protocol(name, uuid),
- logger_(logging::LoggerFactory::getLogger()) {
+ logger_(logging::LoggerFactory::getLogger()) {
}
MQTTC2Protocol::~MQTTC2Protocol() = default;
[nifi-minifi-cpp] 01/02: MINIFICPP-1553 Support credential refresh in AWSCredentialsService
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit f285b88da04c2895d69f292c95518ef6579958d3
Author: Gabor Gyimesi
AuthorDate: Fri Apr 30 17:30:44 2021 +0200
MINIFICPP-1553 Support credential refresh in AWSCredentialsService
Add test for credential refresh
Signed-off-by: Arpad Boda
This closes #1067
---
.../controllerservices/AWSCredentialsService.cpp | 26 ++---
.../aws/controllerservices/AWSCredentialsService.h | 14 ++---
.../test/aws-tests/AWSCredentialsServiceTest.cpp | 66 ++
3 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/extensions/aws/controllerservices/AWSCredentialsService.cpp
b/extensions/aws/controllerservices/AWSCredentialsService.cpp
index 2f9477e..439f57b 100644
--- a/extensions/aws/controllerservices/AWSCredentialsService.cpp
+++ b/extensions/aws/controllerservices/AWSCredentialsService.cpp
@@ -60,16 +60,26 @@ void AWSCredentialsService::initialize() {
}
void AWSCredentialsService::onEnable() {
- getProperty(AccessKey.getName(), access_key_);
- getProperty(SecretKey.getName(), secret_key_);
- getProperty(CredentialsFile.getName(), credentials_file_);
- getProperty(UseDefaultCredentials.getName(), use_default_credentials_);
+ std::string value;
+ getProperty(AccessKey.getName(), value);
+ aws_credentials_provider_.setAccessKey(value);
+ getProperty(SecretKey.getName(), value);
+ aws_credentials_provider_.setSecretKey(value);
+ getProperty(CredentialsFile.getName(), value);
+ aws_credentials_provider_.setCredentialsFile(value);
+ bool use_default_credentials = false;
+ getProperty(UseDefaultCredentials.getName(), use_default_credentials);
+ aws_credentials_provider_.setUseDefaultCredentials(use_default_credentials);
+}
- aws_credentials_provider_.setAccessKey(access_key_);
- aws_credentials_provider_.setSecretKey(secret_key_);
- aws_credentials_provider_.setCredentialsFile(credentials_file_);
- aws_credentials_provider_.setUseDefaultCredentials(use_default_credentials_);
+Aws::Auth::AWSCredentials AWSCredentialsService::getAWSCredentials() {
+ if (aws_credentials_.IsExpiredOrEmpty()) {
+cacheCredentials();
+ }
+ return aws_credentials_;
+}
+void AWSCredentialsService::cacheCredentials() {
auto aws_credentials_result = aws_credentials_provider_.getAWSCredentials();
if (aws_credentials_result) {
aws_credentials_ = aws_credentials_result.value();
diff --git a/extensions/aws/controllerservices/AWSCredentialsService.h
b/extensions/aws/controllerservices/AWSCredentialsService.h
index 0551b5b..22d29fd 100644
--- a/extensions/aws/controllerservices/AWSCredentialsService.h
+++ b/extensions/aws/controllerservices/AWSCredentialsService.h
@@ -29,6 +29,8 @@
#include "core/logging/LoggerConfiguration.h"
#include "AWSCredentialsProvider.h"
+class AWSCredentialsServiceTestAccessor;
+
namespace org {
namespace apache {
namespace nifi {
@@ -66,16 +68,14 @@ class AWSCredentialsService : public
core::controller::ControllerService {
void onEnable() override;
- Aws::Auth::AWSCredentials getAWSCredentials() {
-return aws_credentials_;
- }
+ Aws::Auth::AWSCredentials getAWSCredentials();
private:
+ friend class ::AWSCredentialsServiceTestAccessor;
+
+ void cacheCredentials();
+
const utils::AWSInitializer& AWS_INITIALIZER = utils::AWSInitializer::get();
- std::string access_key_;
- std::string secret_key_;
- std::string credentials_file_;
- bool use_default_credentials_ = false;
Aws::Auth::AWSCredentials aws_credentials_;
AWSCredentialsProvider aws_credentials_provider_;
};
diff --git a/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
new file mode 100644
index 000..27e4353
--- /dev/null
+++ b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
@@ -0,0 +1,66 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include
+#include
+
+#include "../TestBase.h"
+#include "controllerservices/AWSCredentialsService.h"
+#include &q
[nifi-minifi-cpp] branch main updated: MINIFICPP-1502 Pass name and uuid arguments as const refs
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 4e47492 MINIFICPP-1502 Pass name and uuid arguments as const refs 4e47492 is described below commit 4e474928550dc08ba021359f3eff3d3e54ccd9af Author: Gabor Gyimesi AuthorDate: Mon Apr 26 12:00:27 2021 +0200 MINIFICPP-1502 Pass name and uuid arguments as const refs Signed-off-by: Arpad Boda This closes #1062 --- .../aws/controllerservices/AWSCredentialsService.h | 2 +- .../controllerservices/AzureStorageCredentialsService.h | 2 +- extensions/bustache/ApplyTemplate.h | 2 +- extensions/civetweb/processors/ListenHTTP.h | 2 +- extensions/coap/controllerservice/CoapConnector.h| 2 +- extensions/gps/GetGPS.h | 2 +- extensions/http-curl/client/HTTPClient.cpp | 2 +- extensions/http-curl/client/HTTPClient.h | 2 +- extensions/http-curl/processors/InvokeHTTP.h | 2 +- extensions/http-curl/protocols/AgentPrinter.cpp | 2 +- extensions/http-curl/protocols/AgentPrinter.h| 2 +- extensions/http-curl/protocols/RESTReceiver.cpp | 2 +- extensions/http-curl/protocols/RESTReceiver.h| 2 +- extensions/http-curl/sitetosite/HTTPProtocol.h | 2 +- extensions/http-curl/sitetosite/PeersEntity.h| 4 ++-- extensions/jni/ExecuteJavaControllerService.h| 2 +- extensions/jni/ExecuteJavaProcessor.h| 2 +- extensions/jni/JVMCreator.h | 2 +- extensions/jni/jvm/JavaControllerService.h | 2 +- extensions/libarchive/CompressContent.h | 2 +- extensions/libarchive/FocusArchiveEntry.h| 2 +- extensions/libarchive/ManipulateArchive.h| 2 +- extensions/libarchive/MergeContent.h | 2 +- extensions/libarchive/UnfocusArchiveEntry.h | 2 +- extensions/librdkafka/PublishKafka.h | 2 +- .../mqtt/controllerservice/MQTTControllerService.h | 4 ++-- extensions/mqtt/processors/AbstractMQTTProcessor.h | 2 +- extensions/mqtt/processors/ConsumeMQTT.h | 2 +- extensions/mqtt/processors/ConvertBase.h | 2 +- extensions/mqtt/processors/ConvertHeartBeat.h| 2 +- extensions/mqtt/processors/ConvertJSONAck.h | 2 +- extensions/mqtt/processors/ConvertUpdate.h | 2 +- extensions/mqtt/processors/PublishMQTT.h | 2 +- extensions/mqtt/protocol/MQTTC2Protocol.cpp | 2 +- extensions/mqtt/protocol/MQTTC2Protocol.h| 2 +- extensions/opc/include/fetchopc.h| 2 +- extensions/opc/include/opcbase.h | 2 +- extensions/opc/include/putopc.h | 2 +- extensions/opencv/CaptureRTSPFrame.h | 2 +- extensions/opencv/MotionDetector.h | 2 +- .../processors/SourceInitiatedSubscriptionListener.cpp | 2 +- .../processors/SourceInitiatedSubscriptionListener.h | 2 +- extensions/pcap/CapturePacket.h | 2 +- extensions/rocksdb-repos/DatabaseContentRepository.h | 2 +- extensions/rocksdb-repos/FlowFileRepository.h| 2 +- extensions/rocksdb-repos/ProvenanceRepository.h | 2 +- .../RocksDbPersistableKeyValueStoreService.cpp | 2 +- .../controllers/RocksDbPersistableKeyValueStoreService.h | 2 +- extensions/script/ExecuteScript.h| 2 +- extensions/script/python/ExecutePythonProcessor.h| 2 +- extensions/script/python/PyProcCreator.h | 2 +- extensions/script/python/PythonCreator.h | 2 +- extensions/sensors/GetEnvironmentalSensors.h | 2 +- extensions/sensors/GetMovementSensors.h | 2 +- extensions/sensors/SensorBase.h | 2 +- extensions/sftp/processors/FetchSFTP.cpp | 2 +- extensions/sftp/processors/FetchSFTP.h | 2 +- extensions/sftp/processors/ListSFTP.cpp | 2 +- extensions/sftp/processors/ListSFTP.h| 2 +- extensions/sftp/processors/PutSFTP.cpp | 2 +- extensions/sftp/processors/PutSFTP.h | 2 +- extensions/sftp/processors/SFTPProcessorBase.cpp | 2 +- extensions/sftp/processors/SFTPProcessorBase.h | 2 +- extensions/sql/processors/ExecuteSQL.cpp | 2 +- extensions/sql/processors/ExecuteSQL.h | 2 +- extensions/sql/processors/PutSQL.cpp | 2 +- extensions/sql/processors/PutSQL.h
[nifi-minifi-cpp] branch main updated (828fc38 -> d4cda02)
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git.
from 828fc38 MINIFICPP-1510 Register and fix InvokeHTTPTests
add 3f5e509 MINIFICPP-1544 Add Custom Text property to GenerateFlowFile
processor
add d4cda02 MINIFICPP-1504: Add Resource consumption data to heartbeats
No new revisions were added by this update.
Summary of changes:
PROCESSORS.md | 1 +
.../C2VerifyResourceConsumptionInHeartbeat.cpp | 133 +++
extensions/http-curl/tests/CMakeLists.txt | 1 +
.../processors/GenerateFlowFile.cpp| 19 +++
.../processors/GenerateFlowFile.h | 1 +
.../standard-processors/tests/CMakeLists.txt | 3 +-
.../tests/unit/GenerateFlowFileTests.cpp | 129 +-
libminifi/include/agent/build_description.h| 4 +
libminifi/include/core/PropertyValue.h | 2 +
libminifi/include/core/state/Value.h | 172 +--
.../include/core/state/nodes/AgentInformation.h| 142 ++-
.../include/core/state/nodes/DeviceInformation.h | 165 +-
libminifi/include/utils/OsUtils.h | 13 +-
libminifi/include/utils/ProcessCPUUsageTracker.h | 92 ++
libminifi/include/utils/SystemCPUUsageTracker.h| 128 ++
.../include/utils/{MapUtils.h => ValueCaster.h}| 28 +--
libminifi/include/utils/ValueParser.h | 31
libminifi/src/c2/protocols/RESTProtocol.cpp| 4 +
libminifi/src/core/state/Value.cpp | 1 +
.../nodes/AgentInformation.cpp}| 13 +-
.../src/core/state/nodes/DeviceInformation.cpp | 19 +--
libminifi/src/utils/OsUtils.cpp| 143 +---
libminifi/src/utils/ProcessCPUUsageTracker.cpp | 140 +++
libminifi/src/utils/SystemCPUUsageTracker.cpp | 190 +
libminifi/test/unit/CPUUsageTest.cpp | 90 ++
libminifi/test/unit/MemoryUsageTest.cpp| 27 ++-
libminifi/test/unit/ResponseNodeValueTests.cpp | 187
27 files changed, 1634 insertions(+), 244 deletions(-)
create mode 100644
extensions/http-curl/tests/C2VerifyResourceConsumptionInHeartbeat.cpp
create mode 100644 libminifi/include/utils/ProcessCPUUsageTracker.h
create mode 100644 libminifi/include/utils/SystemCPUUsageTracker.h
copy libminifi/include/utils/{MapUtils.h => ValueCaster.h} (66%)
copy libminifi/src/core/{TypedValues.cpp => state/nodes/AgentInformation.cpp}
(79%)
copy extensions/jni/jvm/JniReferenceObjects.cpp =>
libminifi/src/core/state/nodes/DeviceInformation.cpp (79%)
create mode 100644 libminifi/src/utils/ProcessCPUUsageTracker.cpp
create mode 100644 libminifi/src/utils/SystemCPUUsageTracker.cpp
create mode 100644 libminifi/test/unit/CPUUsageTest.cpp
create mode 100644 libminifi/test/unit/ResponseNodeValueTests.cpp
[nifi-minifi-cpp] branch main updated (016e034 -> 6bc4b90)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 016e034 MINIFICPP-1345 Add flake8 check for python files new 870249f MINIFICPP-1032 Refactor parse_url new 6bc4b90 MINIFICPP-1547 - Change default c2 protocol The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: C2.md | 8 +- bootstrap.sh | 2 +- conf/minifi.properties | 5 +- encrypt-config/tests/resources/minifi.properties | 6 +- ...th-additional-sensitive-props.minifi.properties | 8 +- libminifi/include/RemoteProcessorGroupPort.h | 22 ++- libminifi/include/utils/HTTPClient.h | 24 +++- libminifi/src/c2/C2Agent.cpp | 24 ++-- libminifi/src/utils/HTTPClient.cpp | 155 ++--- .../test/resources/encrypted.minifi.properties | 6 +- libminifi/test/unit/HTTPUtilTests.cpp | 97 - 11 files changed, 220 insertions(+), 137 deletions(-)
[nifi-minifi-cpp] 01/02: MINIFICPP-1032 Refactor parse_url
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 870249f713219cce57f0719efa3e14c6e25e15bf
Author: Ferenc Gerlits
AuthorDate: Tue Apr 6 12:51:29 2021 +0200
MINIFICPP-1032 Refactor parse_url
Modernize the parse_url() function and add error checking.
Signed-off-by: Arpad Boda
This closes #1046
---
libminifi/include/RemoteProcessorGroupPort.h | 22 ++--
libminifi/include/utils/HTTPClient.h | 24 -
libminifi/src/c2/C2Agent.cpp | 16 ++-
libminifi/src/utils/HTTPClient.cpp | 155 ++-
libminifi/test/unit/HTTPUtilTests.cpp| 97 ++---
5 files changed, 198 insertions(+), 116 deletions(-)
diff --git a/libminifi/include/RemoteProcessorGroupPort.h
b/libminifi/include/RemoteProcessorGroupPort.h
index 3c33706..8598103 100644
--- a/libminifi/include/RemoteProcessorGroupPort.h
+++ b/libminifi/include/RemoteProcessorGroupPort.h
@@ -146,23 +146,17 @@ class RemoteProcessorGroupPort : public core::Processor {
*/
void setURL(std::string val) {
auto urls = utils::StringUtils::split(val, ",");
-for (auto url : urls) {
- logger_->log_trace("Parsing %s", url);
- std::string host, protocol;
- int port = -1;
- url = utils::StringUtils::trim(url);
- utils::parse_url(, , , );
- logger_->log_trace("Parsed -%s- %s %s, %d", url, protocol, host, port);
- if (port == -1) {
-if (protocol.find("https") != std::string::npos) {
- port = 443;
-} else if (protocol.find("http") != std::string::npos) {
- port = 80;
-}
+for (const auto& url : urls) {
+ utils::URL parsed_url{utils::StringUtils::trim(url)};
+ if (parsed_url.isValid()) {
+logger_->log_debug("Parsed RPG URL '%s' -> '%s'", url,
parsed_url.hostPort());
+nifi_instances_.push_back({parsed_url.host(), parsed_url.port(),
parsed_url.protocol()});
+ } else {
+logger_->log_error("Could not parse RPG URL '%s'", url);
}
- nifi_instances_.push_back({ host, port, protocol });
}
}
+
void setHTTPProxy(const utils::HTTPProxy ) {
this->proxy_ = proxy;
}
diff --git a/libminifi/include/utils/HTTPClient.h
b/libminifi/include/utils/HTTPClient.h
index 82aa55b..425c716 100644
--- a/libminifi/include/utils/HTTPClient.h
+++ b/libminifi/include/utils/HTTPClient.h
@@ -36,6 +36,7 @@
#include "controllers/SSLContextService.h"
#include "core/Deprecated.h"
#include "utils/gsl.h"
+#include "utils/OptionalUtils.h"
namespace org {
namespace apache {
@@ -362,10 +363,27 @@ class BaseHTTPClient {
virtual inline bool matches(const std::string , const std::string
) = 0;
};
-extern std::string get_token(utils::BaseHTTPClient *client, std::string
username, std::string password);
+std::string get_token(utils::BaseHTTPClient *client, std::string username,
std::string password);
+
+class URL {
+ public:
+ explicit URL(const std::string& url_input);
+ bool isValid() const { return is_valid_; }
+ std::string protocol() const { return protocol_; }
+ std::string host() const { return host_; }
+ int port() const;
+ std::string hostPort() const;
+ std::string toString() const;
+
+ private:
+ std::string protocol_;
+ std::string host_;
+ utils::optional port_;
+ utils::optional path_;
+ bool is_valid_ = false;
+ std::shared_ptr logger_ =
logging::LoggerFactory::getLogger();
+};
-extern void parse_url(const std::string *url, std::string *host, int *port,
std::string *protocol);
-extern void parse_url(const std::string *url, std::string *host, int *port,
std::string *protocol, std::string *path, std::string *query);
} // namespace utils
} // namespace minifi
} // namespace nifi
diff --git a/libminifi/src/c2/C2Agent.cpp b/libminifi/src/c2/C2Agent.cpp
index 0914ed5..9c914cc 100644
--- a/libminifi/src/c2/C2Agent.cpp
+++ b/libminifi/src/c2/C2Agent.cpp
@@ -760,22 +760,20 @@ utils::optional C2Agent::fetchFlow(const
std::string& uri) const {
std::stringstream adjusted_url;
std::string base;
if (configuration_->get(minifi::Configure::nifi_c2_flow_base_url, base)) {
+ base = utils::StringUtils::trim(base);
adjusted_url << base;
if (!utils::StringUtils::endsWith(base, "/")) {
adjusted_url << "/";
}
adjusted_url << uri;
resolved_url = adjusted_url.str();
-} else if (configuration_->get("c2.rest.url", base)) {
- std::string host, protocol;
- int port = -1;
- utils::parse_url(, , , );
- adjusted_url << protocol << host;
- if (port > 0) {
-adjusted_url << ":" << p
[nifi-minifi-cpp] 02/02: MINIFICPP-1547 - Change default c2 protocol
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 6bc4b9057ee841e170bf5087bb86ac379e82a0b8
Author: Adam Debreceni
AuthorDate: Wed Apr 21 15:49:30 2021 +0200
MINIFICPP-1547 - Change default c2 protocol
Signed-off-by: Arpad Boda
This closes #1058
---
C2.md | 8
bootstrap.sh | 2 +-
conf/minifi.properties| 5 +++--
encrypt-config/tests/resources/minifi.properties | 6 +++---
.../resources/with-additional-sensitive-props.minifi.properties | 8
libminifi/src/c2/C2Agent.cpp | 8
libminifi/test/resources/encrypted.minifi.properties | 6 +++---
7 files changed, 22 insertions(+), 21 deletions(-)
diff --git a/C2.md b/C2.md
index 560b8ea..864e5202 100644
--- a/C2.md
+++ b/C2.md
@@ -63,11 +63,11 @@ be requested via C2 DESCRIBE manifest command.
# specify classes for the AST response
nifi.c2.root.classes=DeviceInfoNode,AgentInformation,FlowInformation
- # specify C2 protocol -- default is CoapProtocol if not specified
- #nifi.c2.agent.protocol.class=CoapProtocol
- # may also use MQTT or REST
- # nifi.c2.agent.protocol.class=MQTTC2Protocol
+ # specify C2 protocol -- default is RESTSender if not specified
nifi.c2.agent.protocol.class=RESTSender
+ # may also use MQTT or CoapProtocol
+ # nifi.c2.agent.protocol.class=MQTTC2Protocol
+ # nifi.c2.agent.protocol.class=CoapProtocol
# control c2 heartbeat interval in millisecocnds
nifi.c2.agent.heartbeat.period=250
diff --git a/bootstrap.sh b/bootstrap.sh
index 2401996..34fa8d1 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -292,7 +292,7 @@ add_disabled_option MQTT_ENABLED ${FALSE} "ENABLE_MQTT"
add_disabled_option PYTHON_ENABLED ${FALSE} "ENABLE_PYTHON"
add_dependency PYTHON_ENABLED "python"
-add_disabled_option COAP_ENABLED ${TRUE} "ENABLE_COAP"
+add_disabled_option COAP_ENABLED ${FALSE} "ENABLE_COAP"
add_dependency COAP_ENABLED "automake"
add_dependency COAP_ENABLED "autoconf"
add_dependency COAP_ENABLED "libtool"
diff --git a/conf/minifi.properties b/conf/minifi.properties
index e5f2a1a..66dfbe9 100644
--- a/conf/minifi.properties
+++ b/conf/minifi.properties
@@ -65,8 +65,9 @@ nifi.content.repository.class.name=DatabaseContentRepository
## define those with missing options
#nifi.c2.enable=true
## define protocol parameters
-## The default is CoAP, if that extension is built.
-## Alternatively, you may use RESTSender if http-curl is built
+## The default is RESTSender.
+## Alternatively, you may use CoapProtocol if that extension is built.
+#nifi.c2.agent.protocol.class=RESTSender
#nifi.c2.agent.protocol.class=CoapProtocol
#nifi.c2.agent.coap.host=
#nifi.c2.agent.coap.port=
diff --git a/encrypt-config/tests/resources/minifi.properties
b/encrypt-config/tests/resources/minifi.properties
index f1ca0a2..1cf2549 100644
--- a/encrypt-config/tests/resources/minifi.properties
+++ b/encrypt-config/tests/resources/minifi.properties
@@ -52,10 +52,10 @@ nifi.rest.api.password=password
## define those with missing options
nifi.c2.enable=true
## define protocol parameters
-## The default is CoAP, if that extension is built.
-## Alternatively, you may use RESTSender if http-curl is built
-#nifi.c2.agent.protocol.class=CoapProtocol
+## The default is RESTSender.
+## Alternatively, you may use CoapProtocol if that extension is built.
nifi.c2.agent.protocol.class=RESTSender
+#nifi.c2.agent.protocol.class=CoapProtocol
#nifi.c2.agent.coap.host=
#nifi.c2.agent.coap.port=
## base URL of the c2 server,
diff --git
a/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties
b/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties
index d021f0f..2ca29b9 100644
---
a/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties
+++
b/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties
@@ -29,7 +29,7 @@
nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_reposi
nifi.remote.input.secure=true
nifi.security.need.ClientAuth=
-nifi.security.client.certificate=
+nifi.security.client.certificate=
nifi.security.client.private.key=
nifi.security.client.pass.phrase=correct_horse_battery_staple
nifi.security.client.ca.certificate=
@@ -54,10 +54,10 @@ nifi.rest.api.password=password
## define those with missing options
nifi.c2.enable=true
## define protocol parameters
-## The default is CoAP, if that extension is built.
-## Alternatively, you may use RESTSender i
[nifi-minifi-cpp] 03/03: MINIFICPP-1499 - Add clcache to windows builds
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 2c51f1a53bb225b5f5e05496e339f041fd0237ef
Author: Gabor Gyimesi
AuthorDate: Wed Feb 17 17:39:46 2021 +0100
MINIFICPP-1499 - Add clcache to windows builds
Signed-off-by: Arpad Boda
This closes #1012
---
.github/workflows/ci.yml | 40 +++-
1 file changed, 39 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 4c17288..a9d6197 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -69,9 +69,19 @@ jobs:
name: "windows-vs2017"
runs-on: windows-2016
timeout-minutes: 120
+env:
+ CLCACHE_DIR: ${{ GITHUB.WORKSPACE }}\clcache
steps:
- id: checkout
uses: actions/checkout@v2
+ - id: cache
+uses: actions/cache@v2
+with:
+ path: ${{ env.CLCACHE_DIR }}
+ key: windows-vs2017-clcache-${{github.ref}}-${{github.sha}}
+ restore-keys: |
+windows-vs2017-clcache-${{github.ref}}-
+windows-vs2017-clcache-refs/heads/main-
- name: Setup PATH
uses: microsoft/setup-msbuild@v1.0.2
- id: install-sqliteodbc-driver
@@ -79,6 +89,15 @@ jobs:
Invoke-WebRequest -Uri
"http://www.ch-werner.de/sqliteodbc/sqliteodbc.exe; -OutFile "sqliteodbc.exe"
./sqliteodbc.exe /S
shell: powershell
+ - name: Setup clcache
+run: |
+ (New-Object
System.Net.WebClient).DownloadFile('https://github.com/frerich/clcache/releases/download/v4.2.0/clcache-4.2.0.zip',
"$pwd\clcache.zip")
+ $cl_exe_dir = Split-Path -parent $(vswhere -latest -requires
Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -find
VC\Tools\MSVC\**\bin\Hostx86\x86\cl.exe | select-object -last 1)
+ Expand-Archive -Force -Path clcache.zip -DestinationPath
"$cl_exe_dir";
+ move "$cl_exe_dir\cl.exe" "$cl_exe_dir\cl_original.exe"
+ move "$cl_exe_dir\cl.exe.config" "$cl_exe_dir\cl_original.exe.config"
+ move "$cl_exe_dir\clcache.exe" "$cl_exe_dir\cl.exe"
+ echo "CLCACHE_CL=$cl_exe_dir\cl_original.exe" >> $env:GITHUB_ENV
- id: build
run: |
PATH %PATH%;C:\Program Files (x86)\Windows
Kits\10\bin\10.0.17763.0\x86
@@ -88,10 +107,20 @@ jobs:
windows_VS2019:
name: "windows-vs2019"
runs-on: windows-2019
-timeout-minutes: 90
+timeout-minutes: 120
+env:
+ CLCACHE_DIR: ${{ GITHUB.WORKSPACE }}\clcache
steps:
- id: checkout
uses: actions/checkout@v2
+ - id: cache
+uses: actions/cache@v2
+with:
+ path: ${{ env.CLCACHE_DIR }}
+ key: windows-vs2019-clcache-${{github.ref}}-${{github.sha}}
+ restore-keys: |
+windows-vs2019-clcache-${{github.ref}}-
+windows-vs2019-clcache-refs/heads/main-
- name: Setup PATH
uses: microsoft/setup-msbuild@v1.0.2
- id: install-sqliteodbc-driver
@@ -99,6 +128,15 @@ jobs:
Invoke-WebRequest -Uri
"http://www.ch-werner.de/sqliteodbc/sqliteodbc_w64.exe; -OutFile
"sqliteodbc_w64.exe"
./sqliteodbc_w64.exe /S
shell: powershell
+ - name: Setup clcache
+run: |
+ (New-Object
System.Net.WebClient).DownloadFile('https://github.com/frerich/clcache/releases/download/v4.2.0/clcache-4.2.0.zip',
"$pwd\clcache.zip")
+ $cl_exe_dir = Split-Path -parent $(vswhere -latest -requires
Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -find
VC\Tools\MSVC\**\bin\Hostx64\x64\cl.exe | select-object -last 1)
+ Expand-Archive -Force -Path clcache.zip -DestinationPath
"$cl_exe_dir";
+ move "$cl_exe_dir\cl.exe" "$cl_exe_dir\cl_original.exe"
+ move "$cl_exe_dir\cl.exe.config" "$cl_exe_dir\cl_original.exe.config"
+ move "$cl_exe_dir\clcache.exe" "$cl_exe_dir\cl.exe"
+ echo "CLCACHE_CL=$cl_exe_dir\cl_original.exe" >> $env:GITHUB_ENV
- id: build
run: |
PATH %PATH%;C:\Program Files (x86)\Windows
Kits\10\bin\10.0.19041.0\x64
[nifi-minifi-cpp] branch main updated (a29b414 -> 2c51f1a)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from a29b414 MINIFICPP-1535 - Support process groups new c759c96 MINIFICPP-1539 Check if Message Key Field is really set new 88f5e04 MINIFICPP-1543 Add 'brew update' to MacOS CI jobs new 2c51f1a MINIFICPP-1499 - Add clcache to windows builds The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/ci.yml | 48 +-- extensions/librdkafka/PublishKafka.cpp| 2 +- extensions/librdkafka/tests/PublishKafkaTests.cpp | 16 3 files changed, 62 insertions(+), 4 deletions(-)
[nifi-minifi-cpp] 01/03: MINIFICPP-1539 Check if Message Key Field is really set
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit c759c96c9ab6f59a98a624dd99a9540d73d8d6b4
Author: Ferenc Gerlits
AuthorDate: Fri Apr 9 18:04:47 2021 +0200
MINIFICPP-1539 Check if Message Key Field is really set
Signed-off-by: Arpad Boda
This closes #1049
---
extensions/librdkafka/PublishKafka.cpp| 2 +-
extensions/librdkafka/tests/PublishKafkaTests.cpp | 16
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/extensions/librdkafka/PublishKafka.cpp
b/extensions/librdkafka/PublishKafka.cpp
index ceb76f3..b83d028 100644
--- a/extensions/librdkafka/PublishKafka.cpp
+++ b/extensions/librdkafka/PublishKafka.cpp
@@ -544,7 +544,7 @@ void PublishKafka::onSchedule(const
std::shared_ptr
configureNewConnection(context);
std::string message_key_field;
- if (context->getProperty(MessageKeyField.getName(), message_key_field)) {
+ if (context->getProperty(MessageKeyField.getName(), message_key_field) &&
!message_key_field.empty()) {
logger_->log_error("The %s property is set. This property is DEPRECATED
and has no effect; please use Kafka Key instead.", MessageKeyField.getName());
}
diff --git a/extensions/librdkafka/tests/PublishKafkaTests.cpp
b/extensions/librdkafka/tests/PublishKafkaTests.cpp
index ea1c4ef..db44eb3 100644
--- a/extensions/librdkafka/tests/PublishKafkaTests.cpp
+++ b/extensions/librdkafka/tests/PublishKafkaTests.cpp
@@ -190,3 +190,19 @@ TEST_CASE("PublishKafka dynamic properties can use
expression language", "[Publi
REQUIRE(LogTestController::getInstance().contains("PublishKafka:
DynamicProperty: [retry.backoff.ms] -> [18]"));
}
+
+TEST_CASE("PublishKafka complains if Message Key Field is set, but only if it
is set", "[PublishKafka][properties]") {
+ PublishKafkaTestRunner test_runner;
+
+ SECTION("Message Key Field is not set, so there is no error") {
+test_runner.runProcessor();
+REQUIRE_FALSE(LogTestController::getInstance().contains("error"));
+ }
+
+ SECTION("Message Key Field is set, so there is an error log") {
+
test_runner.setPublishKafkaProperty(processors::PublishKafka::MessageKeyField,
"kafka.key");
+test_runner.runProcessor();
+REQUIRE(LogTestController::getInstance().contains("The " +
processors::PublishKafka::MessageKeyField.getName() +
+" property is set. This property is DEPRECATED and has no effect"));
+ }
+}
[nifi-minifi-cpp] 02/03: MINIFICPP-1543 Add 'brew update' to MacOS CI jobs
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 88f5e04a20606f3e054582f637917d68873bcd43
Author: Ferenc Gerlits
AuthorDate: Mon Apr 12 19:02:35 2021 +0200
MINIFICPP-1543 Add 'brew update' to MacOS CI jobs
... and remove packages which are already installed, because these cause
`brew install` to return with a non-zero status, for some reason.
Signed-off-by: Arpad Boda
This closes #1051
---
.github/workflows/ci.yml | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 306ec15..4c17288 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -20,7 +20,9 @@ jobs:
macos-xcode11.2.1-ccache-${{github.ref}}-
macos-xcode11.2.1-ccache-refs/heads/main-
- id: install_dependencies
-run: brew install ossp-uuid boost flex openssl python lua@5.3 xz
libssh2 ccache sqliteodbc
+run: |
+ brew update
+ brew install ossp-uuid boost flex lua@5.3 ccache sqliteodbc
- id: setup_env
run: |
echo
"PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH"
>> $GITHUB_ENV
@@ -50,7 +52,9 @@ jobs:
macos-xcode12.0-ccache-${{github.ref}}-
macos-xcode12.0-ccache-refs/heads/main-
- id: install_dependencies
-run: brew install ossp-uuid boost flex openssl python lua@5.3 xz
libssh2 ccache sqliteodbc
+run: |
+ brew update
+ brew install ossp-uuid boost flex lua@5.3 ccache sqliteodbc
- id: setup_env
run: |
echo
"PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH"
>> $GITHUB_ENV
[nifi] branch main updated (a44ab0d -> 1497a6e)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git. from a44ab0d NIFI-8397 update to simple-syslog-5424 0.0.16 add 1497a6e Update KEYS No new revisions were added by this update. Summary of changes: KEYS | 129 +++ 1 file changed, 129 insertions(+)
[nifi-minifi-cpp] branch main updated (0a3d5df -> 2b27556)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 0a3d5df MINIFICPP-1533 Update list of dependencies in README file. add 2b27556 MINIFICPP-1520 - Fix PublishKafka properties to support expression language No new revisions were added by this update. Summary of changes: PROCESSORS.md | 8 +- extensions/librdkafka/PublishKafka.cpp| 55 --- extensions/librdkafka/PublishKafka.h | 3 +- extensions/librdkafka/tests/CMakeLists.txt| 3 +- extensions/librdkafka/tests/PublishKafkaTests.cpp | 192 ++ 5 files changed, 232 insertions(+), 29 deletions(-) create mode 100644 extensions/librdkafka/tests/PublishKafkaTests.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1531 Use release build type for Ubuntu 20.04 job
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 4bb1fd9 MINIFICPP-1531 Use release build type for Ubuntu 20.04 job
4bb1fd9 is described below
commit 4bb1fd97f5df103d5bee9a884839b24e2f77a2c5
Author: Gabor Gyimesi
AuthorDate: Thu Mar 18 18:20:23 2021 +0100
MINIFICPP-1531 Use release build type for Ubuntu 20.04 job
Fix exception catch warning in ListS3Tests
Signed-off-by: Arpad Boda
This closes #1034
---
.github/workflows/ci.yml | 4 ++--
controller/Controller.h| 2 +-
extensions/http-curl/tests/HTTPHandlers.h | 2 ++
libminifi/test/aws-tests/ListS3Tests.cpp | 2 +-
libminifi/test/rocksdb-tests/RepoTests.cpp | 1 +
nanofi/include/sitetosite/CPeer.h | 3 +--
6 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 40f2cda..89ab136 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -159,7 +159,7 @@ jobs:
sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
-run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=
-DENABLE_BUSTACHE=ON -DENABLE_SQLITE=ON -DENABLE_PCAP=ON
-DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4 VERBOSE=1 &&
make test ARGS="--timeout 300 -j2 --output-on-failure"
+run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=
-DCMAKE_BUILD_TYPE=Release -DENABLE_BUSTACHE=ON -DENABLE_SQLITE=ON
-DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4
VERBOSE=1 && make test ARGS="--timeout 300 -j2 --output-on-failure"
ubuntu_20_04_all_clang:
name: "ubuntu-20.04-all-clang"
runs-on: ubuntu-20.04
@@ -181,7 +181,7 @@ jobs:
sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev
openjdk-8-jdk maven libusb-1.0-0-dev libpng-dev libgps-dev
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
-run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=
-DENABLE_OPENWSMAN=ON -DENABLE_OPENCV=ON -DENABLE_MQTT=ON -DENABLE_GPS=ON
-DENABLE_USB_CAMERA=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_OPC=ON -DENABLE_SFTP=ON
-DENABLE_MQTT=ON -DENABLE_COAP=ON -DENABLE_PYTHON=ON -DENABLE_SQL=ON
-DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .. && cmake --build .
--parallel 4 && make test ARGS="--timeout 300 -j8 --o [...]
+run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=
-DCMAKE_BUILD_TYPE=Release -DENABLE_OPENWSMAN=ON -DENABLE_OPENCV=ON
-DENABLE_MQTT=ON -DENABLE_GPS=ON -DENABLE_USB_CAMERA=ON -DENABLE_LIBRDKAFKA=ON
-DENABLE_OPC=ON -DENABLE_SFTP=ON -DENABLE_MQTT=ON -DENABLE_COAP=ON
-DENABLE_PYTHON=ON -DENABLE_SQL=ON -DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT
-DFAIL_ON_WARNINGS=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
&& cmake --build . --parallel 4 && make test [...]
ubuntu_16_04_all:
name: "ubuntu-16.04-all"
runs-on: ubuntu-16.04
diff --git a/controller/Controller.h b/controller/Controller.h
index 6c1f77c..0544f50 100644
--- a/controller/Controller.h
+++ b/controller/Controller.h
@@ -146,7 +146,7 @@ int getJstacks(std::unique_ptr socket,
std::ostream ) {
for (uint64_t i = 0; i < size; i++) {
std::string name;
- uint64_t lines;
+ uint64_t lines = 0;
socket->read(name);
socket->read(lines);
for (uint64_t j = 0; j < lines; j++) {
diff --git a/extensions/http-curl/tests/HTTPHandlers.h
b/extensions/http-curl/tests/HTTPHandlers.h
index af9b0a8..76015e4 100644
--- a/extensions/http-curl/tests/HTTPHandlers.h
+++ b/extensions/http-curl/tests/HTTPHandlers.h
@@ -416,6 +416,7 @@ class HeartbeatHandler : public ServerAwareHandler {
}
}
assert(found);
+(void)found; // unused in release builds
}
virtual void handleHeartbeat(const rapidjson::Document& root, struct
mg_connection *) {
@@ -431,6 +432,7 @@ class HeartbeatHandler : public ServerAwareHandler {
rapidjson::Document root;
rapidjson::ParseResult ok = root.Parse(post_data.data(),
post_data.size());
assert(ok);
+ (void)ok; // unused in release builds
std::string operation = root["operation"].GetString();
if (operation == "heartbeat") {
handleHeartbeat(root, conn);
diff --git a/libminifi/test/aws-tests/ListS3Tests.cpp
b/libminifi/test/aws-tests/ListS3Te
[nifi-minifi-cpp] branch main updated: MINIFICPP-1438 Fix transient failures of FlowControllerTests
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 9427bce MINIFICPP-1438 Fix transient failures of FlowControllerTests
9427bce is described below
commit 9427bceeb49d6b2000c9e3ff68f0c72ef4b92f09
Author: Gabor Gyimesi
AuthorDate: Thu Mar 11 10:58:01 2021 +0100
MINIFICPP-1438 Fix transient failures of FlowControllerTests
Signed-off-by: Arpad Boda
This closes #1030
---
libminifi/test/flow-tests/FlowControllerTests.cpp | 31 ---
1 file changed, 16 insertions(+), 15 deletions(-)
diff --git a/libminifi/test/flow-tests/FlowControllerTests.cpp
b/libminifi/test/flow-tests/FlowControllerTests.cpp
index acc7ee8..3e0b5d1 100644
--- a/libminifi/test/flow-tests/FlowControllerTests.cpp
+++ b/libminifi/test/flow-tests/FlowControllerTests.cpp
@@ -154,16 +154,16 @@ TEST_CASE("Flow shutdown waits for a while",
"[TestFlow2]") {
testController.startFlow();
// wait for the source processor to enqueue its flowFiles
- auto flowFilesEnqueued = [&] {return root->getTotalFlowFileCount() == 3;};
- REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{200},
flowFilesEnqueued));
+ auto flowFilesEnqueued = [&] { return root->getTotalFlowFileCount() >= 3; };
+ REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{500},
flowFilesEnqueued));
- REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sourceProc->trigger_count.load() >= 1);
execSinkPromise.set_value();
controller->stop();
- REQUIRE(sourceProc->trigger_count.load() == 1);
- REQUIRE(sinkProc->trigger_count.load() == 3);
+ REQUIRE(sourceProc->trigger_count.load() >= 1);
+ REQUIRE(sinkProc->trigger_count.load() >= 3);
}
TEST_CASE("Flow stopped after grace period", "[TestFlow3]") {
@@ -191,16 +191,16 @@ TEST_CASE("Flow stopped after grace period",
"[TestFlow3]") {
testController.startFlow();
// wait for the source processor to enqueue its flowFiles
- auto flowFilesEnqueued = [&] {return root->getTotalFlowFileCount() == 3;};
- REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{200},
flowFilesEnqueued));
+ auto flowFilesEnqueued = [&] { return root->getTotalFlowFileCount() >= 3; };
+ REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{500},
flowFilesEnqueued));
- REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sourceProc->trigger_count.load() >= 1);
execSinkPromise.set_value();
controller->stop();
- REQUIRE(sourceProc->trigger_count.load() == 1);
- REQUIRE(sinkProc->trigger_count.load() == 1);
+ REQUIRE(sourceProc->trigger_count.load() >= 1);
+ REQUIRE(sinkProc->trigger_count.load() >= 1);
}
TEST_CASE("Extend the waiting period during shutdown", "[TestFlow4]") {
@@ -230,10 +230,11 @@ TEST_CASE("Extend the waiting period during shutdown",
"[TestFlow4]") {
testController.startFlow();
// wait for the source processor to enqueue its flowFiles
- auto flowFilesEnqueued = [&] {return root->getTotalFlowFileCount() == 3;};
- REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{200},
flowFilesEnqueued));
+ std::size_t flow_file_count = 0;
+ auto flowFilesEnqueued = [&] { return root->getTotalFlowFileCount() >= 3; };
+ REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{500},
flowFilesEnqueued));
- REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sourceProc->trigger_count.load() >= 1);
std::thread shutdownThread([&]{
execSinkPromise.set_value();
@@ -257,6 +258,6 @@ TEST_CASE("Extend the waiting period during shutdown",
"[TestFlow4]") {
shutdownThread.join();
- REQUIRE(sourceProc->trigger_count.load() == 1);
- REQUIRE(sinkProc->trigger_count.load() == 3);
+ REQUIRE(sourceProc->trigger_count.load() >= 1);
+ REQUIRE(sinkProc->trigger_count.load() >= 3);
}
[nifi-minifi-cpp] branch main updated: MINIFICPP-1203 - Make cpplint ignore bison generated expression language code
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 1b44f74 MINIFICPP-1203 - Make cpplint ignore bison generated expression language code 1b44f74 is described below commit 1b44f74c956561210321bd9c9c352d87539ce026 Author: Adam Hunyadi AuthorDate: Fri Feb 19 09:25:30 2021 +0100 MINIFICPP-1203 - Make cpplint ignore bison generated expression language code Signed-off-by: Arpad Boda This closes #1010 --- extensions/expression-language/CPPLINT.cfg | 6 ++ 1 file changed, 6 insertions(+) diff --git a/extensions/expression-language/CPPLINT.cfg b/extensions/expression-language/CPPLINT.cfg new file mode 100644 index 000..344ec8c --- /dev/null +++ b/extensions/expression-language/CPPLINT.cfg @@ -0,0 +1,6 @@ +exclude_files=Scanner.cpp +exclude_files=Parser.cpp +exclude_files=Parser.hpp +exclude_files=location.hh +exclude_files=position.hh +exclude_files=stack.hh
[nifi-minifi-cpp] branch main updated: MINIFICPP-1527 Add update before install to avoid fetch errors
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new c4b29af MINIFICPP-1527 Add update before install to avoid fetch errors c4b29af is described below commit c4b29af8a528125c01b14f29106284c89fbd4ce3 Author: Gabor Gyimesi AuthorDate: Mon Mar 8 18:32:26 2021 +0100 MINIFICPP-1527 Add update before install to avoid fetch errors Signed-off-by: Arpad Boda This closes #1024 --- .github/workflows/ci.yml | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b894a30..77b105c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -108,6 +108,7 @@ jobs: ubuntu-16.04-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -154,6 +155,7 @@ jobs: ubuntu-20.04-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -198,6 +200,7 @@ jobs: debian-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -219,6 +222,7 @@ jobs: centos-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -240,6 +244,7 @@ jobs: fedora-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -261,6 +266,7 @@ jobs: ubuntu-18.04-ccache-refs/heads/main - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -282,6 +288,7 @@ jobs: ubuntu-16.04-shared-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -296,6 +303,8 @@ jobs: - id: build run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_JNI=OFF -DDISABLE_JEMALLOC=ON -DENABLE_AWS=ON -DENABLE_LIBRDKAFKA=ON .. && make docker - id: install_deps -run: sudo apt install -y python3-virtualenv +run: | + sudo apt update + sudo apt install -y python3-virtualenv - id: test run: cd build && make docker-verify
[nifi-minifi-cpp] branch main updated: MINIFICPP-1509 - Use _stat64 on windows
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new f9279bc MINIFICPP-1509 - Use _stat64 on windows
f9279bc is described below
commit f9279bc47caab16de282e7f22e0045dd0df85e39
Author: Adam Debreceni
AuthorDate: Wed Feb 24 14:26:54 2021 +0100
MINIFICPP-1509 - Use _stat64 on windows
Signed-off-by: Arpad Boda
This closes #1015
---
.../standard-processors/processors/GetFile.cpp | 59 --
.../standard-processors/processors/PutFile.cpp | 10 +---
libminifi/include/utils/file/FileUtils.h | 38 +++---
3 files changed, 56 insertions(+), 51 deletions(-)
diff --git a/extensions/standard-processors/processors/GetFile.cpp
b/extensions/standard-processors/processors/GetFile.cpp
index 5915866..98f2dd5 100644
--- a/extensions/standard-processors/processors/GetFile.cpp
+++ b/extensions/standard-processors/processors/GetFile.cpp
@@ -228,42 +228,49 @@ void GetFile::pollListing(std::queue ,
const GetFileRequest
bool GetFile::acceptFile(std::string fullName, std::string name, const
GetFileRequest ) {
logger_->log_trace("Checking file: %s", fullName);
+#ifdef WIN32
+ struct _stat64 statbuf;
+ if (_stat64(fullName.c_str(), ) != 0) {
+return false;
+ }
+#else
struct stat statbuf;
+ if (stat(fullName.c_str(), ) != 0) {
+return false;
+ }
+#endif
+ uint64_t file_size = gsl::narrow(statbuf.st_size);
+ uint64_t modifiedTime = gsl::narrow(statbuf.st_mtime) * 1000;
- if (stat(fullName.c_str(), ) == 0) {
-if (request.minSize > 0 && statbuf.st_size < (int32_t) request.minSize)
- return false;
-
-if (request.maxSize > 0 && statbuf.st_size > (int32_t) request.maxSize)
- return false;
+ if (request.minSize > 0 && file_size < request.minSize)
+return false;
-uint64_t modifiedTime = ((uint64_t) (statbuf.st_mtime) * 1000);
-uint64_t fileAge = utils::timeutils::getTimeMillis() - modifiedTime;
-if (request.minAge > 0 && fileAge < request.minAge)
- return false;
-if (request.maxAge > 0 && fileAge > request.maxAge)
- return false;
+ if (request.maxSize > 0 && file_size > request.maxSize)
+return false;
-if (request.ignoreHiddenFile &&
utils::file::FileUtils::is_hidden(fullName))
- return false;
+ uint64_t fileAge = utils::timeutils::getTimeMillis() - modifiedTime;
+ if (request.minAge > 0 && fileAge < request.minAge)
+return false;
+ if (request.maxAge > 0 && fileAge > request.maxAge)
+return false;
-if (utils::file::FileUtils::access(fullName.c_str(), R_OK) != 0)
- return false;
+ if (request.ignoreHiddenFile && utils::file::FileUtils::is_hidden(fullName))
+return false;
-if (request.keepSourceFile == false &&
utils::file::FileUtils::access(fullName.c_str(), W_OK) != 0)
- return false;
+ if (utils::file::FileUtils::access(fullName.c_str(), R_OK) != 0)
+return false;
-utils::Regex rgx(request.fileFilter);
-if (!rgx.match(name)) {
- return false;
-}
+ if (request.keepSourceFile == false &&
utils::file::FileUtils::access(fullName.c_str(), W_OK) != 0)
+return false;
-metrics_->input_bytes_ += statbuf.st_size;
-metrics_->accepted_files_++;
-return true;
+ utils::Regex rgx(request.fileFilter);
+ if (!rgx.match(name)) {
+return false;
}
- return false;
+ metrics_->input_bytes_ += file_size;
+ metrics_->accepted_files_++;
+ return true;
}
void GetFile::performListing(const GetFileRequest ) {
diff --git a/extensions/standard-processors/processors/PutFile.cpp
b/extensions/standard-processors/processors/PutFile.cpp
index 5503190..3765856 100644
--- a/extensions/standard-processors/processors/PutFile.cpp
+++ b/extensions/standard-processors/processors/PutFile.cpp
@@ -19,7 +19,6 @@
*/
#include "PutFile.h"
-#include
#include
#include
#include
@@ -149,9 +148,6 @@ void PutFile::onTrigger(core::ProcessContext *context,
core::ProcessSession *ses
logger_->log_debug("PutFile writing file %s into directory %s", filename,
directory);
- // If file exists, apply conflict resolution strategy
- struct stat statResult;
-
if ((max_dest_files_ != -1) &&
utils::file::FileUtils::is_directory(directory.c_str())) {
int64_t count = 0;
@@ -171,7 +167,7 @@ void PutFile::onTrigger(core::ProcessContext *context,
core::ProcessSession *ses
}
}
- if (stat(destFile.c_str(), ) == 0) {
+ if (utils::file::exists(destFile)) {
logger_->log_warn("Destination file %s exists; applying Conflict
Resolution Strategy: %s", destFile, conflict_resolution_);
if (c
[nifi-minifi-cpp] branch main updated (c8c4482 -> 95baec6)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from c8c4482 MINIFICPP-1487 Do not trigger the processor if the incoming queue has penalized flow files only add 95baec6 MINIFICPP-1523 - cxxopts: Add limits header No new revisions were added by this update. Summary of changes: libminifi/src/utils/StringUtils.cpp| 2 ++ thirdparty/cxxopts/include/cxxopts.hpp | 1 + 2 files changed, 3 insertions(+)
[nifi-minifi-cpp] branch main updated: MINIFICPP-1487 Do not trigger the processor if the incoming queue has penalized flow files only
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new c8c4482 MINIFICPP-1487 Do not trigger the processor if the incoming
queue has penalized flow files only
c8c4482 is described below
commit c8c448240690e2398534c99b101e7d536a367cb7
Author: Ferenc Gerlits
AuthorDate: Thu Feb 18 16:28:11 2021 +0100
MINIFICPP-1487 Do not trigger the processor if the incoming queue has
penalized flow files only
* make mutex_ mutable and mark const methods const
* remove C-style (void) from function declarations
* remove pointless comments
* add a missing virtual-override pair on isYield()
* add a missing 'explicit'
* remove the declaration of unimplemented method getNextIncomingConnection()
* Test that the processor detects correctly whether it has any work to do
* Document the current behavior of Connection::poll(), to be fixed later
* Use a custom queue which is aware of penalty expirations
Signed-off-by: Arpad Boda
This closes #1013
---
.../tests/unit/ProcessorTests.cpp | 63
libminifi/include/Connection.h | 6 +-
libminifi/include/core/Processor.h | 51 +++
libminifi/include/utils/FlowFileQueue.h| 58
libminifi/src/Connection.cpp | 22 +--
libminifi/src/SchedulingAgent.cpp | 2 +-
libminifi/src/core/Processor.cpp | 19 +--
libminifi/src/utils/FlowFileQueue.cpp | 103 +
libminifi/test/unit/ConnectionTests.cpp| 88 +++
libminifi/test/unit/FlowFileQueueTests.cpp | 161 +
libminifi/test/unit/MockClasses.h | 2 +-
11 files changed, 500 insertions(+), 75 deletions(-)
diff --git a/extensions/standard-processors/tests/unit/ProcessorTests.cpp
b/extensions/standard-processors/tests/unit/ProcessorTests.cpp
index 5d34024..1ebcd39 100644
--- a/extensions/standard-processors/tests/unit/ProcessorTests.cpp
+++ b/extensions/standard-processors/tests/unit/ProcessorTests.cpp
@@ -614,3 +614,66 @@ TEST_CASE("TestRPGWithoutHostInvalidPort", "[TestRPG5]") {
TEST_CASE("TestRPGValid", "[TestRPG6]") {
testRPGBypass("", "8080", "8080", false);
}
+
+TEST_CASE("A Processor detects correctly if it has incoming flow files it can
process", "[isWorkAvailable]") {
+ LogTestController::getInstance().setDebug();
+
+ const auto repo = std::make_shared();
+ const auto content_repo =
std::make_shared();
+ content_repo->initialize(std::make_shared());
+
+ const std::shared_ptr processor =
std::make_shared("test_processor");
+ const auto incoming_connection = std::make_shared(repo,
content_repo, "incoming_connection");
+ incoming_connection->addRelationship(core::Relationship{"success", ""});
+ incoming_connection->setDestinationUUID(processor->getUUID());
+ processor->addConnection(incoming_connection);
+ processor->initialize();
+
+ const auto processor_node = std::make_shared(processor);
+ const auto context = std::make_shared(processor_node,
nullptr, repo, repo, content_repo);
+ const auto session_factory =
std::make_shared(context);
+ const auto session = session_factory->createSession();
+
+ SECTION("Initially, the queue is empty, so there is no work available") {
+REQUIRE_FALSE(processor->isWorkAvailable());
+ }
+
+ SECTION("When a non-penalized flow file is queued, there is work available")
{
+const auto flow_file = session->create();
+incoming_connection->put(flow_file);
+
+REQUIRE(processor->isWorkAvailable());
+ }
+
+ SECTION("When a penalized flow file is queued, there is no work available
(until the penalty expires)") {
+const auto flow_file = session->create();
+session->penalize(flow_file);
+incoming_connection->put(flow_file);
+
+REQUIRE_FALSE(processor->isWorkAvailable());
+ }
+
+ SECTION("If there is both a penalized and a non-penalized flow file queued,
there is work available") {
+const auto normal_flow_file = session->create();
+incoming_connection->put(normal_flow_file);
+
+const auto penalized_flow_file = session->create();
+session->penalize(penalized_flow_file);
+incoming_connection->put(penalized_flow_file);
+
+REQUIRE(processor->isWorkAvailable());
+ }
+
+ SECTION("When a penalized flow file is queued, there is work available after
the penalty expires") {
+processor->setPenalizationPeriodMsec(10);
+
+const auto flow_file = session->create();
+session->penali
[nifi-minifi-cpp] branch main updated: MINIFICPP-1512 update README
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 3c57dc0 MINIFICPP-1512 update README 3c57dc0 is described below commit 3c57dc065d59720a019bc5828acbfbc0b9c4d71a Author: Marton Szasz AuthorDate: Mon Mar 8 09:12:28 2021 +0100 MINIFICPP-1512 update README Signed-off-by: Arpad Boda This closes #1023 --- README.md | 93 --- 1 file changed, 54 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index cc8f421..4a4d4d7 100644 --- a/README.md +++ b/README.md @@ -27,10 +27,12 @@ MiNiFi is a child project effort of Apache NiFi. This repository is for a nativ - [System Requirements](#system-requirements) - [Bootstrapping](#bootstrapping) - [Building For Other Distros](#building-for-other-distros) - - [Cleaning](#cleaning) + - [Snapcraft](#snapcraft) + - [Installation](#installation) - [Configuring](#configuring) - [Running](#running) - [Deploying](#deploying) + - [Cleaning](#cleaning) - [Extensions](#extensions) - [Security](#security) - [Operations](#operations) @@ -73,7 +75,7 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension | Extension Set| Processors | CMAKE Flag | | - |:-| :-| | Archive Extensions| [ApplyTemplate](PROCESSORS.md#applytemplate)[CompressContent](PROCESSORS.md#compresscontent)[ManipulateArchive](PROCESSORS.md#manipulatearchive)[MergeContent](PROCESSORS.md#mergecontent)[FocusArchiveEntry](PROCESSORS.md#focusarchiveentry)[UnfocusArchiveEntry](PROCESSORS.md#unfocusarchiveentry) | -DBUILD_LIBARCHIVE=ON | -| AWS | [AWSCredentialsService](CONTROLLERS.md#awsCredentialsService)[PutS3Object](PROCESSORS.md#puts3object)[DeleteS3Object](PROCESSORS.md#deletes3object) | -DENABLE_AWS=ON | +| AWS | [AWSCredentialsService](CONTROLLERS.md#awscredentialsservice)[PutS3Object](PROCESSORS.md#puts3object)[DeleteS3Object](PROCESSORS.md#deletes3object) | -DENABLE_AWS=ON | | CivetWeb | [ListenHTTP](PROCESSORS.md#listenhttp) | -DDISABLE_CIVET=ON | | CURL | [InvokeHTTP](PROCESSORS.md#invokehttp) |-DDISABLE_CURL=ON | | GPS | GetGPS |-DENABLE_GPS=ON | @@ -88,7 +90,7 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension | SFTP | [FetchSFTP](PROCESSORS.md#fetchsftp)[ListSFTP](PROCESSORS.md#listsftp)[PutSFTP](PROCESSORS.md#putsftp) | -DENABLE_SFTP=ON | | SQL | ExecuteSQLPutSQLQueryDatabaseTable | -DENABLE_SQL=ON | | SQLite | [ExecuteSQL](PROCESSORS.md#executesql)[PutSQL](PROCESSORS.md#putsql) |-DENABLE_SQLITE=ON | -| Tensorflow | [TFApplyGraph](PROCESSORS.md#tfapplygraph)[TFConvertImageToTensor](PROCESSORS.md#tfconvertimagetotensor)[TFExtractTopLabels](PROCESSORS.md#tfextracttoplabels) |-DENABLE_TENSORFLOW=ON | +| Tensorflow | TFApplyGraphTFConvertImageToTensorTFExtractTopLabels | -DENABLE_TENSORFLOW=ON | | USB Camera | [GetUSBCamera](PROCESSORS.md#getusbcamera) | -DENABLE_USB_CAMERA=ON | | Windows Event Log (Windows only) | CollectorInitiatedSubscriptionConsumeWindowsEventLogTailEventLog | -DENABLE_WEL=ON | @@ -99,22 +101,18 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension * Build and usage currently only supports Windows, Linux and OS X environments. MiNiFi C++ can be built and run through the Windows Subsystem for Linux but we provide no support for this platform. * Provenance events generation is supported and are persisted using RocksDB. Volatile repositories can be used on systems without persistent storage. * If MiNiFi C++ is built with the OPC-UA extension enabled, it bundles [open62541](https://open62541.org/), which is available under the Mozilla Public License Version 2.0, a Category B license under [ASF 3rd party license policy](https://www.apache.org/legal/resolved.html#category-b). +* If MiNiFi C++ packaged on Windows, the resulting MSI may not be publicly redistributed under the Apache license, because it contains Microsoft redistributable DLLs, which fall under Category X of the [ASF 3rd party license policy](https://www.apache.org/legal/resolved.html#category-x). ## System Requirements ### To build Utilities -* CMake - * 3.10 or greater -* gcc - * 4.8.4 or greater -* g++ - * 4.8.4 or greater -* bison - * 3.0.x (3.2 has been shown to fail builds) -* flex - * 2.5 or greater +* CMake 3.11 or greater +* gcc 4.8.4 or greater +* g++ 4.8.4 or greater +* bison 3.0.x (3.2 has been shown to fail builds) +* flex 2.5 or greater # External Projects @@ -126,12 +124,11 @@ versions of LibreSSL, cURL, or zlib are used: * automake * libtool -**NOTE** if Lua support is enabled, then a C++ compiler with support
[nifi-minifi-cpp] 01/02: MINIFICPP-1513 - Increase timeouts for behave tests involving http proxies, clean up logging
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 63860782fdbb27def1d2d7fda8bd85ac91bbb74b
Author: Adam Hunyadi
AuthorDate: Fri Feb 26 15:34:49 2021 +0100
MINIFICPP-1513 - Increase timeouts for behave tests involving http proxies,
clean up logging
Signed-off-by: Arpad Boda
This closes #1018
---
docker/DockerVerify.sh | 2 +-
.../integration/MiNiFi_integration_test_driver.py | 1 -
docker/test/integration/environment.py | 1 -
docker/test/integration/features/http.feature | 2 +-
docker/test/integration/features/s3.feature| 6 +++---
.../integration/minifi/core/DockerTestCluster.py | 23 +++---
.../minifi/core/DockerTestDirectoryBindings.py | 1 -
7 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/docker/DockerVerify.sh b/docker/DockerVerify.sh
index dc897c2..afb4077 100755
--- a/docker/DockerVerify.sh
+++ b/docker/DockerVerify.sh
@@ -72,7 +72,7 @@ export PATH
PYTHONPATH="${PYTHONPATH}:${docker_dir}/test/integration"
export PYTHONPATH
-BEHAVE_OPTS="-f pretty --logging-level INFO --no-capture"
+BEHAVE_OPTS="-f pretty --logging-level INFO --logging-clear-handlers"
cd "${docker_dir}/test/integration"
exec
diff --git a/docker/test/integration/MiNiFi_integration_test_driver.py
b/docker/test/integration/MiNiFi_integration_test_driver.py
index a1c63a0..ca5f04a 100644
--- a/docker/test/integration/MiNiFi_integration_test_driver.py
+++ b/docker/test/integration/MiNiFi_integration_test_driver.py
@@ -22,7 +22,6 @@ from minifi.validators.SingleFileOutputValidator import
SingleFileOutputValidato
class MiNiFi_integration_test():
def __init__(self, context):
-logging.info("MiNiFi_integration_test init")
self.test_id = str(uuid.uuid4())
self.clusters = {}
diff --git a/docker/test/integration/environment.py
b/docker/test/integration/environment.py
index 68ddd42..4a3c12f 100644
--- a/docker/test/integration/environment.py
+++ b/docker/test/integration/environment.py
@@ -11,7 +11,6 @@ def raise_exception(exception):
@fixture
def test_driver_fixture(context):
-logging.info("Integration test setup")
context.test = MiNiFi_integration_test(context)
yield context.test
logging.info("Integration test teardown...")
diff --git a/docker/test/integration/features/http.feature
b/docker/test/integration/features/http.feature
index 0652fd9..1f5e89d 100644
--- a/docker/test/integration/features/http.feature
+++ b/docker/test/integration/features/http.feature
@@ -40,7 +40,7 @@ Feature: Sending data using InvokeHTTP to a receiver using
ListenHTTP
And the "success" relationship of the ListenHTTP processor is connected to
the PutFile
When all instances start up
-Then a flowfile with the content "test" is placed in the monitored
directory in less than 60 seconds
+Then a flowfile with the content "test" is placed in the monitored
directory in less than 120 seconds
And no errors were generated on the "http-proxy" regarding
"http://minifi-listen:8080/contentListener;
Scenario: A MiNiFi instance and transfers hashed data to another MiNiFi
instance
diff --git a/docker/test/integration/features/s3.feature
b/docker/test/integration/features/s3.feature
index 897bf18..03f0e11 100644
--- a/docker/test/integration/features/s3.feature
+++ b/docker/test/integration/features/s3.feature
@@ -40,7 +40,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
And the http proxy server "http-proxy" is set up
When all instances start up
-Then a flowfile with the content "test" is placed in the monitored
directory in less than 90 seconds
+Then a flowfile with the content "test" is placed in the monitored
directory in less than 150 seconds
And the object on the "s3" s3 server is "LH_O#L|FDhttp://s3-server:9090/test_bucket/test_object_key;
@@ -102,7 +102,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
When all instances start up
-Then a flowfile with the content "test" is placed in the monitored
directory in less than 120 seconds
+Then a flowfile with the content "test" is placed in the monitored
directory in less than 150 seconds
And the object bucket on the "s3" s3 server is empty
And no errors were generated on the "http-proxy" regarding
"http://s3-server:9090/test_bucket/test_object_key;
@@ -151,5 +151,5 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
When all instances start up
-Then a flowfile with the content "test" is placed in the monitored
directory in less than 120 seconds
+Then a fl
[nifi-minifi-cpp] 02/02: MINIFICPP-1517 - Fix integration test log cluttering
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 5b6d78d65dd1a97ee4e8e9049fd49b2ec3640e24
Author: Adam Hunyadi
AuthorDate: Mon Mar 1 11:04:00 2021 +0100
MINIFICPP-1517 - Fix integration test log cluttering
Signed-off-by: Arpad Boda
This closes #1019
---
CMakeLists.txt | 13 -
cmake/DockerConfig.cmake | 1 +
docker/DockerBuild.sh| 2 ++
docker/Dockerfile| 5 +++--
4 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index e43b850..68e1fc8 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -819,7 +819,18 @@ registerTest("encrypt-config/tests")
include(BuildDocs)
-include(DockerConfig)
+if (ENABLE_PYTHON OR NOT DISABLE_SCRIPTING)
+ message(STATUS "Python and scripting extensions will disabled for the
docker build as they produce many error messages on MiNiFi startup.")
+ set(ENABLE_PYTHON_CACHE ${ENABLE_PYTHON})
+ set(DISABLE_SCRIPTING_CACHE ${DISABLE_SCRIPTING})
+ set(ENABLE_PYTHON OFF)
+ set(DISABLE_SCRIPTING ON) # Implies DISABLE_PYTHON_SCRIPTING as well
+ include(DockerConfig)
+ set(ENABLE_PYTHON ${ENABLE_PYTHON_CACHE})
+ set(DISABLE_SCRIPTING ${DISABLE_SCRIPTING_CACHE})
+else()
+ include(DockerConfig)
+endif()
if(NOT WIN32)
# Create a custom build target that will run the linter.
diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake
index c935f27..5ae9b3b 100644
--- a/cmake/DockerConfig.cmake
+++ b/cmake/DockerConfig.cmake
@@ -53,6 +53,7 @@ add_custom_target(
-c DISABLE_LZMA=${DISABLE_LZMA}
-c DISABLE_BZIP2=${DISABLE_BZIP2}
-c DISABLE_SCRIPTING=${DISABLE_SCRIPTING}
+-c DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING}
-c DISABLE_CONTROLLER=${DISABLE_CONTROLLER}
-c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE}
-c BUILD_NUMBER=${BUILD_NUMBER}
diff --git a/docker/DockerBuild.sh b/docker/DockerBuild.sh
index 51814cd..40e4701 100755
--- a/docker/DockerBuild.sh
+++ b/docker/DockerBuild.sh
@@ -58,6 +58,7 @@ DISABLE_LIBARCHIVE=${DISABLE_LIBARCHIVE:-}
DISABLE_LZMA=${DISABLE_LZMA:-}
DISABLE_BZIP2=${DISABLE_BZIP2:-}
DISABLE_SCRIPTING=${DISABLE_SCRIPTING:-}
+DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING:-}
DISABLE_CONTROLLER=${DISABLE_CONTROLLER:-}
DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE:-}
@@ -194,6 +195,7 @@ BUILD_ARGS="--build-arg UID=${UID_ARG} \
--build-arg DISABLE_LZMA=${DISABLE_LZMA} \
--build-arg DISABLE_BZIP2=${DISABLE_BZIP2} \
--build-arg DISABLE_SCRIPTING=${DISABLE_SCRIPTING} \
+--build-arg DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING} \
--build-arg DISABLE_CONTROLLER=${DISABLE_CONTROLLER} "
if [ -n "${DOCKER_BASE_IMAGE}" ]; then
diff --git a/docker/Dockerfile b/docker/Dockerfile
index ea6049f..20e8ce8 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -78,7 +78,7 @@ FROM build_deps AS build_minimal
RUN cd ${MINIFI_BASE_DIR} \
&& mkdir build \
&& cd build \
- && cmake -DDISABLE_SCRIPTING=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_AWS=ON
-DSKIP_TESTS=true -DCMAKE_BUILD_TYPE=MinSizeRel .. \
+ && cmake -DENABLE_PYTHON=OFF -DDISABLE_SCRIPTING=ON -DENABLE_LIBRDKAFKA=ON
-DENABLE_AWS=ON -DSKIP_TESTS=true -DCMAKE_BUILD_TYPE=MinSizeRel .. \
&& make -j$(nproc) package \
&& tar -xzvf
${MINIFI_BASE_DIR}/build/nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz -C
${MINIFI_BASE_DIR}
@@ -115,6 +115,7 @@ ARG DISABLE_LIBARCHIVE
ARG DISABLE_LZMA
ARG DISABLE_BZIP2
ARG DISABLE_SCRIPTING
+ARG DISABLE_PYTHON_SCRIPTING
ARG DISABLE_CONTROLLER
RUN cd ${MINIFI_BASE_DIR} \
&& mkdir build \
@@ -128,7 +129,7 @@ RUN cd ${MINIFI_BASE_DIR} \
-DDISABLE_CURL=${DISABLE_CURL} -DDISABLE_JEMALLOC=${DISABLE_JEMALLOC}
-DDISABLE_CIVET=${DISABLE_CIVET} \
-DDISABLE_EXPRESSION_LANGUAGE=${DISABLE_EXPRESSION_LANGUAGE}
-DDISABLE_ROCKSDB=${DISABLE_ROCKSDB} \
-DDISABLE_LIBARCHIVE=${DISABLE_LIBARCHIVE} -DDISABLE_LZMA=${DISABLE_LZMA}
-DDISABLE_BZIP2=${DISABLE_BZIP2} \
--DDISABLE_SCRIPTING=${DISABLE_SCRIPTING}
-DDISABLE_CONTROLLER=${DISABLE_CONTROLLER} -DCMAKE_BUILD_TYPE=Release .. \
+-DDISABLE_SCRIPTING=${DISABLE_SCRIPTING}
-DDISABLE_PYTHON_SCRIPTING=${DDISABLE_PYTHON_SCRIPTING}
-DDISABLE_CONTROLLER=${DISABLE_CONTROLLER} -DCMAKE_BUILD_TYPE=Release .. \
&& make -j$(nproc) package \
&& tar -xzvf
${MINIFI_BASE_DIR}/build/nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz -C
${MINIFI_BASE_DIR}
[nifi-minifi-cpp] branch main updated (287f184 -> 5b6d78d)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 287f184 MINIFICPP-1496 Remove ASAN from the list of extensions in bootstrap.sh new 6386078 MINIFICPP-1513 - Increase timeouts for behave tests involving http proxies, clean up logging new 5b6d78d MINIFICPP-1517 - Fix integration test log cluttering The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CMakeLists.txt | 13 +++- cmake/DockerConfig.cmake | 1 + docker/DockerBuild.sh | 2 ++ docker/DockerVerify.sh | 2 +- docker/Dockerfile | 5 +++-- .../integration/MiNiFi_integration_test_driver.py | 1 - docker/test/integration/environment.py | 1 - docker/test/integration/features/http.feature | 2 +- docker/test/integration/features/s3.feature| 6 +++--- .../integration/minifi/core/DockerTestCluster.py | 23 +++--- .../minifi/core/DockerTestDirectoryBindings.py | 1 - 11 files changed, 39 insertions(+), 18 deletions(-)
svn commit: r46347 - /dev/nifi/nifi-minifi-cpp/0.9.0/ /release/nifi/nifi-minifi-cpp/0.9.0/
Author: aboda Date: Fri Feb 26 11:30:25 2021 New Revision: 46347 Log: MINIFICPP-1463 Promote NiFi MiNiFi C++ vote artifacts to release Added: release/nifi/nifi-minifi-cpp/0.9.0/ - copied from r46346, dev/nifi/nifi-minifi-cpp/0.9.0/ Removed: dev/nifi/nifi-minifi-cpp/0.9.0/
[nifi-minifi-cpp] branch main updated: MINIFICPP-1511 - Update documentation with Visual Studio 2019
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new a2e8068 MINIFICPP-1511 - Update documentation with Visual Studio 2019 a2e8068 is described below commit a2e80689889b41809c6110e47a87ee9c987622f1 Author: Adam Debreceni AuthorDate: Thu Feb 25 15:46:22 2021 +0100 MINIFICPP-1511 - Update documentation with Visual Studio 2019 Signed-off-by: Arpad Boda This closes #1016 --- README.md | 4 ++-- Windows.md | 8 +--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 28d39d1..cc8f421 100644 --- a/README.md +++ b/README.md @@ -192,8 +192,8 @@ On all distributions please use -DUSE_SHARED_LIBS=OFF to statically link zlib, l below. Windows - Build and Installation has been tested with Windows 10 using Visual Studio 2017. You can build - and create an MSI via the CPACK command. This requires the installation of the WiX + Build and Installation has been tested with Windows 10 using Visual Studio 2017 and Visual Studio 2019. + You can build and create an MSI via the CPACK command. This requires the installation of the WiX toolset (http://wixtoolset.org/). To do this, open up a prompt into your build directory and type 'cpack' . The CPACK command will automatically generate and provide you a path to the distributable msi file. See [Windows.md](Windows.md) for more details. diff --git a/Windows.md b/Windows.md index ab58abb..7e5223b 100644 --- a/Windows.md +++ b/Windows.md @@ -18,13 +18,13 @@ ## Requirements Apache NiFi MiNiFi C++ has been built on Window Server 2016, 2019, and Windows 10 operating systems. The project is CMake focused we suggest building via -Visual Studio 2017 or our `win_build_vs.bat` script. +Visual Studio 2017, Visual Studio 2019 or our `win_build_vs.bat` script. The project previously required OpenSSL to be installed. If you follow our build procedures, below, you will not need to install that dependency. ### Required software - - Visual Studio 2017 + - Visual Studio 2017 or Visual Studio 2019 - [CMake](https://cmake.org/download/) - [Git](https://git-scm.com/download/win) - (Optional) [WiX Toolset](https://wixtoolset.org/releases/) (only for building the MSI) @@ -68,12 +68,14 @@ After the build directory it will take optional parameters modifying the CMake c | /D | Builds RelWithDebInfo build instead of Release | | /DD | Builds Debug build instead of Release | | /CI | Sets STRICT_GSL_CHECKS to AUDIT | +| /2019 | Use the Visual Studio 2019 environment instead of the default Visual Studio 2017 | Examples: - 32-bit build with kafka, disabling tests, enabling MSI creation: `win_build_vs.bat build32 /T /K /P` - 64-bit build with JNI, with debug symbols: `win_build_vs.bat build64 /64 /J /D` -`win_build_vs.bat` requires a Visual Studio 2017 build environment to be set up. For 32-bit builds this can be achieved by using the `x86 Native Tools Command Prompt for VS 2017`, for 64-bit builds by using the `x64 Native Tools Command Prompt for VS 2017`. +`win_build_vs.bat` requires either a Visual Studio 2017 or a Visual Studio 2019 (if called with the `/2019` flag) build environment to be set up. With Visual Studio 2017 use the `x86 Native Tools Command Prompt for VS 2017` for 32-bit, or the `x64 Native Tools Command Prompt for VS 2017` for 64-bit builds. +For Visual Studio 2019 use the `x86 Native Tools Command Prompt for VS 2019`, or the `x64 Native Tools Command Prompt for VS 2019` for 32-bit and 64-bit builds respectively. ## Building directly with CMake
[nifi-minifi-cpp] branch main updated: MINIFICPP-1373 - Implement ConsumeKafka
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 6cfbf79 MINIFICPP-1373 - Implement ConsumeKafka
6cfbf79 is described below
commit 6cfbf797a6c060a4781e0185a46c7cd9c4229384
Author: Adam Hunyadi
AuthorDate: Fri Feb 26 10:56:02 2021 +0100
MINIFICPP-1373 - Implement ConsumeKafka
Signed-off-by: Arpad Boda
This closes #940
---
CMakeLists.txt | 2 +-
PROCESSORS.md | 33 +-
extensions/librdkafka/ConsumeKafka.cpp | 570
extensions/librdkafka/ConsumeKafka.h | 192 +++
.../{tests => docker_tests}/CMakeLists.txt | 2 +-
extensions/librdkafka/rdkafka_utils.cpp| 121 +
extensions/librdkafka/rdkafka_utils.h | 105
extensions/librdkafka/tests/CMakeLists.txt | 6 +-
extensions/librdkafka/tests/ConsumeKafkaTests.cpp | 590 +
libminifi/include/Connection.h | 4 +-
libminifi/include/core/FlowFile.h | 5 +-
libminifi/include/utils/GeneralUtils.h | 12 +
libminifi/include/utils/ProcessorConfigUtils.h | 81 +++
libminifi/include/utils/StringUtils.h | 8 +-
libminifi/src/Connection.cpp | 2 +-
libminifi/src/core/FlowFile.cpp| 17 +-
libminifi/src/utils/StringUtils.cpp| 17 +-
libminifi/test/TestBase.cpp| 169 --
libminifi/test/TestBase.h | 51 +-
libminifi/test/unit/StringUtilsTests.cpp | 10 +
20 files changed, 1908 insertions(+), 89 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index ff0024f..e43b850 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -444,7 +444,7 @@ option(ENABLE_LIBRDKAFKA "Enables the librdkafka
extension." OFF)
if (ENABLE_ALL OR ENABLE_LIBRDKAFKA)
include(BundledLibRdKafka)
use_bundled_librdkafka(${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR})
- createExtension(RDKAFKA-EXTENSIONS "RDKAFKA EXTENSIONS" "This Enables
librdkafka functionality including PublishKafka" "extensions/librdkafka"
"extensions/librdkafka/tests")
+ createExtension(RDKAFKA-EXTENSIONS "RDKAFKA EXTENSIONS" "This Enables
librdkafka functionality including PublishKafka" "extensions/librdkafka"
"extensions/librdkafka/tests" "extensions/librdkafka/docker-tests")
endif()
## Scripting extensions
diff --git a/PROCESSORS.md b/PROCESSORS.md
index c2cd82a..0ca385d 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -9,6 +9,7 @@
- [CapturePacket](#capturepacket)
- [CaptureRTSPFrame](#capturertspframe)
- [CompressContent](#compresscontent)
+- [ConsumeKafka](#consumekafka)
- [ConsumeMQTT](#consumemqtt)
- [DeleteS3Object](#deletes3object)
- [ExecuteProcess](#executeprocess)
@@ -180,6 +181,37 @@ In the list below, the names of required properties appear
in bold. Any other pr
|failure|FlowFiles will be transferred to the failure relationship if they
fail to compress/decompress|
|success|FlowFiles will be transferred to the success relationship after
successfully being compressed or decompressed|
+## ConsumeKafka
+
+### Description
+
+Consumes messages from Apache Kafka and transform them into MiNiFi FlowFiles.
The application should make sure that the processor is triggered at regular
intervals, even if no messages are expected, to serve any queued callbacks
waiting to be called. Rebalancing can also only happen on trigger.
+### Properties
+
+In the list below, the names of required properties appear in bold. Any other
properties (not in bold) are considered optional. The table also indicates any
default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+| - | - | - | - |
+|Duplicate Header Handling|Keep Latest|Comma-separated MergeKeep
FirstKeep Latest|For headers to be added as attributes, this option
specifies how to handle cases where multiple headers are present with the same
key. For example in case of receiving these two headers: "Accept: text/html"
and "Accept: application/xml" and we want to attach the value of "Accept" as a
FlowFile attribute: - "Keep First" attaches: "Accept -> text/html" -
"Keep Latest" attaches: "Accept - [...]
+|**Group ID**|||A Group ID is used to identify consumers that are within the
same consumer group. Corresponds to Kafka's 'group.id' property.**Supports
Expression Language: true**|
+|Headers To Add As Attributes|||A comma separated list to match against all
mes
[nifi-minifi-cpp] branch main updated: MINIFICPP-1506 - Apply patch #3236 from librdkafka repository Upgrade librdkafka to 1.6.0
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new a975d23 MINIFICPP-1506 - Apply patch #3236 from librdkafka repository
Upgrade librdkafka to 1.6.0
a975d23 is described below
commit a975d232fa1e755b960383b7447581a36b34c3fa
Author: Adam Debreceni
AuthorDate: Mon Feb 22 14:38:43 2021 +0100
MINIFICPP-1506 - Apply patch #3236 from librdkafka repository
Upgrade librdkafka to 1.6.0
Signed-off-by: Arpad Boda
This closes #1014
---
cmake/BundledLibRdKafka.cmake| 7 +-
thirdparty/librdkafka/high-cpu.patch | 185 +++
2 files changed, 190 insertions(+), 2 deletions(-)
diff --git a/cmake/BundledLibRdKafka.cmake b/cmake/BundledLibRdKafka.cmake
index fe49890..bbaaf96 100644
--- a/cmake/BundledLibRdKafka.cmake
+++ b/cmake/BundledLibRdKafka.cmake
@@ -16,6 +16,8 @@
# under the License.
function(use_bundled_librdkafka SOURCE_DIR BINARY_DIR)
+set(PC "${Patch_EXECUTABLE}" -p1 -i
"${SOURCE_DIR}/thirdparty/librdkafka/high-cpu.patch")
+
# Define byproducts
if(WIN32)
set(BYPRODUCT "lib/rdkafka.lib")
@@ -41,10 +43,11 @@ function(use_bundled_librdkafka SOURCE_DIR BINARY_DIR)
# Build project
ExternalProject_Add(
kafka-external
-URL "https://github.com/edenhill/librdkafka/archive/v1.5.0.tar.gz;
-URL_HASH
"SHA256=f7fee59fdbf1286ec23ef0b35b2dfb41031c8727c90ced6435b8cf576f23a656"
+URL "https://github.com/edenhill/librdkafka/archive/v1.6.0.tar.gz;
+URL_HASH
"SHA256=3130cbd391ef683dc9acf9f83fe82ff93b8730a1a34d0518e93c250929be9f6b"
LIST_SEPARATOR % # This is needed for passing semicolon-separated
lists
CMAKE_ARGS ${LIBRDKAFKA_CMAKE_ARGS}
+PATCH_COMMAND ${PC}
BUILD_BYPRODUCTS
"${BINARY_DIR}/thirdparty/librdkafka-install/${BYPRODUCT}"
EXCLUDE_FROM_ALL TRUE
)
diff --git a/thirdparty/librdkafka/high-cpu.patch
b/thirdparty/librdkafka/high-cpu.patch
new file mode 100644
index 000..78099ac
--- /dev/null
+++ b/thirdparty/librdkafka/high-cpu.patch
@@ -0,0 +1,185 @@
+diff -rupN orig/src/rdkafka_broker.h patched/src/rdkafka_broker.h
+--- orig/src/rdkafka_broker.h 2021-01-25 23:25:19.0 +0100
patched/src/rdkafka_broker.h 2021-02-22 16:44:21.665943600 +0100
+@@ -317,8 +317,17 @@ struct rd_kafka_broker_s { /* rd_kafka_b
+ rd_kafka_resp_err_t err; /**< Last error code */
+ int cnt;/**< Number of identical errors */
+ } rkb_last_err;
++
++/** Recovery actions that need to be performed*/
++uint32_t rkb_recovery_actions;
+ };
+
++/*
++ * Recovery actions bit flag
++ */
++#define RKB_RECOVERY_ACTIONS_NONE 0x
++#define RKB_RECOVERY_ACTIONS_REINITIALIZE_WAKEUP_FD 0x0001
++
+ #define rd_kafka_broker_keep(rkb) rd_refcnt_add(&(rkb)->rkb_refcnt)
+ #define rd_kafka_broker_keep_fl(FUNC,LINE,RKB) \
+ rd_refcnt_add_fl(FUNC, LINE, &(RKB)->rkb_refcnt)
+@@ -450,6 +459,7 @@ rd_kafka_broker_controller_async (rd_kaf
+
+ int rd_kafka_brokers_add0 (rd_kafka_t *rk, const char *brokerlist);
+ void rd_kafka_broker_set_state (rd_kafka_broker_t *rkb, int state);
++void rd_kafka_broker_set_recovery_action (rd_kafka_broker_t* rkbs, uint32_t
action);
+
+ void rd_kafka_broker_fail (rd_kafka_broker_t *rkb,
+int level, rd_kafka_resp_err_t err,
+diff -rupN orig/src/rdkafka_broker.c patched/src/rdkafka_broker.c
+--- orig/src/rdkafka_broker.c 2021-01-25 23:25:19.0 +0100
patched/src/rdkafka_broker.c 2021-02-22 16:43:33.391455200 +0100
+@@ -272,6 +272,74 @@ int16_t rd_kafka_broker_ApiVersion_suppo
+ return maxver;
+ }
+
++/**
++ * @brief Setup the wake fd for IO events
++ *
++ * @locality broker creation or reconnection
++ * @locks none
++ */
++static void rd_kafka_broker_setup_queue_wakeup_fd(rd_kafka_broker_t* rkb) {
++int r;
++
++/*
++ * Fd-based queue wake-ups using a non-blocking pipe.
++ * Writes are best effort, if the socket queue is full
++ * the write fails (silently) but this has no effect on latency
++ * since the POLLIN flag will already have been raised for fd.
++ */
++rkb->rkb_wakeup_fd[0] = -1;
++rkb->rkb_wakeup_fd[1] = -1;
++if ((r = rd_pipe_nonblocking(rkb->rkb_wakeup_fd)) == -1) {
++rd_rkb_log(rkb, LOG_ERR, "WAKEUPFD",
++"Failed to setup broker queue wake-up fds: "
++"%s: disabling low-latency mode",
++rd_strerror(r));
++
++}
++else if (rkb->rkb_source == RD_KAFKA_INTERNAL) {
++/* nop: internal broker has no IO transpor
[nifi-minifi-cpp] branch main updated: MINIFICPP-1445 - Move docker integration tests to python behave
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new ae74606 MINIFICPP-1445 - Move docker integration tests to python
behave
ae74606 is described below
commit ae746065319c89b6df23ef6a1bd6902306087cb8
Author: Adam Hunyadi
AuthorDate: Tue Jan 12 11:56:03 2021 +0100
MINIFICPP-1445 - Move docker integration tests to python behave
Signed-off-by: Arpad Boda
This closes #995
---
docker/DockerVerify.sh | 25 +-
docker/Dockerfile | 1 +
.../integration/MiNiFi_integration_test_driver.py | 210 +
docker/test/integration/README.md | 193 +---
docker/test/integration/__init__.py| 0
docker/test/integration/environment.py | 27 +++
.../features/file_system_operations.feature| 40
docker/test/integration/features/http.feature | 60 +
docker/test/integration/features/https.feature | 23 ++
docker/test/integration/features/kafka.feature | 59 +
docker/test/integration/features/s2s.feature | 45
docker/test/integration/features/s3.feature| 155 +
docker/test/integration/minifi/__init__.py | 69 --
docker/test/integration/minifi/core/Cluster.py | 2 +-
docker/test/integration/minifi/core/Connectable.py | 45 +---
.../integration/minifi/core/ControllerService.py | 1 +
.../integration/minifi/core/DockerTestCluster.py | 192
.../minifi/core/DockerTestDirectoryBindings.py | 104 +
.../integration/minifi/core/FileSystemObserver.py | 46
.../integration/minifi/core/OutputEventHandler.py | 15 +-
docker/test/integration/minifi/core/Processor.py | 11 +
.../integration/minifi/core/RemoteProcessGroup.py | 10 +-
.../test/integration/minifi/core/SSL_cert_utils.py | 54 +
.../minifi/core/SingleNodeDockerCluster.py | 153 ++--
.../minifi/processors/GenerateFlowFile.py | 3 +-
.../test/integration/minifi/processors/GetFile.py | 2 +-
.../integration/minifi/processors/HashContent.py | 8 +
.../integration/minifi/processors/InvokeHTTP.py| 18 +-
.../integration/minifi/processors/ListenHTTP.py| 4 +-
.../minifi/processors/PublishKafkaSSL.py | 16 --
.../test/integration/minifi/processors/PutFile.py | 2 +-
.../integration/minifi/processors/PutS3Object.py | 2 +-
docker/test/integration/steps/steps.py | 256 +
docker/test/integration/test_filesystem_ops.py | 51
docker/test/integration/test_filter_zero_file.py | 36 ---
docker/test/integration/test_hash_content.py | 32 ---
docker/test/integration/test_http.py | 57 -
docker/test/integration/test_rdkafka.py| 98
docker/test/integration/test_s2s.py| 38 ---
docker/test/integration/test_s3.py | 144
docker/test/integration/test_zero_file.py | 36 ---
docker/test/test_https.py | 100
42 files changed, 1287 insertions(+), 1156 deletions(-)
diff --git a/docker/DockerVerify.sh b/docker/DockerVerify.sh
index f4efbc9..dc897c2 100755
--- a/docker/DockerVerify.sh
+++ b/docker/DockerVerify.sh
@@ -58,7 +58,8 @@ if ! command swig -version &> /dev/null; then
fi
pip install --upgrade \
-pytest \
+behave \
+pytimeparse \
docker \
PyYAML \
m2crypto \
@@ -71,4 +72,24 @@ export PATH
PYTHONPATH="${PYTHONPATH}:${docker_dir}/test/integration"
export PYTHONPATH
-exec pytest -s -v "${docker_dir}"/test/integration
+BEHAVE_OPTS="-f pretty --logging-level INFO --no-capture"
+
+cd "${docker_dir}/test/integration"
+exec
+ behave $BEHAVE_OPTS "features/file_system_operations.feature" -n "Get and
put operations run in a simple flow" &&
+ behave $BEHAVE_OPTS "features/file_system_operations.feature" -n "PutFile
does not overwrite a file that already exists" &&
+ behave $BEHAVE_OPTS "features/s2s.feature" -n "A MiNiFi instance produces
and transfers data to a NiFi instance via s2s" &&
+ behave $BEHAVE_OPTS "features/s2s.feature" -n "Zero length files are
transfered between via s2s if the \"drop empty\" connection property is false"
&&
+ behave $BEHAVE_OPTS "features/s2s.feature" -n "Zero length files are not
transfered between via s2s if the \"drop empty\" connection property is true" &&
+ behave $BEHAVE_OPTS "features/http.feature" -n "A MiNiFi instance tran
[nifi-minifi-cpp] branch main updated: Introduce ccache in MacOS builds
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 281801f Introduce ccache in MacOS builds
281801f is described below
commit 281801faff6c51037fdeabfa012ea5361baaa290
Author: Gabor Gyimesi
AuthorDate: Mon Feb 15 10:00:18 2021 +0100
Introduce ccache in MacOS builds
Signed-off-by: Arpad Boda
This closes #1007
---
.github/workflows/ci.yml | 32
1 file changed, 28 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f85f3bb..b894a30 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -4,14 +4,26 @@ jobs:
macos_xcode_11_2_1:
name: "macos-xcode11.2.1"
runs-on: macos-10.15
-timeout-minutes: 60
+timeout-minutes: 90
+env:
+ CCACHE_BASEDIR: ${{ GITHUB.WORKSPACE }}
+ CCACHE_DIR: ${{ GITHUB.WORKSPACE }}/.ccache
steps:
- id: checkout
uses: actions/checkout@v2
+ - id: cache
+uses: actions/cache@v2
+with:
+ path: ${{ env.CCACHE_DIR }}
+ key: macos-xcode11.2.1-ccache-${{github.ref}}-${{github.sha}}
+ restore-keys: |
+macos-xcode11.2.1-ccache-${{github.ref}}-
+macos-xcode11.2.1-ccache-refs/heads/main-
- id: install_dependencies
-run: brew install ossp-uuid boost flex openssl python lua@5.3 xz
libssh2
+run: brew install ossp-uuid boost flex openssl python lua@5.3 xz
libssh2 ccache
- id: setup_env
run: |
+ echo
"PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH"
>> $GITHUB_ENV
echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts > /dev/null
sudo xcode-select -switch /Applications/Xcode_11.2.1.app
- id: build
@@ -22,14 +34,26 @@ jobs:
macos_xcode_12_0:
name: "macos-xcode12.0"
runs-on: macos-10.15
-timeout-minutes: 60
+timeout-minutes: 90
+env:
+ CCACHE_BASEDIR: ${{ GITHUB.WORKSPACE }}
+ CCACHE_DIR: ${{ GITHUB.WORKSPACE }}/.ccache
steps:
- id: checkout
uses: actions/checkout@v2
+ - id: cache
+uses: actions/cache@v2
+with:
+ path: ${{ env.CCACHE_DIR }}
+ key: macos-xcode12.0-ccache-${{github.ref}}-${{github.sha}}
+ restore-keys: |
+macos-xcode12.0-ccache-${{github.ref}}-
+macos-xcode12.0-ccache-refs/heads/main-
- id: install_dependencies
-run: brew install ossp-uuid boost flex openssl python lua@5.3 xz
libssh2
+run: brew install ossp-uuid boost flex openssl python lua@5.3 xz
libssh2 ccache
- id: setup_env
run: |
+ echo
"PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH"
>> $GITHUB_ENV
echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts > /dev/null
sudo xcode-select -switch /Applications/Xcode_12.app
- id: build
[nifi-minifi-cpp] branch main updated (53df4c2 -> 0d74aa4)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 53df4c2 MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI add 0d74aa4 MINIFICPP-1498 Add libarchive to minimal docker image No new revisions were added by this update. Summary of changes: docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[nifi-minifi-cpp] 01/02: MINIFICPP-1491: Fixing build failures in Tensorflow extension
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 4518628bd502ce99f456283607fd419d385a
Author: Martin Zink
AuthorDate: Mon Feb 15 15:04:43 2021 +0100
MINIFICPP-1491: Fixing build failures in Tensorflow extension
Signed-off-by: Arpad Boda
This closes #1001
---
extensions/tensorflow/TFExtractTopLabels.cpp| 4 ++--
libminifi/test/tensorflow-tests/TensorFlowTests.cpp | 20 ++--
2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/extensions/tensorflow/TFExtractTopLabels.cpp
b/extensions/tensorflow/TFExtractTopLabels.cpp
index 9bcb067..f0853c1 100644
--- a/extensions/tensorflow/TFExtractTopLabels.cpp
+++ b/extensions/tensorflow/TFExtractTopLabels.cpp
@@ -133,11 +133,11 @@ int64_t
TFExtractTopLabels::LabelsReadCallback::process(const std::shared_ptrgetSize()) {
+ while (total_read < stream->size()) {
auto read = stream->read(reinterpret_cast([0]),
static_cast(buf_size));
for (auto i = 0; i < read; i++) {
- if (buf[i] == '\n' || total_read + i == stream->getSize()) {
+ if (buf[i] == '\n' || total_read + i == stream->size()) {
labels_->emplace_back(label.substr(0, label_size));
label_size = 0;
} else {
diff --git a/libminifi/test/tensorflow-tests/TensorFlowTests.cpp
b/libminifi/test/tensorflow-tests/TensorFlowTests.cpp
index 0334a2d..d4dc27c 100644
--- a/libminifi/test/tensorflow-tests/TensorFlowTests.cpp
+++ b/libminifi/test/tensorflow-tests/TensorFlowTests.cpp
@@ -45,8 +45,8 @@ TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]") { //
NOLINT
auto repo = std::make_shared();
// Define directory for input protocol buffers
- std::string in_dir("/tmp/gt.XX");
- REQUIRE(testController.createTempDirectory(_dir[0]) != nullptr);
+ char in_dir_format[] = "/tmp/gt.XX";
+ std::string in_dir = testController.createTempDirectory(in_dir_format);
// Define input graph protocol buffer file
std::string in_graph_file(in_dir);
@@ -57,8 +57,8 @@ TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]") { //
NOLINT
in_tensor_file.append("/tensor.pb");
// Define directory for output protocol buffers
- std::string out_dir("/tmp/gt.XX");
- REQUIRE(testController.createTempDirectory(_dir[0]) != nullptr);
+ char out_dir_format[] = "/tmp/gt.XX";
+ std::string out_dir = testController.createTempDirectory(out_dir_format);
// Define output tensor protocol buffer file
std::string out_tensor_file(out_dir);
@@ -184,16 +184,16 @@ TEST_CASE("TensorFlow: ConvertImageToTensor",
"[tfConvertImageToTensor]") { // N
auto repo = std::make_shared();
// Define directory for input protocol buffers
- std::string in_dir("/tmp/gt.XX");
- REQUIRE(testController.createTempDirectory(_dir[0]) != nullptr);
+ char in_dir_format[] = "/tmp/gt.XX";
+ std::string in_dir = testController.createTempDirectory(in_dir_format);
// Define input tensor protocol buffer file
std::string in_img_file(in_dir);
in_img_file.append("/img");
// Define directory for output protocol buffers
- std::string out_dir("/tmp/gt.XX");
- REQUIRE(testController.createTempDirectory(_dir[0]) != nullptr);
+ char out_dir_format[] = "/tmp/gt.XX";
+ std::string out_dir = testController.createTempDirectory(out_dir_format);
// Define output tensor protocol buffer file
std::string out_tensor_file(out_dir);
@@ -312,8 +312,8 @@ TEST_CASE("TensorFlow: Extract Top Labels",
"[tfExtractTopLabels]") { // NOLINT
auto repo = std::make_shared();
// Define directory for input protocol buffers
- std::string in_dir("/tmp/gt.XX");
- REQUIRE(testController.createTempDirectory(_dir[0]) != nullptr);
+ char in_dir_format[] = "/tmp/gt.XX";
+ std::string in_dir = testController.createTempDirectory(in_dir_format);
// Define input labels file
std::string in_labels_file(in_dir);
[nifi-minifi-cpp] 02/02: MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 53df4c282c58bc239fcf03d52aabc3c3b801ea69
Author: Martin Zink
AuthorDate: Wed Feb 10 09:21:45 2021 +0100
MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI
Signed-off-by: Arpad Boda
This closes #1000
---
extensions/http-curl/tests/ThreadPoolAdjust.cpp | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/extensions/http-curl/tests/ThreadPoolAdjust.cpp
b/extensions/http-curl/tests/ThreadPoolAdjust.cpp
index 1f55ef8..cb59aad 100644
--- a/extensions/http-curl/tests/ThreadPoolAdjust.cpp
+++ b/extensions/http-curl/tests/ThreadPoolAdjust.cpp
@@ -31,9 +31,11 @@
#include "processors/LogAttribute.h"
#include "utils/IntegrationTestUtils.h"
+constexpr uint64_t WAITTIME_MSECS = 5000;
+
class HttpTestHarness : public IntegrationBase {
public:
- HttpTestHarness() {
+ HttpTestHarness() : IntegrationBase(WAITTIME_MSECS) {
char format[] = "/tmp/ssth.XX";
dir = testController.createTempDirectory(format);
}
[nifi-minifi-cpp] branch main updated (a52faa5 -> 53df4c2)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from a52faa5 MINIFICPP-1485 Improve 'exclusive property' error message new 4518628 MINIFICPP-1491: Fixing build failures in Tensorflow extension new 53df4c2 MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: extensions/http-curl/tests/ThreadPoolAdjust.cpp | 4 +++- extensions/tensorflow/TFExtractTopLabels.cpp| 4 ++-- libminifi/test/tensorflow-tests/TensorFlowTests.cpp | 20 ++-- 3 files changed, 15 insertions(+), 13 deletions(-)
[nifi-minifi-cpp] branch main updated (1a41b44 -> f6d6ee7)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 1a41b44 Add build number parameter for alpine based images add f6d6ee7 MINIFICPP-1468 - Fix transient failures in mock S3 checks No new revisions were added by this update. Summary of changes: .../test/integration/minifi/core/DockerTestCluster.py | 7 ++- docker/test/integration/minifi/core/utils.py | 18 ++ 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 docker/test/integration/minifi/core/utils.py
[nifi-minifi-cpp] branch main updated: Add build number parameter for alpine based images
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 1a41b44 Add build number parameter for alpine based images
1a41b44 is described below
commit 1a41b44495aa9b955b96779ff311239d8eea5f07
Author: Gabor Gyimesi
AuthorDate: Tue Feb 9 10:58:09 2021 +0100
Add build number parameter for alpine based images
Signed-off-by: Arpad Boda
This closes #998
---
cmake/DockerConfig.cmake | 2 ++
1 file changed, 2 insertions(+)
diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake
index 8db6c23..c935f27 100644
--- a/cmake/DockerConfig.cmake
+++ b/cmake/DockerConfig.cmake
@@ -55,6 +55,7 @@ add_custom_target(
-c DISABLE_SCRIPTING=${DISABLE_SCRIPTING}
-c DISABLE_CONTROLLER=${DISABLE_CONTROLLER}
-c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE}
+-c BUILD_NUMBER=${BUILD_NUMBER}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/)
# Create minimal docker image
@@ -66,6 +67,7 @@ add_custom_target(
-v
${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH}
-i minimal
-c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE}
+-c BUILD_NUMBER=${BUILD_NUMBER}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/)
add_custom_target(
[nifi-minifi-cpp] 01/02: MINIFICPP-1460: ProcessSession should penalise flowfiles on rollback
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 44493c07f2b907463912f12065256071df4d21dd
Author: Martin Zink
AuthorDate: Thu Feb 4 09:26:01 2021 +0100
MINIFICPP-1460: ProcessSession should penalise flowfiles on rollback
Signed-off-by: Arpad Boda
This closes #991
---
libminifi/src/core/ProcessSession.cpp | 1 +
libminifi/test/unit/ProcessSessionTests.cpp | 28
2 files changed, 29 insertions(+)
diff --git a/libminifi/src/core/ProcessSession.cpp
b/libminifi/src/core/ProcessSession.cpp
index f8f885e..83ee2aa 100644
--- a/libminifi/src/core/ProcessSession.cpp
+++ b/libminifi/src/core/ProcessSession.cpp
@@ -791,6 +791,7 @@ void ProcessSession::rollback() {
auto flowFile = it.second.modified;
// restore flowFile to original state
*flowFile = *it.second.snapshot;
+ penalize(flowFile);
logger_->log_debug("ProcessSession rollback for %s, record %s, to
connection %s",
process_context_->getProcessorNode()->getName(),
flowFile->getUUIDStr(),
diff --git a/libminifi/test/unit/ProcessSessionTests.cpp
b/libminifi/test/unit/ProcessSessionTests.cpp
index 0847a4c..45b592c 100644
--- a/libminifi/test/unit/ProcessSessionTests.cpp
+++ b/libminifi/test/unit/ProcessSessionTests.cpp
@@ -74,3 +74,31 @@ TEST_CASE("ProcessSession::existsFlowFileInRelationship
works", "[existsFlowFile
REQUIRE(process_session.existsFlowFileInRelationship(Failure));
REQUIRE(process_session.existsFlowFileInRelationship(Success));
}
+
+TEST_CASE("ProcessSession::rollback penalizes affected flowfiles",
"[rollback]") {
+ Fixture fixture;
+ core::ProcessSession _session = fixture.processSession();
+
+ const auto flow_file_1 = process_session.create();
+ const auto flow_file_2 = process_session.create();
+ const auto flow_file_3 = process_session.create();
+ process_session.transfer(flow_file_1, Success);
+ process_session.transfer(flow_file_2, Success);
+ process_session.transfer(flow_file_3, Success);
+ process_session.commit();
+
+ auto next_flow_file_to_be_processed = process_session.get();
+ REQUIRE(next_flow_file_to_be_processed == flow_file_1);
+ next_flow_file_to_be_processed = process_session.get();
+ REQUIRE(next_flow_file_to_be_processed == flow_file_2);
+ REQUIRE_FALSE(flow_file_1->isPenalized());
+ REQUIRE_FALSE(flow_file_2->isPenalized());
+ REQUIRE_FALSE(flow_file_3->isPenalized());
+
+ process_session.rollback();
+ REQUIRE(flow_file_1->isPenalized());
+ REQUIRE(flow_file_2->isPenalized());
+ REQUIRE_FALSE(flow_file_3->isPenalized());
+ next_flow_file_to_be_processed = process_session.get();
+ REQUIRE(next_flow_file_to_be_processed == flow_file_3);
+}
[nifi-minifi-cpp] 02/02: MINIFICPP-1484 fix linter when not in a git tree
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit b69562aa6212cad9c0761d01936e016739a0bec6
Author: Marton Szasz
AuthorDate: Mon Feb 8 17:44:45 2021 +0100
MINIFICPP-1484 fix linter when not in a git tree
Signed-off-by: Arpad Boda
This closes #997
---
thirdparty/google-styleguide/run_linter.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/thirdparty/google-styleguide/run_linter.sh
b/thirdparty/google-styleguide/run_linter.sh
index 9f79313..ca0a1ad 100755
--- a/thirdparty/google-styleguide/run_linter.sh
+++ b/thirdparty/google-styleguide/run_linter.sh
@@ -40,4 +40,5 @@ done
HEADERS=`find $INCLUDE_DIRS -name '*.h' | sort | uniq | tr '\n' ' '`
SOURCES=`find $SOURCE_DIRS -name '*.cpp' | sort | uniq | tr '\n' ' '`
-python ${SCRIPT_DIR}/cpplint.py --linelength=200 ${HEADERS} ${SOURCES}
+REPOSITORY="$(realpath --physical "$(dirname "$0")/../..")"
+python ${SCRIPT_DIR}/cpplint.py --linelength=200 --repository="$REPOSITORY"
${HEADERS} ${SOURCES}
[nifi-minifi-cpp] branch main updated (85f05b0 -> b69562a)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 85f05b0 MINIFICPP-1455 - Fix FileStream error handling and reporting new 44493c0 MINIFICPP-1460: ProcessSession should penalise flowfiles on rollback new b69562a MINIFICPP-1484 fix linter when not in a git tree The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: libminifi/src/core/ProcessSession.cpp | 1 + libminifi/test/unit/ProcessSessionTests.cpp | 28 thirdparty/google-styleguide/run_linter.sh | 3 ++- 3 files changed, 31 insertions(+), 1 deletion(-)
[nifi-minifi-cpp] branch main updated (b977773 -> 85f05b0)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from b93 MINIFICPP-1434: Fix flaky CSite2SiteTests on macOS new 1e4c820 MINIFICPP-1477 - fix StringUtils::trim new 85f05b0 MINIFICPP-1455 - Fix FileStream error handling and reporting The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: extensions/expression-language/Expression.cpp | 5 +- libminifi/include/io/FileStream.h | 3 + libminifi/include/utils/StringUtils.h | 4 +- libminifi/include/utils/ValueParser.h | 2 +- libminifi/src/io/FileStream.cpp | 82 --- libminifi/test/TestBase.h | 4 ++ libminifi/test/unit/FileStreamTests.cpp | 81 ++ 7 files changed, 154 insertions(+), 27 deletions(-)
[nifi-minifi-cpp] 02/02: MINIFICPP-1455 - Fix FileStream error handling and reporting
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 85f05b08ed8f5d10b2745159d0f64475aa741230
Author: Martin Zink
AuthorDate: Tue Jan 26 19:27:06 2021 +0100
MINIFICPP-1455 - Fix FileStream error handling and reporting
Signed-off-by: Arpad Boda
This closes #987
---
libminifi/include/io/FileStream.h | 3 ++
libminifi/src/io/FileStream.cpp | 82 +
libminifi/test/TestBase.h | 4 ++
libminifi/test/unit/FileStreamTests.cpp | 81
4 files changed, 150 insertions(+), 20 deletions(-)
diff --git a/libminifi/include/io/FileStream.h
b/libminifi/include/io/FileStream.h
index 54b8d59..3156a30 100644
--- a/libminifi/include/io/FileStream.h
+++ b/libminifi/include/io/FileStream.h
@@ -86,6 +86,9 @@ class FileStream : public io::BaseStream {
*/
int write(const uint8_t *value, int size) override;
+ private:
+ void seekToEndOfFile(const char* caller_error_msg);
+
std::mutex file_lock_;
std::unique_ptr file_stream_;
size_t offset_;
diff --git a/libminifi/src/io/FileStream.cpp b/libminifi/src/io/FileStream.cpp
index 700ec9f..20d4e6d 100644
--- a/libminifi/src/io/FileStream.cpp
+++ b/libminifi/src/io/FileStream.cpp
@@ -31,6 +31,19 @@ namespace nifi {
namespace minifi {
namespace io {
+constexpr const char *FILE_OPENING_ERROR_MSG = "Error opening file: ";
+constexpr const char *READ_ERROR_MSG = "Error reading from file: ";
+constexpr const char *WRITE_ERROR_MSG = "Error writing to file: ";
+constexpr const char *SEEK_ERROR_MSG = "Error seeking in file: ";
+constexpr const char *INVALID_FILE_STREAM_ERROR_MSG = "invalid file stream";
+constexpr const char *TELLG_CALL_ERROR_MSG = "tellg call on file stream
failed";
+constexpr const char *INVALID_BUFFER_ERROR_MSG = "invalid buffer";
+constexpr const char *FLUSH_CALL_ERROR_MSG = "flush call on file stream
failed";
+constexpr const char *WRITE_CALL_ERROR_MSG = "write call on file stream
failed";
+constexpr const char *EMPTY_MESSAGE_ERROR_MSG = "empty message";
+constexpr const char *SEEKG_CALL_ERROR_MSG = "seekg call on file stream
failed";
+constexpr const char *SEEKP_CALL_ERROR_MSG = "seekp call on file stream
failed";
+
FileStream::FileStream(const std::string , bool append)
: logger_(logging::LoggerFactory::getLogger()),
path_(path),
@@ -38,14 +51,22 @@ FileStream::FileStream(const std::string , bool append)
file_stream_ = std::unique_ptr(new std::fstream());
if (append) {
file_stream_->open(path.c_str(), std::fstream::in | std::fstream::out |
std::fstream::app | std::fstream::binary);
-file_stream_->seekg(0, file_stream_->end);
-file_stream_->seekp(0, file_stream_->end);
-std::streamoff len = file_stream_->tellg();
-length_ = len > 0 ? gsl::narrow(len) : 0;
-seek(offset_);
+if (file_stream_->is_open()) {
+ seekToEndOfFile(FILE_OPENING_ERROR_MSG);
+ auto len = file_stream_->tellg();
+ if (len == std::streampos(-1))
+logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG <<
TELLG_CALL_ERROR_MSG;
+ length_ = len > 0 ? gsl::narrow(len) : 0;
+ seek(offset_);
+} else {
+ logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << path << " " <<
strerror(errno);
+}
} else {
file_stream_->open(path.c_str(), std::fstream::out | std::fstream::binary);
length_ = 0;
+if (!file_stream_->is_open()) {
+ logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << path << " " <<
strerror(errno);
+}
}
}
@@ -59,15 +80,16 @@ FileStream::FileStream(const std::string , uint32_t
offset, bool write_enab
} else {
file_stream_->open(path.c_str(), std::fstream::in | std::fstream::binary);
}
- file_stream_->seekg(0, file_stream_->end);
- file_stream_->seekp(0, file_stream_->end);
- std::streamoff len = file_stream_->tellg();
- if (len > 0) {
-length_ = gsl::narrow(len);
+ if (file_stream_->is_open()) {
+seekToEndOfFile(FILE_OPENING_ERROR_MSG);
+auto len = file_stream_->tellg();
+if (len == std::streampos(-1))
+ logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG <<
TELLG_CALL_ERROR_MSG;
+length_ = len > 0 ? gsl::narrow(len) : 0;
+seek(offset_);
} else {
-length_ = 0;
+logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << path << " " <<
strerror(errno);
}
- seek(offset);
}
void FileStream::close() {
@@ -77,10 +99,16 @@ void FileStream::close() {
void FileStream::seek(uint64_t offset) {
std::lock_guard lock(file_lock_);
+ if (
[nifi-minifi-cpp] 01/02: MINIFICPP-1477 - fix StringUtils::trim
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 1e4c820481cec72d6543fe26f40a5085184d91de
Author: Marton Szasz
AuthorDate: Thu Feb 4 17:04:20 2021 +0100
MINIFICPP-1477 - fix StringUtils::trim
Signed-off-by: Arpad Boda
This closes #990
---
extensions/expression-language/Expression.cpp | 5 +
libminifi/include/utils/StringUtils.h | 4 ++--
libminifi/include/utils/ValueParser.h | 2 +-
3 files changed, 4 insertions(+), 7 deletions(-)
diff --git a/extensions/expression-language/Expression.cpp
b/extensions/expression-language/Expression.cpp
index 545983f..5ce6f4d 100644
--- a/extensions/expression-language/Expression.cpp
+++ b/extensions/expression-language/Expression.cpp
@@ -840,10 +840,7 @@ Value expr_find(const std::vector ) {
#endif // EXPRESSION_LANGUAGE_USE_REGEX
Value expr_trim(const std::vector ) {
- std::string result = args[0].asString();
- auto ws_front = std::find_if_not(result.begin(), result.end(), [](int c)
{return std::isspace(c);});
- auto ws_back = std::find_if_not(result.rbegin(), result.rend(), [](int c)
{return std::isspace(c);}).base();
- return (ws_back <= ws_front ? Value(std::string()) :
Value(std::string(ws_front, ws_back)));
+ return Value{utils::StringUtils::trim(args[0].asString())};
}
Value expr_append(const std::vector ) {
diff --git a/libminifi/include/utils/StringUtils.h
b/libminifi/include/utils/StringUtils.h
index 7dcfc49..54bab6c 100644
--- a/libminifi/include/utils/StringUtils.h
+++ b/libminifi/include/utils/StringUtils.h
@@ -99,7 +99,7 @@ class StringUtils {
* @returns modified string
*/
static inline std::string trimLeft(std::string s) {
-s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](char c) -> bool {
return !isspace(c); }));
+s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](unsigned char c) ->
bool { return !isspace(c); }));
return s;
}
@@ -110,7 +110,7 @@ class StringUtils {
*/
static inline std::string trimRight(std::string s) {
-s.erase(std::find_if(s.rbegin(), s.rend(), [](char c) -> bool { return
!isspace(c); }).base(), s.end());
+s.erase(std::find_if(s.rbegin(), s.rend(), [](unsigned char c) -> bool {
return !isspace(c); }).base(), s.end());
return s;
}
diff --git a/libminifi/include/utils/ValueParser.h
b/libminifi/include/utils/ValueParser.h
index f7b11ba..ce4f0cc 100644
--- a/libminifi/include/utils/ValueParser.h
+++ b/libminifi/include/utils/ValueParser.h
@@ -148,7 +148,7 @@ class ValueParser {
}
void skipWhitespace() {
-while (offset < str.length() && std::isspace(str[offset])) {
+while (offset < str.length() && std::isspace(static_cast(str[offset]))) {
++offset;
}
}
[nifi-minifi-cpp] 03/05: MINIFICPP-1478 - Fix compilation errors of extensions
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit cb54b1051c67be6d9afe611d04b7448b4ddc831e
Author: Gabor Gyimesi
AuthorDate: Thu Feb 4 15:46:04 2021 +0100
MINIFICPP-1478 - Fix compilation errors of extensions
Fix GPS extension compilation
Fix SQLite compilation
Fix compilation of OpenCV extension
Fix openwsman compilation
Fix bustache extension compilation
Add extenstion dependencies to CI runners
Extend timeout for 16.04-all build job
Signed-off-by: Arpad Boda
This closes #993
---
.github/workflows/ci.yml | 8
cmake/BundledBustache.cmake | 5 -
extensions/bustache/ApplyTemplate.cpp | 2 +-
extensions/gps/GetGPS.cpp | 2 +-
extensions/opencv/FrameIO.h | 6 +++---
extensions/opencv/tests/CaptureRTSPFrameTest.cpp | 3 ++-
.../openwsman/processors/SourceInitiatedSubscriptionListener.cpp | 4 ++--
extensions/sqlite/ExecuteSQL.cpp | 2 +-
libminifi/test/bustache-tests/CMakeLists.txt | 1 -
9 files changed, 18 insertions(+), 15 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 211ec49..f85f3bb 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -133,11 +133,11 @@ jobs:
sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
-run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=
-DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT .. && make -j4 VERBOSE=1 && make
test ARGS="--timeout 300 -j2 --output-on-failure"
+run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=
-DENABLE_BUSTACHE=ON -DENABLE_SQLITE=ON -DENABLE_PCAP=ON
-DSTRICT_GSL_CHECKS=AUDIT .. && make -j4 VERBOSE=1 && make test
ARGS="--timeout 300 -j2 --output-on-failure"
ubuntu_16_04_all:
name: "ubuntu-16.04-all"
runs-on: ubuntu-16.04
-timeout-minutes: 60
+timeout-minutes: 90
steps:
- id: checkout
uses: actions/checkout@v2
@@ -153,10 +153,10 @@ jobs:
run: |
sudo apt-add-repository -y "ppa:ubuntu-toolchain-r/test"
sudo apt update
- sudo apt install -y ccache openjdk-8-jdk maven
+ sudo apt install -y ccache openjdk-8-jdk maven libusb-1.0-0-dev
libpng12-dev libgps-dev
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
-run: sudo mount tmpfs -t tmpfs /tmp && ./bootstrap.sh -e -t && cd
build && cmake -DUSE_SHARED_LIBS= -DENABLE_LIBRDKAFKA=ON -DENABLE_OPC=ON
-DENABLE_SFTP=ON -DENABLE_MQTT=ON -DENABLE_COAP=ON -DENABLE_PYTHON=ON
-DENABLE_SQL=ON -DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT .. && cmake --build
. --parallel 4 && make test ARGS="--timeout 300 -j8 --output-on-failure"
+run: sudo mount tmpfs -t tmpfs /tmp && ./bootstrap.sh -e -t && cd
build && cmake -DUSE_SHARED_LIBS= -DENABLE_OPENWSMAN=ON -DENABLE_OPENCV=ON
-DENABLE_MQTT=ON -DENABLE_GPS=ON -DENABLE_USB_CAMERA=ON -DENABLE_LIBRDKAFKA=ON
-DENABLE_OPC=ON -DENABLE_SFTP=ON -DENABLE_MQTT=ON -DENABLE_COAP=ON
-DENABLE_PYTHON=ON -DENABLE_SQL=ON -DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT ..
&& cmake --build . --parallel 4 && make test ARGS="--timeout 300 -j8
--output-on-failure"
debian:
name: "debian"
runs-on: ubuntu-18.04
diff --git a/cmake/BundledBustache.cmake b/cmake/BundledBustache.cmake
index 4ee1804..97d9c4d 100644
--- a/cmake/BundledBustache.cmake
+++ b/cmake/BundledBustache.cmake
@@ -33,15 +33,18 @@ function(use_bundled_bustache SOURCE_DIR BINARY_DIR)
"-DCMAKE_INSTALL_PREFIX=${BUSTACHE_BYPRODUCT_DIR}"
"-DBUSTACHE_ENABLE_TESTING=OFF")
+append_third_party_passthrough_args(BUSTACHE_CMAKE_ARGS
"${BUSTACHE_CMAKE_ARGS}")
+
# Build project
ExternalProject_Add(
bustache-external
-GIT "https://github.com/jamboree/bustache.git;
+GIT_REPOSITORY "https://github.com/jamboree/bustache.git;
GIT_TAG "42dee8ef9bbcae7e9a33500a116cfd9c314662d6"
SOURCE_DIR "${BINARY_DIR}/thirdparty/bustache-src"
CMAKE_ARGS ${BUSTACHE_CMAKE_ARGS}
BUILD_BYPRODUCTS "${BUSTACHE_BYPRODUCT_DIR}/${BYPRODUCT}"
EXCLUDE_FROM_ALL TRUE
+LIST_SEPARA
[nifi-minifi-cpp] 02/05: MINIFICPP-1470 ByteArrayCallback: avoid OOB indexing during address calculation
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit a955854ab0723242079b25b254b85c4f1106affa
Author: Marton Szasz
AuthorDate: Tue Feb 2 16:22:39 2021 +0100
MINIFICPP-1470 ByteArrayCallback: avoid OOB indexing during address
calculation
Signed-off-by: Arpad Boda
This closes #989
---
extensions/http-curl/client/HTTPCallback.h | 4 ++--
libminifi/include/utils/ByteArrayCallback.h | 24
2 files changed, 10 insertions(+), 18 deletions(-)
diff --git a/extensions/http-curl/client/HTTPCallback.h
b/extensions/http-curl/client/HTTPCallback.h
index 827de10..921a8c5 100644
--- a/extensions/http-curl/client/HTTPCallback.h
+++ b/extensions/http-curl/client/HTTPCallback.h
@@ -120,7 +120,7 @@ class HttpStreamingCallback : public ByteInputCallBack {
return ptr_ + relative_pos;
}
- const size_t getRemaining(size_t pos) override {
+ size_t getRemaining(size_t pos) override {
logger_->log_trace("getRemaining(pos: %zu) called", pos);
std::unique_lock lock(mutex_);
@@ -128,7 +128,7 @@ class HttpStreamingCallback : public ByteInputCallBack {
return total_bytes_loaded_ - pos;
}
- const size_t getBufferSize() override {
+ size_t getBufferSize() override {
logger_->log_trace("getBufferSize() called");
std::unique_lock lock(mutex_);
diff --git a/libminifi/include/utils/ByteArrayCallback.h
b/libminifi/include/utils/ByteArrayCallback.h
index cbcdb4c..84e3362 100644
--- a/libminifi/include/utils/ByteArrayCallback.h
+++ b/libminifi/include/utils/ByteArrayCallback.h
@@ -36,13 +36,10 @@ namespace utils {
*/
class ByteInputCallBack : public InputStreamCallback {
public:
- ByteInputCallBack()
- : ptr(nullptr) {
- }
-
- virtual ~ByteInputCallBack() = default;
+ ByteInputCallBack() = default;
+ ~ByteInputCallBack() override = default;
- virtual int64_t process(const std::shared_ptr& stream) {
+ int64_t process(const std::shared_ptr& stream) override {
stream->seek(0);
if (stream->size() > 0) {
@@ -51,34 +48,29 @@ class ByteInputCallBack : public InputStreamCallback {
stream->read(reinterpret_cast(vec.data()),
gsl::narrow(stream->size()));
}
-ptr = reinterpret_cast([0]);
-
return vec.size();
}
- virtual void seek(size_t pos) {
-ptr = [pos];
- }
+ virtual void seek(size_t) { }
virtual void write(std::string content) {
vec.assign(content.begin(), content.end());
-ptr = [0];
}
virtual char *getBuffer(size_t pos) {
-return [pos];
+gsl_Expects(pos <= vec.size());
+return vec.data() + pos;
}
- virtual const size_t getRemaining(size_t pos) {
+ virtual size_t getRemaining(size_t pos) {
return getBufferSize() - pos;
}
- virtual const size_t getBufferSize() {
+ virtual size_t getBufferSize() {
return vec.size();
}
private:
- char *ptr;
std::vector vec;
};
[nifi-minifi-cpp] 01/05: MINIFICPP-1464 synchronize LICENSE and msi/LICENSE.txt
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 92575054631a31a1fd1f1f90d4ae8a2b23527e10 Author: Marton Szasz AuthorDate: Thu Jan 28 16:31:50 2021 +0100 MINIFICPP-1464 synchronize LICENSE and msi/LICENSE.txt Signed-off-by: Arpad Boda This closes #983 --- msi/LICENSE.txt | 1596 +++ 1 file changed, 1386 insertions(+), 210 deletions(-) diff --git a/msi/LICENSE.txt b/msi/LICENSE.txt index 4e0c84b..0f30eab 100644 --- a/msi/LICENSE.txt +++ b/msi/LICENSE.txt @@ -201,9 +201,15 @@ See the License for the specific language governing permissions and limitations under the License. -APACHE NIFI - MINIFI SUBCOMPONENTS: +APACHE NIFI - MINIFI C++ SUBCOMPONENTS: -The Apache NiFi - MiNiFi project contains subcomponents with separate copyright +The Apache NiFi - MiNiFi C++ project contains subcomponents licensed under the Apache License, Version 2.0: +This product bundles 'Simple-Windows-Posix-Semaphore' which is available under an ALv2 license +This project bundles 'mbedTLS' which is available under an ALv2 license +This project bundles 'RocksDB' which is available under an ALv2 license +This project bundles 'AWS SDK for C++' which is available under an ALv2 license + +The Apache NiFi - MiNiFi C++ project contains subcomponents with separate copyright notices and license terms. Your use of the source code for the these subcomponents is subject to the terms and conditions of the following licenses. @@ -238,68 +244,54 @@ This product bundles 'cpplint.py' which is available under a 3-Clause BSD Licen (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -This product bundles 'libuuid' which is available under a "3-clause BSD" license. +This product bundles 'spdlog' which is available under an MIT license. - Copyright (C) 1996, 1997 Theodore Ts'o. + Copyright (c) 2016 Alexander Dalshov. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - 1. Redistributions of source code must retain the above copyright - notice, and the entire permission notice in its entirety, - including the disclaimer of warranties. - 2. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - 3. The name of the author may not be used to endorse or promote - products derived from this software without specific prior - written permission. + Copyright (c) 2019 zvyagin.alexan...@.gmail.com - THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED - WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES - OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF - WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE - LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT - OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR - BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF - LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE - USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH - DAMAGE. + Copyright(c) 2015 Ruslan Baratov. + + Copyright (c) 2015-2019 Gabi Melman. -This product bundles 'spdlog' which is available under an MIT license. + Copyright (c) 2016-2019 spdlog contributors. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR O
[nifi-minifi-cpp] 04/05: MINIFICPP-1481 - Extract flow id from url without bucket id
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 022afb415172618d48876eea0247e23e5bdf990c
Author: Adam Debreceni
AuthorDate: Mon Feb 8 14:31:07 2021 +0100
MINIFICPP-1481 - Extract flow id from url without bucket id
Signed-off-by: Arpad Boda
This closes #996
---
libminifi/src/core/FlowConfiguration.cpp | 13 +
1 file changed, 5 insertions(+), 8 deletions(-)
diff --git a/libminifi/src/core/FlowConfiguration.cpp
b/libminifi/src/core/FlowConfiguration.cpp
index b007b52..232002d 100644
--- a/libminifi/src/core/FlowConfiguration.cpp
+++ b/libminifi/src/core/FlowConfiguration.cpp
@@ -76,14 +76,11 @@ std::unique_ptr
FlowConfiguration::updateFromPayload(const s
if (!url.empty() && payload != nullptr) {
std::string flow_id, bucket_id;
auto path_split = utils::StringUtils::split(url, "/");
-// Registry API docs:
nifi.apache.org/docs/nifi-registry-docs/rest-api/index.html
-// GET /buckets/{bucketId}/flows/{flowId}: Gets a flow
-const auto bucket_token_found = std::find(path_split.cbegin(),
path_split.cend(), "buckets");
-if (bucket_token_found != path_split.cend() &&
std::next(bucket_token_found) != path_split.cend()) {
- bucket_id = *std::next(bucket_token_found);
- const auto flows_token_found = std::find(std::next(bucket_token_found,
2), path_split.cend(), "flows");
- if (flows_token_found != path_split.cend() &&
std::next(flows_token_found) != path_split.cend()) {
-flow_id = *std::next(flows_token_found);
+for (auto it = path_split.cbegin(); it != path_split.cend(); ++it) {
+ if (*it == "flows" && std::next(it) != path_split.cend()) {
+flow_id = *++it;
+ } else if (*it == "buckets" && std::next(it) != path_split.cend()) {
+bucket_id = *++it;
}
}
flow_version_->setFlowVersion(url, bucket_id, flow_id);
