szaszm commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1668618742
##########
CONFIGURE.md:
##########
@@ -121,8 +124,101 @@ It's recommended to create your configuration in YAML
format or configure the ag
max concurrent tasks: 1
Properties:
+Besides YAML configuration format, MiNiFi C++ also supports JSON
configuration. To see different uses cases in both formats, please refer to the
[examples page](examples/README.md) for flow config examples.
+
**NOTE:** Make sure to specify id for each component (Processor, Connection,
Controller, RPG etc.) to make sure that Apache MiNiFi C++ can reload the state
after a process restart. The id should be unique in the flow configuration.
+### Parameter Contexts
+
+Processor properties in flow configurations can be parameterized using
parameters defined in parameter contexts. Flow configurations can define
parameter contexts that define parameter-value pairs to be reused in the flow
configuration as per the following rules:
+ - Parameters within a process group can only be used if the parameter context
is assigned to that process group.
+ - Each process group can be assigned only one parameter context.
+ - Parameter contexts can be assigned to multiple process groups.
+ - The assigned parameter context has the scope of the process group, but not
its child process groups.
+ - The parameters can be used in the flow configuration by using the
`#{parameterName}` syntax for processor properties.
+ - Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens ( - ), underscores (
_ ), periods ( . ), and spaces are allowed in parameter name.
+ - `#` character can be used to escape the parameter syntax. E.g. if the
`parameterName` parameter's value is `xxx` then `#{parameterName}` will be
replaced with `xxx`, `##{parameterName}` will be replaced with
`#{parameterName}`, and `#####{parameterName}` will be replaced with `##xxx`.
+ - Parameters can only be used in values of non-sensitive processor properties.
+
+An example for using parameters in a JSON configuration file:
+
+```json
+{
+ "parameterContexts": [
+ {
+ "identifier": "804e6b47-ea22-45cd-a472-545801db98e6",
+ "name": "root-process-group-context",
+ "description": "Root process group parameter context",
+ "parameters": [
+ {
+ "name": "tail_base_dir",
+ "description": "Base dir of tailed files",
+ "value": "/tmp/tail/file/path"
+ }
+ ]
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Tail test_file1.log",
+ "identifier": "83b58f9f-e661-4634-96fb-0e82b92becdf",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "#{tail_base_dir}/test_file1.log"
+ }
+ },
+ {
+ "name": "Tail test_file2.log",
+ "identifier": "8a772a10-7c34-48e7-b152-b1a32c5db83e",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "#{tail_base_dir}/test_file2.log"
+ }
+ }
+ ],
+ "parameterContextName": "root-process-group-context"
Review Comment:
Are the parameter contexts identified by name in nifi too?
##########
libminifi/src/core/flow/StructuredConfiguration.cpp:
##########
@@ -799,6 +862,19 @@ void StructuredConfiguration::parsePorts(const flow::Node&
node, core::ProcessGr
}
}
+void StructuredConfiguration::parseParameterContextName(const flow::Node&
node, core::ProcessGroup* parent) {
+ if (!parent) {
+ logger_->log_error("parseParameterContextName: no parent group was
provided");
+ return;
+ }
Review Comment:
this could take a reference instead, that way no null checking is necessary
##########
libminifi/include/core/ParameterContext.h:
##########
@@ -0,0 +1,61 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include <string>
+#include <unordered_set>
+#include <optional>
+
+#include "Core.h"
+
+namespace org::apache::nifi::minifi::core {
+
+class ParameterException : public std::runtime_error {
Review Comment:
This could be derived from minifi::Exception. Maybe have its own type enum
value
##########
extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp:
##########
@@ -1072,3 +1075,581 @@ TEST_CASE("Test serialization", "[YamlConfiguration]") {
const std::string serialized_flow_definition_masked =
std::regex_replace(serialized_flow_definition, std::regex{"enc\\{.*\\}"},
"enc{...}");
CHECK(serialized_flow_definition_masked ==
TEST_FLOW_WITH_SENSITIVE_PROPERTIES_ENCRYPTED);
}
+
+TEST_CASE("Yaml configuration can use parameter contexts",
"[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow =
yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Batch Size") == "12");
+ REQUIRE(proc->getProperty("Lookup frequency") == "12 min");
+}
+
+TEST_CASE("Yaml config should not replace parameter from different parameter
context", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: other-context
+ description: my other context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML),
"Parameter 'batch_size' not found");
+}
+
+TEST_CASE("Cannot use the same parameter context name twice",
"[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML),
"Parameter context name 'my-context' already exists, parameter context names
must be unique!");
+}
+
+TEST_CASE("Cannot use the same parameter name within a parameter context
twice", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: lookup.frequency
+ description: ''
+ value: 1 min
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML),
"Parameter name 'lookup.frequency' already exists, parameter names must be
unique within a parameter context!");
+}
+
+class DummyFlowYamlProcessor : public core::Processor {
+ public:
+ using core::Processor::Processor;
+
+ static constexpr const char* Description = "A processor that does nothing.";
+ static constexpr auto SimpleProperty =
core::PropertyDefinitionBuilder<>::createProperty("Simple Property")
+ .withDescription("Just a simple string property")
+ .build();
+ static constexpr auto SensitiveProperty =
core::PropertyDefinitionBuilder<>::createProperty("Sensitive Property")
+ .withDescription("Sensitive property")
+ .isSensitive(true)
+ .build();
+ static constexpr auto Properties = std::array<core::PropertyReference,
2>{SimpleProperty, SensitiveProperty};
+ static constexpr auto Relationships =
std::array<core::RelationshipDefinition, 0>{};
+ static constexpr bool SupportsDynamicProperties = true;
+ static constexpr bool SupportsDynamicRelationships = true;
+ static constexpr core::annotation::Input InputRequirement =
core::annotation::Input::INPUT_ALLOWED;
+ static constexpr bool IsSingleThreaded = false;
+ ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS
+
+ void initialize() override { setSupportedProperties(Properties); }
+};
+
+REGISTER_RESOURCE(DummyFlowYamlProcessor, Processor);
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property",
"[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML),
"Non-sensitive parameter 'my_value' cannot be referenced in a sensitive
property");
+}
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property value
sequence", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property:
+ - value: first value
+ - value: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML),
"Non-sensitive parameter 'my_value' cannot be referenced in a sensitive
property");
+}
+
+TEST_CASE("Parameters can be used in nested process groups",
"[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376456
+ name: sub-context
+ description: my sub context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: 1
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+Process Groups:
+ - id: 2a3aaf32-8574-4fa7-b720-84001f8dde43
+ name: Sub process group
+ Processors:
+ - id: 12304f28-0158-1000-0000-000000000000
+ name: SubTailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: 1 sec
+ Parameter Context Name: sub-context
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow =
yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ CHECK(proc->getProperty("Batch Size") == "1");
+ CHECK(proc->getProperty("Lookup frequency") == "12 min");
+ auto* subproc = flow->findProcessorByName("SubTailFile");
+ REQUIRE(subproc);
+ CHECK(subproc->getProperty("Batch Size") == "12");
+ CHECK(subproc->getProperty("Lookup frequency") == "1 sec");
+}
+
+TEST_CASE("Subprocessgroups cannot inherit parameters from parent
processgroup", "[YamlConfiguration]") {
Review Comment:
Is this on purpose? I thought they'd be inherited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
