Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
szaszm closed pull request #1708: MINIFICPP-2229 Encrypt sensitive properties in the flow configuration URL: https://github.com/apache/nifi-minifi-cpp/pull/1708 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
lordgamez commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1445961695 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -900,4 +918,131 @@ void StructuredConfiguration::addNewId(const std::string& uuid) { } } +void StructuredConfiguration::encryptSensitivePropertiesInYaml(YAML::Node property_yamls, const std::map& properties) const { + for (auto kv : property_yamls) { Review Comment: I see, thanks for the explanation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1445957906 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -900,4 +918,131 @@ void StructuredConfiguration::addNewId(const std::string& uuid) { } } +void StructuredConfiguration::encryptSensitivePropertiesInYaml(YAML::Node property_yamls, const std::map& properties) const { + for (auto kv : property_yamls) { Review Comment: The yaml_cpp API is quite ~~bad~~unusual: `*YAML::Node::begin()` returns an object by value, so changing the type of `kv` to `const&` would bind it to this temporary object. I think that would be misleading: it's not really `const`, as we modify `property_yamls` (indirectly) through `kv` at line 932; and it's not really a reference to anything inside `property_yamls` (it's a reference to a temporary object). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1445957906 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -900,4 +918,131 @@ void StructuredConfiguration::addNewId(const std::string& uuid) { } } +void StructuredConfiguration::encryptSensitivePropertiesInYaml(YAML::Node property_yamls, const std::map& properties) const { + for (auto kv : property_yamls) { Review Comment: The yaml_cpp API is quite ̶b̶a̶d̶ unusual: `*YAML::Node::begin()` returns an object by value, so changing the type of `kv` to `const&` would bind it to this temporary object. I think that would be misleading: it's not really `const`, as we modify `property_yamls` (indirectly) through `kv` at line 932; and it's not really a reference to anything inside `property_yamls` (it's a reference to a temporary object). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
lordgamez commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1445844105 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -900,4 +918,131 @@ void StructuredConfiguration::addNewId(const std::string& uuid) { } } +void StructuredConfiguration::encryptSensitivePropertiesInYaml(YAML::Node property_yamls, const std::map& properties) const { + for (auto kv : property_yamls) { Review Comment: Minor: this could be const ref ```suggestion for (const auto& kv : property_yamls) { ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1444694551 ## libminifi/include/core/flow/StructuredConfiguration.h: ## @@ -34,6 +34,8 @@ #include "utils/file/FileSystem.h" #include "core/flow/Node.h" #include "FlowSchema.h" +#include "rapidjson/document.h" +#include "yaml-cpp/yaml.h" Review Comment: I've created https://issues.apache.org/jira/browse/MINIFICPP-2281 for this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
szaszm commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1444537801 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -620,20 +628,26 @@ void StructuredConfiguration::parseRPGPort(const Node& port_node, core::ProcessG } void StructuredConfiguration::parsePropertyValueSequence(const std::string& property_name, const Node& property_value_node, core::ConfigurableComponent& component) { + core::Property myProp(property_name, "", ""); + component.getProperty(property_name, myProp); + for (const auto& nodeVal : property_value_node) { if (nodeVal) { Node propertiesNode = nodeVal["value"]; - // must insert the sequence in differently. - const auto rawValueString = propertiesNode.getString().value(); - logger_->log_debug("Found {}={}", property_name, rawValueString); + auto rawValueString = propertiesNode.getString().value(); + if (myProp.isSensitive()) { +rawValueString = decryptProperty(rawValueString); + } + logger_->log_debug("Found property {}", property_name); Review Comment: I don't think it's a big problem to break properties that use the `enc{`/`}` wrappers on their own, but it's also not ideal. To limit the set of broken property values, we could check the format of the string inside the wrapper, and only decrypt if it looks like what our encrypted values look. Maybe it's best to leave it as is for now, and figure this out later if/when necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
szaszm commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1444537801 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -620,20 +628,26 @@ void StructuredConfiguration::parseRPGPort(const Node& port_node, core::ProcessG } void StructuredConfiguration::parsePropertyValueSequence(const std::string& property_name, const Node& property_value_node, core::ConfigurableComponent& component) { + core::Property myProp(property_name, "", ""); + component.getProperty(property_name, myProp); + for (const auto& nodeVal : property_value_node) { if (nodeVal) { Node propertiesNode = nodeVal["value"]; - // must insert the sequence in differently. - const auto rawValueString = propertiesNode.getString().value(); - logger_->log_debug("Found {}={}", property_name, rawValueString); + auto rawValueString = propertiesNode.getString().value(); + if (myProp.isSensitive()) { +rawValueString = decryptProperty(rawValueString); + } + logger_->log_debug("Found property {}", property_name); Review Comment: I don't think it's a big problem to break properties that use the `enc{`/`}` wrappers on their own, but it's also not ideal. To limit the set of broken property values, we could check the format of the string inside the wrapper, and only act if it looks like what our encrypted values look. Maybe it's best to leave it as is for now, and figure this out later if/when necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
adamdebreceni commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r125152 ## libminifi/include/core/flow/StructuredConfiguration.h: ## @@ -34,6 +34,8 @@ #include "utils/file/FileSystem.h" #include "core/flow/Node.h" #include "FlowSchema.h" +#include "rapidjson/document.h" +#include "yaml-cpp/yaml.h" Review Comment: although it would be nice not to have this this dependency here, I understand that it would require extra indirections/tests which would bloat this PR, could we create a ticket to tuck the clone/modification/save implementations away like we do with flow::Node? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1444320712 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -620,20 +628,26 @@ void StructuredConfiguration::parseRPGPort(const Node& port_node, core::ProcessG } void StructuredConfiguration::parsePropertyValueSequence(const std::string& property_name, const Node& property_value_node, core::ConfigurableComponent& component) { + core::Property myProp(property_name, "", ""); + component.getProperty(property_name, myProp); + for (const auto& nodeVal : property_value_node) { if (nodeVal) { Node propertiesNode = nodeVal["value"]; - // must insert the sequence in differently. - const auto rawValueString = propertiesNode.getString().value(); - logger_->log_debug("Found {}={}", property_name, rawValueString); + auto rawValueString = propertiesNode.getString().value(); + if (myProp.isSensitive()) { +rawValueString = decryptProperty(rawValueString); + } + logger_->log_debug("Found property {}", property_name); Review Comment: Yes, good point. We could even make the change now; the one drawback is that no properties would be allowed to start with `enc{` and end with `}`. I don't know if that is a big problem. Alternatively, we could introduce a new (and possibly temporary) `wasSensitive()` flag if/when we change a property from sensitive to non-sensitive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
szaszm commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1443124349 ## libminifi/src/core/flow/StructuredConfiguration.cpp: ## @@ -620,20 +628,26 @@ void StructuredConfiguration::parseRPGPort(const Node& port_node, core::ProcessG } void StructuredConfiguration::parsePropertyValueSequence(const std::string& property_name, const Node& property_value_node, core::ConfigurableComponent& component) { + core::Property myProp(property_name, "", ""); + component.getProperty(property_name, myProp); + for (const auto& nodeVal : property_value_node) { if (nodeVal) { Node propertiesNode = nodeVal["value"]; - // must insert the sequence in differently. - const auto rawValueString = propertiesNode.getString().value(); - logger_->log_debug("Found {}={}", property_name, rawValueString); + auto rawValueString = propertiesNode.getString().value(); + if (myProp.isSensitive()) { +rawValueString = decryptProperty(rawValueString); + } + logger_->log_debug("Found property {}", property_name); Review Comment: Could this be changed in the future to allow decrypting properties that are _not_ marked as sensitive, without breaking backwards compatibility? I'm thinking about a scenario where we accidentally marked a property as sensitive that's not really sensitive, so we make the property non-sensitive, but existing configs have the encrypted value after upgrade to this hypothetical future version. I'm not requesting the change now, just want to make sure that should we ever make a property non-sensitive, we can also make this change then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#issuecomment-1878680825 > I found a couple more properties we might wanna flag as sensitive Thanks! I have added them in d3228e93aa06a98189d08005ba4e4095ef007a04. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
martinzink commented on PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#issuecomment-1878611363 I found a couple more properties we might wanna flag as sensitive https://github.com/fgerlits/nifi-minifi-cpp/blob/MINIFICPP-2229_Encrypt-sensitive-properties-in-the-flow-definition/extensions/splunk/SplunkHECProcessor.h#L48-L51 https://github.com/fgerlits/nifi-minifi-cpp/blob/MINIFICPP-2229_Encrypt-sensitive-properties-in-the-flow-definition/extensions/elasticsearch/ElasticsearchCredentialsControllerService.h#L41-L44 https://github.com/fgerlits/nifi-minifi-cpp/blob/MINIFICPP-2229_Encrypt-sensitive-properties-in-the-flow-definition/extensions/sql/services/DatabaseService.h#L55-L58 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1427928523 ## libminifi/src/core/FlowConfiguration.cpp: ## @@ -174,4 +180,21 @@ std::shared_ptr FlowConfiguration::crea return controllerServicesNode; } +std::string FlowConfiguration::decryptProperty(const std::string& encrypted_value) const { + static constexpr std::string_view WrapperBegin = "enc{"; + static constexpr std::string_view WrapperEnd = "}"; + + if (!(encrypted_value.starts_with(WrapperBegin) && encrypted_value.ends_with(WrapperEnd))) { +// this is normal: sensitive properties come from the C2 server in cleartext over TLS +return encrypted_value; + } + + const std::string unwrapped_value = encrypted_value.substr(WrapperBegin.size(), encrypted_value.length() - (WrapperBegin.size() + WrapperEnd.size())); + return sensitive_properties_encryptor_.decrypt(unwrapped_value); Review Comment: I have changed this and a few of the related functions to take a `string_view` argument in 51d37c808a977631653b7a71e8e16a14aebf0b4c -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1427928523 ## libminifi/src/core/FlowConfiguration.cpp: ## @@ -174,4 +180,21 @@ std::shared_ptr FlowConfiguration::crea return controllerServicesNode; } +std::string FlowConfiguration::decryptProperty(const std::string& encrypted_value) const { + static constexpr std::string_view WrapperBegin = "enc{"; + static constexpr std::string_view WrapperEnd = "}"; + + if (!(encrypted_value.starts_with(WrapperBegin) && encrypted_value.ends_with(WrapperEnd))) { +// this is normal: sensitive properties come from the C2 server in cleartext over TLS +return encrypted_value; + } + + const std::string unwrapped_value = encrypted_value.substr(WrapperBegin.size(), encrypted_value.length() - (WrapperBegin.size() + WrapperEnd.size())); + return sensitive_properties_encryptor_.decrypt(unwrapped_value); Review Comment: I have changed this and a few of the related functions to take a `string_view` argument in 38b383ab2e4810505116a11c2a3c5e06b77f9243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
szaszm commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1426968034 ## libminifi/src/core/FlowConfiguration.cpp: ## @@ -174,4 +180,21 @@ std::shared_ptr FlowConfiguration::crea return controllerServicesNode; } +std::string FlowConfiguration::decryptProperty(const std::string& encrypted_value) const { + static constexpr std::string_view WrapperBegin = "enc{"; + static constexpr std::string_view WrapperEnd = "}"; + + if (!(encrypted_value.starts_with(WrapperBegin) && encrypted_value.ends_with(WrapperEnd))) { +// this is normal: sensitive properties come from the C2 server in cleartext over TLS +return encrypted_value; + } + + const std::string unwrapped_value = encrypted_value.substr(WrapperBegin.size(), encrypted_value.length() - (WrapperBegin.size() + WrapperEnd.size())); + return sensitive_properties_encryptor_.decrypt(unwrapped_value); Review Comment: I think it would be better to change this snippet and the encryption providers to pass the data as `std::string_view` instead of `const std::string&`. If you think this change has its place in this PR, consider changing it. If not, maybe just this `decryptProperty` function can be changed to take `std::string_view`, it makes a copy of the value (substring) anyway. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1426957069 ## extensions/azure/processors/AzureBlobStorageProcessorBase.h: ## @@ -51,10 +51,13 @@ class AzureBlobStorageProcessorBase : public AzureStorageProcessorBase { .withDescription("The storage account key. This is an admin-like password providing access to every container in this account. " "It is recommended one uses Shared Access Signature (SAS) token instead for fine-grained control with policies.") .supportsExpressionLanguage(true) + .isSensitive(true) .build(); EXTENSIONAPI static constexpr auto SASToken = core::PropertyDefinitionBuilder<>::createProperty("SAS Token") .withDescription("Shared Access Signature token. Specify either SAS Token (recommended) or Storage Account Key together with Storage Account Name if Managed Identity is not used.") - .supportsExpressionLanguage(true).build(); + .supportsExpressionLanguage(true) Review Comment: `onSchedule` only does a sanity check that `SASToken` is non-empty; in `AzureBlobStorageProcessorBase::getAzureCredentialsFromProperties()` we use `getProperty()` with a flow file argument -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
fgerlits commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1426957582 ## libminifi/src/core/FlowConfiguration.cpp: ## @@ -174,4 +181,22 @@ std::shared_ptr FlowConfiguration::crea return controllerServicesNode; } +std::string FlowConfiguration::decryptProperty(const std::string& encrypted_value) const { + static const utils::Regex is_encrypted{R"(enc\{.*\})"}; + if (!utils::regexMatch(encrypted_value, is_encrypted)) { Review Comment: good idea, fixed in e3c2ca4db93eed4e6ea18b49662bc16be1fb2477 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2229 Encrypt sensitive properties in the flow configuration [nifi-minifi-cpp]
szaszm commented on code in PR #1708: URL: https://github.com/apache/nifi-minifi-cpp/pull/1708#discussion_r1426672898 ## extensions/azure/processors/AzureBlobStorageProcessorBase.h: ## @@ -51,10 +51,13 @@ class AzureBlobStorageProcessorBase : public AzureStorageProcessorBase { .withDescription("The storage account key. This is an admin-like password providing access to every container in this account. " "It is recommended one uses Shared Access Signature (SAS) token instead for fine-grained control with policies.") .supportsExpressionLanguage(true) + .isSensitive(true) .build(); EXTENSIONAPI static constexpr auto SASToken = core::PropertyDefinitionBuilder<>::createProperty("SAS Token") .withDescription("Shared Access Signature token. Specify either SAS Token (recommended) or Storage Account Key together with Storage Account Name if Managed Identity is not used.") - .supportsExpressionLanguage(true).build(); + .supportsExpressionLanguage(true) Review Comment: I don't think this supports EL. The EL extension only overrides the getProperty overloads that take a flow file for context, and the `onSchedule` of this class calls the overload without a flow file parameter. ## libminifi/src/core/FlowConfiguration.cpp: ## @@ -174,4 +181,22 @@ std::shared_ptr FlowConfiguration::crea return controllerServicesNode; } +std::string FlowConfiguration::decryptProperty(const std::string& encrypted_value) const { + static const utils::Regex is_encrypted{R"(enc\{.*\})"}; + if (!utils::regexMatch(encrypted_value, is_encrypted)) { Review Comment: It may be premature optimization, but I'd avoid regex here, and use `starts_with("enc{")` and `ends_with('}')` instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org