[GitHub] [nifi] mattyb149 opened a new pull request #4717: NIFI-8080: Compile Jython scripts before evaluating
mattyb149 opened a new pull request #4717:
URL: https://github.com/apache/nifi/pull/4717
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Apply the performance improvement from ScriptedTransformRecord to
JythonScriptEngineConfigurator (compile script first).
After running the (currently) ignored added unit test, I noticed up to a 10x
performance improvement in ExecuteScript.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8080) Apply ScriptedTransformRecord approach to Jython for other scripted components
Matt Burgess created NIFI-8080: -- Summary: Apply ScriptedTransformRecord approach to Jython for other scripted components Key: NIFI-8080 URL: https://issues.apache.org/jira/browse/NIFI-8080 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess Assignee: Matt Burgess ScriptedTransformRecord (NIFI-7572) has code specifically for Jython (since it implements the Compilable interface) to compile the script first, resulting in up to 10x performance improvements. The same can be done for JythonScriptEngineConfigurator so all other scripted components can make use of those improvements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mtien-apache commented on pull request #4673: NIFI-8019 Added TlsPlatform to provide runtime default TLS Protocols
mtien-apache commented on pull request #4673: URL: https://github.com/apache/nifi/pull/4673#issuecomment-741499574 +1 LGTM. I ran a full build and ran each test class that was changed with: - JDK 8u231 - AdoptOpenJDK 8 Update 275 - JDK 11.0.5 I debugged through`TlsPlatform` and verified it gets the correct TLS protocols during runtime according to the Java versions. I also verified the tests were able to build when disabling legacy TLS protocols of TLSv1 and TLSv1.1 by configuring the `java.security` file. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4712: NIFI-7885 Added Environment Variable to deny LFS access using Hadoop
exceptionfactory commented on pull request #4712: URL: https://github.com/apache/nifi/pull/4712#issuecomment-741487912 Thanks! It looks like the MacOS build failed on an unrelated unit test that is also impacting other recent builds. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4216: NIFI-7356 Enable TLS for embedded Zookeeper when NiFi has TLS enabled
thenatog commented on pull request #4216: URL: https://github.com/apache/nifi/pull/4216#issuecomment-741136069 Unless there's any objections, to get some progress on this one I might take these changes and rebase with the latest TLS + Zookeeper changes that have already been merged, and make the review changes suggested above, and submit a new PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mark-weghorst commented on pull request #4508: NIFI-6576 add basic auth to confluent schema registry service
mark-weghorst commented on pull request #4508: URL: https://github.com/apache/nifi/pull/4508#issuecomment-741103015 @pvillard31 yes I'm still planning on re-working this to address the code review comments from @alopresto Unfortunately, I haven't had the time to revisit this due to some more pressing work, I should have some time towards the end of the week to pick back up on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on a change in pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog commented on a change in pull request #4715:
URL: https://github.com/apache/nifi/pull/4715#discussion_r538839172
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
##
@@ -4053,18 +4062,158 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
def verifyTool = new ConfigEncryptionTool()
verifyTool.isVerbose = true
verifyTool.flowXmlPath = workingFlowXmlFile.path
-String updatedFlowXmlContent = verifyTool.loadFlowXml()
+InputStream updatedFlowXmlContent =
verifyTool.loadFlowXml(workingFlowXmlFile.path)
-// Check that the flow.xml.gz content changed
-assert updatedFlowXmlContent != ORIGINAL_FLOW_XML_CONTENT
+def migratedFlowCipherTexts =
findFieldsInStream(updatedFlowXmlContent, WFXCTR)
// Verify that the cipher texts decrypt correctly
logger.info("Original flow.xml.gz cipher texts:
${originalFlowCipherTexts}")
-def flowCipherTexts = updatedFlowXmlContent.findAll(WFXCTR)
-logger.info("Updated flow.xml.gz cipher texts:
${flowCipherTexts}")
-assert flowCipherTexts.size() == CIPHER_TEXT_COUNT
-flowCipherTexts.every {
-assert ConfigEncryptionTool.decryptFlowElement(it,
newFlowPassword) == "thisIsABadPassword"
+logger.info("Updated flow.xml.gz cipher texts:
${migratedFlowCipherTexts}")
+assert migratedFlowCipherTexts.size() == CIPHER_TEXT_COUNT
+migratedFlowCipherTexts.each {
+String decryptedValue =
ConfigEncryptionTool.decryptFlowElement(it, newFlowPassword)
+logger.info("Decrypted value of migrated
${workingFlowXmlFile.path} was: ${decryptedValue}")
+assert decryptedValue == PASSWORD || decryptedValue ==
ANOTHER_PASSWORD
+}
+}
+})
+
+// Act
+ConfigEncryptionTool.main(args)
+logger.info("Invoked #main with ${args.join(" ")}")
+
+// Assert
+
+// Assertions defined above
Review comment:
These comments relate to formatting tests in a format of Arrange, Act,
and Assert sections. The way the test is designed in this case changes the
layout of how that format looks, but I guess the intention is the same. Looks
like most of the tests have these comments in there. Should I remove?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] thenatog commented on a change in pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog commented on a change in pull request #4715:
URL: https://github.com/apache/nifi/pull/4715#discussion_r538839172
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
##
@@ -4053,18 +4062,158 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
def verifyTool = new ConfigEncryptionTool()
verifyTool.isVerbose = true
verifyTool.flowXmlPath = workingFlowXmlFile.path
-String updatedFlowXmlContent = verifyTool.loadFlowXml()
+InputStream updatedFlowXmlContent =
verifyTool.loadFlowXml(workingFlowXmlFile.path)
-// Check that the flow.xml.gz content changed
-assert updatedFlowXmlContent != ORIGINAL_FLOW_XML_CONTENT
+def migratedFlowCipherTexts =
findFieldsInStream(updatedFlowXmlContent, WFXCTR)
// Verify that the cipher texts decrypt correctly
logger.info("Original flow.xml.gz cipher texts:
${originalFlowCipherTexts}")
-def flowCipherTexts = updatedFlowXmlContent.findAll(WFXCTR)
-logger.info("Updated flow.xml.gz cipher texts:
${flowCipherTexts}")
-assert flowCipherTexts.size() == CIPHER_TEXT_COUNT
-flowCipherTexts.every {
-assert ConfigEncryptionTool.decryptFlowElement(it,
newFlowPassword) == "thisIsABadPassword"
+logger.info("Updated flow.xml.gz cipher texts:
${migratedFlowCipherTexts}")
+assert migratedFlowCipherTexts.size() == CIPHER_TEXT_COUNT
+migratedFlowCipherTexts.each {
+String decryptedValue =
ConfigEncryptionTool.decryptFlowElement(it, newFlowPassword)
+logger.info("Decrypted value of migrated
${workingFlowXmlFile.path} was: ${decryptedValue}")
+assert decryptedValue == PASSWORD || decryptedValue ==
ANOTHER_PASSWORD
+}
+}
+})
+
+// Act
+ConfigEncryptionTool.main(args)
+logger.info("Invoked #main with ${args.join(" ")}")
+
+// Assert
+
+// Assertions defined above
Review comment:
These comments relate to formatting tests in a format of Arrange, Act,
and Assert sections. The way the test is designed in this case changes the
layout of how that format looks, but I guess the intention is the same. I can
remove these comments though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] thenatog commented on a change in pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog commented on a change in pull request #4715:
URL: https://github.com/apache/nifi/pull/4715#discussion_r538836447
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -64,8 +68,8 @@ class ConfigEncryptionTool {
public String outputLoginIdentityProvidersPath
public String authorizersPath
public String outputAuthorizersPath
-public String flowXmlPath
-public String outputFlowXmlPath
+public static flowXmlPath
+public static outputFlowXmlPath
Review comment:
These were changed to static to allow static access in static main() for
calling loadFlowXml(flowXmlPath). I changed this slightly to take the path as a
parameter to allow easier testing for the loadFlowXml method. Not sure how
successful it was though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8066) Bump GRPC dependency versions
[ https://issues.apache.org/jira/browse/NIFI-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-8066: -- Status: Patch Available (was: Open) > Bump GRPC dependency versions > - > > Key: NIFI-8066 > URL: https://issues.apache.org/jira/browse/NIFI-8066 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.12.1 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Update dependencies in nifi-grpc module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] turcsanyip opened a new pull request #4716: NIFI-8066: Bump GRPC dependency versions
turcsanyip opened a new pull request #4716: URL: https://github.com/apache/nifi/pull/4716 https://issues.apache.org/jira/browse/NIFI-8066 In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on JDK 8? - [ ] Have you verified that the full build is successful on JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
exceptionfactory commented on a change in pull request #4715:
URL: https://github.com/apache/nifi/pull/4715#discussion_r538821132
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
##
@@ -4053,18 +4062,158 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
def verifyTool = new ConfigEncryptionTool()
verifyTool.isVerbose = true
verifyTool.flowXmlPath = workingFlowXmlFile.path
-String updatedFlowXmlContent = verifyTool.loadFlowXml()
+InputStream updatedFlowXmlContent =
verifyTool.loadFlowXml(workingFlowXmlFile.path)
-// Check that the flow.xml.gz content changed
-assert updatedFlowXmlContent != ORIGINAL_FLOW_XML_CONTENT
+def migratedFlowCipherTexts =
findFieldsInStream(updatedFlowXmlContent, WFXCTR)
// Verify that the cipher texts decrypt correctly
logger.info("Original flow.xml.gz cipher texts:
${originalFlowCipherTexts}")
-def flowCipherTexts = updatedFlowXmlContent.findAll(WFXCTR)
-logger.info("Updated flow.xml.gz cipher texts:
${flowCipherTexts}")
-assert flowCipherTexts.size() == CIPHER_TEXT_COUNT
-flowCipherTexts.every {
-assert ConfigEncryptionTool.decryptFlowElement(it,
newFlowPassword) == "thisIsABadPassword"
+logger.info("Updated flow.xml.gz cipher texts:
${migratedFlowCipherTexts}")
+assert migratedFlowCipherTexts.size() == CIPHER_TEXT_COUNT
+migratedFlowCipherTexts.each {
+String decryptedValue =
ConfigEncryptionTool.decryptFlowElement(it, newFlowPassword)
+logger.info("Decrypted value of migrated
${workingFlowXmlFile.path} was: ${decryptedValue}")
+assert decryptedValue == PASSWORD || decryptedValue ==
ANOTHER_PASSWORD
+}
+}
+})
+
+// Act
+ConfigEncryptionTool.main(args)
+logger.info("Invoked #main with ${args.join(" ")}")
+
+// Assert
+
+// Assertions defined above
Review comment:
It looks like this comment and the one above should be removed.
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
##
@@ -287,7 +291,10 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
// Assert
assert !TestAppender.events.isEmpty()
-assert TestAppender.events.first().message =~ "The source
nifi.properties and destination nifi.properties are identical \\[.*\\] so the
original will be overwritten"
+assert TestAppender.events.stream().any() {
+it.message =~ "The source nifi.properties and destination
nifi.properties are identical \\[.*\\] so the original will be overwritten"
+}
+//assert TestAppender.events.first().message =~ "The source
nifi.properties and destination nifi.properties are identical \\[.*\\] so the
original will be overwritten"
Review comment:
It looks like this commented line should be removed.
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -64,8 +68,8 @@ class ConfigEncryptionTool {
public String outputLoginIdentityProvidersPath
public String authorizersPath
public String outputAuthorizersPath
-public String flowXmlPath
-public String outputFlowXmlPath
+public static flowXmlPath
+public static outputFlowXmlPath
Review comment:
Is there a reason these values are static?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4712: NIFI-7885 Added Environment Variable to deny LFS access using Hadoop
exceptionfactory commented on pull request #4712: URL: https://github.com/apache/nifi/pull/4712#issuecomment-741050871 Thanks for the confirmation. I refactored the approach to perform the check in `customValidate()` with the `FileSystem.getDefaultUri()` method, which performs some normalization to ensure that if the `fs.defaultFS` value equals `local`, it will be normalized to start with `file:///`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] bbende commented on a change in pull request #4676: NIFI-8016: Support dynamic properties set on the DataSource in Hive connection pools
bbende commented on a change in pull request #4676:
URL: https://github.com/apache/nifi/pull/4676#discussion_r538811709
##
File path:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/dbcp/hive/HiveConnectionPool.java
##
@@ -199,6 +206,26 @@ protected void init(final
ControllerServiceInitializationContext context) {
return properties;
}
+@Override
+protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final
String propertyDescriptorName) {
+final PropertyDescriptor.Builder builder = new
PropertyDescriptor.Builder()
+.name(propertyDescriptorName)
+.required(false)
+
.addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING,
true))
+
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.dynamic(true)
+
.addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING,
true))
+
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR);
Review comment:
Don't think it would cause a problem, but noticed these two validators
seems to be duplicated.
Also, seems like maybe clearer to not set expressionLanguagSupported above,
since its set below in the if/else?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7884) Separate "read-filesystem" restricted permission into local file system and HDFS file system permissions
[
https://issues.apache.org/jira/browse/NIFI-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Bende updated NIFI-7884:
--
Fix Version/s: 1.13.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Separate "read-filesystem" restricted permission into local file system and
> HDFS file system permissions
>
>
> Key: NIFI-7884
> URL: https://issues.apache.org/jira/browse/NIFI-7884
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Extensions
>Affects Versions: 1.12.1
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: file-system, hdfs, restricted, security
> Fix For: 1.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently the {{read-filesystem}} value for {{RequiredPermission}} is used
> for both the processors which read directly from the local file system of the
> machine hosting NiFi ({{GetFile}}, {{ListFile}}, etc.) and the processors
> which read from external file systems like HDFS ({{GetHDFS}}, {{PutHDFS}},
> etc.). There are use cases where NiFi users should be able to interact with
> the HDFS file system without having permissions to access the local file
> system.
> This will also require introducing a global setting in {{nifi.properties}}
> that an admin can set to allow local file system access via the HDFS
> processors (default {{true}} for backward compatibility), and additional
> validation logic in the HDFS processors (ideally the abstract shared logic)
> to ensure that if this setting is disabled, the HDFS processors are not
> accessing the local file system via the {{file:///}} protocol in their
> configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7884) Separate "read-filesystem" restricted permission into local file system and HDFS file system permissions
[
https://issues.apache.org/jira/browse/NIFI-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246142#comment-17246142
]
ASF subversion and git services commented on NIFI-7884:
---
Commit cfbcecc4c6ceae805bf26588a4c587d4ca37763a in nifi's branch
refs/heads/main from exceptionfactory
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=cfbcecc ]
NIFI-7884 Added and applied Distributed File System permissions (#4713)
> Separate "read-filesystem" restricted permission into local file system and
> HDFS file system permissions
>
>
> Key: NIFI-7884
> URL: https://issues.apache.org/jira/browse/NIFI-7884
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Extensions
>Affects Versions: 1.12.1
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: file-system, hdfs, restricted, security
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the {{read-filesystem}} value for {{RequiredPermission}} is used
> for both the processors which read directly from the local file system of the
> machine hosting NiFi ({{GetFile}}, {{ListFile}}, etc.) and the processors
> which read from external file systems like HDFS ({{GetHDFS}}, {{PutHDFS}},
> etc.). There are use cases where NiFi users should be able to interact with
> the HDFS file system without having permissions to access the local file
> system.
> This will also require introducing a global setting in {{nifi.properties}}
> that an admin can set to allow local file system access via the HDFS
> processors (default {{true}} for backward compatibility), and additional
> validation logic in the HDFS processors (ideally the abstract shared logic)
> to ensure that if this setting is disabled, the HDFS processors are not
> accessing the local file system via the {{file:///}} protocol in their
> configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] bbende merged pull request #4713: NIFI-7884 Added and applied Distributed File System permissions
bbende merged pull request #4713: URL: https://github.com/apache/nifi/pull/4713 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] bbende commented on pull request #4712: NIFI-7885 Added Environment Variable to deny LFS access using Hadoop
bbende commented on pull request #4712: URL: https://github.com/apache/nifi/pull/4712#issuecomment-740996110 Ah that is a good point that I had not considered. I think we can say that for all the processors we provide we are ensuring none of them circumvent the check in `preProcessConfiguration`. If someone implements a custom processor, well then all bets are off, they didn't even have to extend from `AbstractHadoopProcessor `and can do anything they want at that point. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory edited a comment on pull request #4712: NIFI-7885 Added Environment Variable to deny LFS access using Hadoop
exceptionfactory edited a comment on pull request #4712: URL: https://github.com/apache/nifi/pull/4712#issuecomment-740983198 @bbende Thanks for the review and feedback. The `resetHDFSResources()` method handles Kerberos setup options prior to calling `getFileSystemAsUser()`, which includes calling `preProcessConfiguration()` that subclasses can override to adjust the `Configuration`. Existing subclasses do not appear to do anything that would influence the file system type, so as long as that doesn't seem like a concern, changing the behavior to fail validation instead of throwing the AccessDeniedException during scheduling sounds like a better approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4712: NIFI-7885 Added Environment Variable to deny LFS access using Hadoop
exceptionfactory commented on pull request #4712: URL: https://github.com/apache/nifi/pull/4712#issuecomment-740983198 @bbende Thanks for the review and feedback. The `resetHDFSResources()` method handles Kerberos setup options prior to calling `getFileSystemAsUser()`, which includes calling `preProcessConfiguration()` that subclasses can override to adjust the `Configuration`. Existing subclasses do not appear to do anything that would influence the file system type, so as long as that doesn't seem like a concern, changing the behavior to fail validation instead of throwing the AccessDeniedException during scheduling sounds ike a better approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog commented on pull request #4715: URL: https://github.com/apache/nifi/pull/4715#issuecomment-740937392 Updated PR to remove junk comments and improve readability etc. as per your recommendation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] bbende commented on pull request #4712: NIFI-7885 Added Environment Variable to deny LFS access using Hadoop
bbende commented on pull request #4712: URL: https://github.com/apache/nifi/pull/4712#issuecomment-740930522 Code looks good, also built and tested this and seems to be working as described. With the check inside `getFileSystemAsUser`, that first gets called during `onScheduled` so the error is produced when the user starts the processor in the UI. We then get an error like: ``` 2020-12-08 19:38:21,329 ERROR org.apache.nifi.processors.hadoop.PutHDFS: PutHDFS[id=4388cbe3-0176-1000--cb860909] HDFS Configuration error - java.nio.file.AccessDeniedException: file:///: LFS Access Denied according to Environment Variable [NIFI_HDFS_DENY_LOCAL_FILE_SYSTEM_ACCESS]: java.nio.file.AccessDeniedException: file:///: LFS Access Denied according to Environment Variable [NIFI_HDFS_DENY_LOCAL_FILE_SYSTEM_ACCESS] java.n 2020-12-08 19:38:21,331 ERROR org.apache.nifi.processors.hadoop.PutHDFS: PutHDFS[id=4388cbe3-0176-1000--cb860909] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.file.AccessDeniedException: file:///: LFS Access Denied according to Environment Variable [NIFI_HDFS_DENY_LOCAL_FILE_SYSTEM_ACCESS]: java.nio.file.AccessDeniedException: file:///: LFS Access Denied according to Environment Variable [NIFI_HDFS_DENY_LOCAL_FILE_SYSTEM_ACCESS] java.nio.file.AccessDeniedException: file:///: LFS Access Denied according to Environment Variable [NIFI_HDFS_DENY_LOCAL_FILE_SYSTEM_ACCESS] ``` At this point if flow files are sent into the processor they remain in the incoming queue since the processor technically isn't running yet, and the framework continues trying to call onScheduled after yielding. I'm wondering if a better user experience would be to perform the check in `customValidate` so that the processor is not valid and can't be started and avoids all of the continuous errors above. We already have the `Configuration` object available and just need to get the value of `fs.defaultFS` and then use your same matcher against that value, thoughts? https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L219 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4714: NIFI-7801 Adding support for HTTP based Splunk put and indexed acknowledgement
exceptionfactory commented on a change in pull request #4714:
URL: https://github.com/apache/nifi/pull/4714#discussion_r538744071
##
File path:
nifi-nar-bundles/nifi-splunk-bundle/nifi-splunk-processors/src/main/java/org/apache/nifi/processors/splunk/SplunkAPICall.java
##
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.splunk;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.splunk.RequestMessage;
+import com.splunk.ResponseMessage;
+import com.splunk.SSLSecurityProtocol;
+import com.splunk.Service;
+import com.splunk.ServiceArgs;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+
+abstract class SplunkAPICall extends AbstractProcessor {
+private static final String REQUEST_CHANNEL_HEADER_NAME =
"X-Splunk-Request-Channel";
+
+private static final String HTTP_SCHEME = "http";
+private static final String HTTPS_SCHEME = "https";
+
+private static final AllowableValue TLS_1_2_VALUE = new
AllowableValue(SSLSecurityProtocol.TLSv1_2.name(),
SSLSecurityProtocol.TLSv1_2.name());
+private static final AllowableValue TLS_1_1_VALUE = new
AllowableValue(SSLSecurityProtocol.TLSv1_1.name(),
SSLSecurityProtocol.TLSv1_1.name());
+private static final AllowableValue TLS_1_VALUE = new
AllowableValue(SSLSecurityProtocol.TLSv1.name(),
SSLSecurityProtocol.TLSv1.name());
+private static final AllowableValue SSL_3_VALUE = new
AllowableValue(SSLSecurityProtocol.SSLv3.name(),
SSLSecurityProtocol.SSLv3.name());
+
+static final PropertyDescriptor SCHEME = new PropertyDescriptor.Builder()
+.name("Scheme")
+.description("The scheme for connecting to Splunk.")
+.allowableValues(HTTPS_SCHEME, HTTP_SCHEME)
+.defaultValue(HTTPS_SCHEME)
+.required(true)
+.build();
+
+static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+.name("Hostname")
+.description("The ip address or hostname of the Splunk server.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("localhost")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+static final PropertyDescriptor PORT = new PropertyDescriptor
+.Builder().name("Port")
+.description("The HTTP Port Number of the Splunk server.")
+.required(true)
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.defaultValue("9088")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+static final PropertyDescriptor SECURITY_PROTOCOL = new
PropertyDescriptor.Builder()
+.name("Security Protocol")
+.description("The security protocol to use for communicating with
Splunk.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.allowableValues(TLS_1_2_VALUE, TLS_1_1_VALUE, TLS_1_VALUE,
SSL_3_VALUE)
+.defaultValue(TLS_1_2_VALUE.getValue())
+.build();
+
+static final PropertyDescriptor OWNER = new PropertyDescriptor.Builder()
+.name("Owner")
+.description("The owner to pass to Splunk.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+static final PropertyDescriptor TOKEN = new PropertyDescriptor.Builder()
+
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
arpadboda closed pull request #947: URL: https://github.com/apache/nifi-minifi-cpp/pull/947 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #940: MINIFICPP-1373 - Implement ConsumeKafka
szaszm commented on a change in pull request #940:
URL: https://github.com/apache/nifi-minifi-cpp/pull/940#discussion_r538697896
##
File path: libminifi/include/utils/GeneralUtils.h
##
@@ -49,6 +49,14 @@ constexpr T intdiv_ceil(T numerator, T denominator) {
: numerator / denominator + (numerator % denominator != 0));
}
+// from https://stackoverflow.com/questions/15202474
+struct identity {
+template
+constexpr auto operator()(U&& v) const noexcept ->
decltype(std::forward(v)) {
+return std::forward(v);
+}
+};
+
Review comment:
Consider transparently falling back to
[`std::identity`](https://en.cppreference.com/w/cpp/utility/functional/identity)
when compiled on >=C++20
##
File path: libminifi/src/utils/StringUtils.cpp
##
@@ -59,13 +63,21 @@ std::vector StringUtils::split(const
std::string &str, const std::s
break;
}
auto next = std::find_if(curr, end, is_func);
-result.push_back(std::string(curr, next));
+result.push_back(transformation(std::string(curr, next)));
Review comment:
I would take `transformation` by value. You could also forward it to the
call, but this doesn't make sense when it's called more than once, because the
move case would call a moved-from function object.
If the caller needs to keep state, they can use `std::ref` and pass by value.
##
File path: libminifi/test/unit/StringUtilsTests.cpp
##
@@ -50,6 +50,16 @@ TEST_CASE("TestStringUtils::split4", "[test split
classname]") {
REQUIRE(expected ==
StringUtils::split(org::apache::nifi::minifi::core::getClassName(),
"::"));
}
+TEST_CASE("TestStringUtils::split5", "[test split delimiter not specified]") {
Review comment:
What was the failure? Most languages split between each character on
empty delimiter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] NissimShiman commented on a change in pull request #4620: NIFI-6242 PutFileTransfer generation incorrect provenance event
NissimShiman commented on a change in pull request #4620:
URL: https://github.com/apache/nifi/pull/4620#discussion_r538713027
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutSFTP.java
##
@@ -243,9 +244,58 @@ public void testPutSFTPBatching() throws IOException {
putSFTPRunner.clearTransferState();
}
+@Test
+public void testPutSFTPProvenanceTransitUri() throws IOException {
+emptyTestDirectory();
+
+putSFTPRunner.setProperty(SFTPTransfer.REJECT_ZERO_BYTE, "false");
+Map attributes = new HashMap<>();
+attributes.put("filename", "testfile.txt");
+attributes.put("transfer-host","localhost");
+
+putSFTPRunner.enqueue(Paths.get(testFile), attributes);
+
+attributes = new HashMap<>();
+attributes.put("filename", "testfile1.txt");
+attributes.put("transfer-host","127.0.0.1");
+
+putSFTPRunner.enqueue(Paths.get(testFile), attributes);
+putSFTPRunner.run();
+
+putSFTPRunner.assertTransferCount(PutSFTP.REL_SUCCESS, 2);
+putSFTPRunner.getProvenanceEvents().forEach(k->{
+assert(k.toString().contains("sftp://localhost";));
+});
+//Two files in batch, should have 2 transferred to success, 0 to
failure
+putSFTPRunner.assertTransferCount(PutSFTP.REL_SUCCESS, 2);
+putSFTPRunner.assertTransferCount(PutSFTP.REL_REJECT, 0);
+
+MockFlowFile flowFile1 =
putSFTPRunner.getFlowFilesForRelationship(PutFileTransfer.REL_SUCCESS).get(0);
+MockFlowFile flowFile2 =
putSFTPRunner.getFlowFilesForRelationship(PutFileTransfer.REL_SUCCESS).get(1);
+putSFTPRunner.clearProvenanceEvents();
+putSFTPRunner.clearTransferState();
+
+//Test different destinations on flow file attributes
+putSFTPRunner.setProperty(SFTPTransfer.HOSTNAME,"${transfer-host}");
//set to derive hostname
+
+putSFTPRunner.enqueue(flowFile1);
+putSFTPRunner.run();
Review comment:
remove putSFTPRunner.run();
test case should pass without it as well
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutFileTransfer.java
##
@@ -95,12 +96,19 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
}
final ComponentLog logger = getLogger();
-final String hostname =
context.getProperty(FileTransfer.HOSTNAME).evaluateAttributeExpressions(flowFile).getValue();
+
+String hostname =
context.getProperty(FileTransfer.HOSTNAME).evaluateAttributeExpressions(flowFile).getValue();
+//Check for constant attribute
+final boolean staticHostname = hostname!=null && !hostname.isEmpty()
&& Objects.equals(hostname, context.getProperty(FileTransfer.HOSTNAME)
+ .getValue());
final int maxNumberOfFiles =
context.getProperty(FileTransfer.BATCH_SIZE).asInteger();
int fileCount = 0;
try (final T transfer = getFileTransfer(context)) {
do {
+if(!staticHostname) {
Review comment:
use:
if (context.getProperty(FileTransfer.HOSTNAME).isExpressionLanguagePresent())
instead
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestFTP.java
##
@@ -144,6 +146,61 @@ public void basicFileUpload() throws IOException {
// Check file was uploaded
Assert.assertTrue(results.exists("c:\\data\\randombytes-1"));
}
+@Test
+public void basicProvenanceEventTest() throws IOException {
+TestRunner runner = TestRunners.newTestRunner(PutFTP.class);
+runner.setValidateExpressionUsage(false);
Review comment:
Assuming you agree with comments above, this can be removed (as well as
from lines 137/138 as well)
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestFTP.java
##
@@ -144,6 +146,61 @@ public void basicFileUpload() throws IOException {
// Check file was uploaded
Assert.assertTrue(results.exists("c:\\data\\randombytes-1"));
}
+@Test
Review comment:
Put a blank line between tests
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestFTP.java
##
@@ -144,6 +146,61 @@ public void basicFileUpload() throws IOException {
// Check file was uploaded
Assert.assertTrue(results.exists("c:\\data\\randombytes-1"));
}
+@Test
+public void basicProvenanceEventTest() throws IOException {
+TestRunner runner = TestRunners.newTestRunner(PutFTP.class);
+runner.setValidateExpressionUsage(false);
+
+
[GitHub] [nifi] thenatog commented on pull request #4599: NIFI-7913 Set Enabled TLS Protocols on ListenSMTP
thenatog commented on pull request #4599: URL: https://github.com/apache/nifi/pull/4599#issuecomment-740829245 Will review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on pull request #920: MINIFICPP-1296 - All tests should use volatile state storage
szaszm commented on pull request #920: URL: https://github.com/apache/nifi-minifi-cpp/pull/920#issuecomment-740814917 35999e0 (on main) broke the build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
szaszm commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538647760
##
File path: libminifi/src/utils/tls/ExtendedKeyUsage.cpp
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifdef OPENSSL_SUPPORT
+
+#include "utils/tls/ExtendedKeyUsage.h"
+
+#include
+
+#include
+#include
+#include
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace tls {
+
+namespace {
+
+struct KeyValuePair {
+ const char* key;
+ uint8_t value;
+};
+constexpr std::array EXT_KEY_USAGE_NAME_TO_BIT_POS{{
+KeyValuePair{"Server Authentication", 1},
+KeyValuePair{"Client Authentication", 2},
+KeyValuePair{"Code Signing", 3},
+KeyValuePair{"Secure Email", 4},
+KeyValuePair{"Time Stamping", 8},
+KeyValuePair{"OCSP Signing", 9}
+}};
+
+} // namespace
+
+void EXTENDED_KEY_USAGE_deleter::operator()(EXTENDED_KEY_USAGE* key_usage)
const { EXTENDED_KEY_USAGE_free(key_usage); }
+
+ExtendedKeyUsage::ExtendedKeyUsage() :
logger_(core::logging::LoggerFactory::getLogger()) {}
+
+ExtendedKeyUsage::ExtendedKeyUsage(const EXTENDED_KEY_USAGE& key_usage_asn1) :
ExtendedKeyUsage{} {
+ const int num_oids = sk_ASN1_OBJECT_num(&key_usage_asn1);
+ for (int i = 0; i < num_oids; ++i) {
+const ASN1_OBJECT* const oid = sk_ASN1_OBJECT_value(&key_usage_asn1, i);
+assert(oid && oid->length > 0);
+const unsigned char bit_pos = oid->data[oid->length - 1];
Review comment:
Ah, I see. I had some trouble reading ASN, but I think I see what you
mean now: id-kp is the same and the last element (byte?) is different.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
fgerlits commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538641311
##
File path: libminifi/src/utils/tls/ExtendedKeyUsage.cpp
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifdef OPENSSL_SUPPORT
+
+#include "utils/tls/ExtendedKeyUsage.h"
+
+#include
+
+#include
+#include
+#include
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace tls {
+
+namespace {
+
+struct KeyValuePair {
+ const char* key;
+ uint8_t value;
+};
+constexpr std::array EXT_KEY_USAGE_NAME_TO_BIT_POS{{
+KeyValuePair{"Server Authentication", 1},
+KeyValuePair{"Client Authentication", 2},
+KeyValuePair{"Code Signing", 3},
+KeyValuePair{"Secure Email", 4},
+KeyValuePair{"Time Stamping", 8},
+KeyValuePair{"OCSP Signing", 9}
+}};
+
+} // namespace
+
+void EXTENDED_KEY_USAGE_deleter::operator()(EXTENDED_KEY_USAGE* key_usage)
const { EXTENDED_KEY_USAGE_free(key_usage); }
+
+ExtendedKeyUsage::ExtendedKeyUsage() :
logger_(core::logging::LoggerFactory::getLogger()) {}
+
+ExtendedKeyUsage::ExtendedKeyUsage(const EXTENDED_KEY_USAGE& key_usage_asn1) :
ExtendedKeyUsage{} {
+ const int num_oids = sk_ASN1_OBJECT_num(&key_usage_asn1);
+ for (int i = 0; i < num_oids; ++i) {
+const ASN1_OBJECT* const oid = sk_ASN1_OBJECT_value(&key_usage_asn1, i);
+assert(oid && oid->length > 0);
+const unsigned char bit_pos = oid->data[oid->length - 1];
Review comment:
In the RFC link https://tools.ietf.org/html/rfc5280#section-4.2.1.12 you
can see that all the OIDs are the same except for the last byte. The ASN.1
format is a list of OIDs; in this class we represent it as a bitfield, that is
not a standard thing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
szaszm commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538636172
##
File path: libminifi/src/controllers/SSLContextService.cpp
##
@@ -128,16 +149,239 @@ bool SSLContextService::configure_ssl_context(SSL_CTX
*ctx) {
}
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, nullptr);
- int retp = SSL_CTX_load_verify_locations(ctx, ca_certificate_.c_str(), 0);
- if (retp == 0) {
-logging::LOG_ERROR(logger_) << "Can not load CA certificate, Exiting, " <<
getLatestOpenSSLErrorString();
+ if (!IsNullOrEmpty(ca_certificate_)) {
+if (SSL_CTX_load_verify_locations(ctx, ca_certificate_.c_str(), 0) == 0) {
+ logging::LOG_ERROR(logger_) << "Cannot load CA certificate, exiting, "
<< getLatestOpenSSLErrorString();
+ return false;
+}
+ }
+
+ if (use_system_cert_store_ && IsNullOrEmpty(certificate_)) {
+if (!addClientCertificateFromSystemStoreToSSLContext(ctx)) {
+ return false;
+}
+ }
+
+ if (use_system_cert_store_ && IsNullOrEmpty(ca_certificate_)) {
+if (!addServerCertificatesFromSystemStoreToSSLContext(ctx)) {
+ return false;
+}
+ }
+
+ return true;
+}
+
+bool SSLContextService::addP12CertificateToSSLContext(SSL_CTX* ctx) const {
+ const auto fp_deleter = [](BIO* ptr) { BIO_free(ptr); };
+ std::unique_ptr fp(BIO_new(BIO_s_file()),
fp_deleter);
+ if (fp == nullptr) {
+logging::LOG_ERROR(logger_) << "Failed create new file BIO, " <<
getLatestOpenSSLErrorString();
+return false;
+ }
+ if (BIO_read_filename(fp.get(), certificate_.c_str()) <= 0) {
+logging::LOG_ERROR(logger_) << "Failed to read certificate file " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+ const auto p12_deleter = [](PKCS12* ptr) { PKCS12_free(ptr); };
+ std::unique_ptr p12(d2i_PKCS12_bio(fp.get(),
nullptr), p12_deleter);
+ if (p12 == nullptr) {
+logging::LOG_ERROR(logger_) << "Failed to DER decode certificate file " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+
+ EVP_PKEY* pkey = nullptr;
+ X509* cert = nullptr;
+ STACK_OF(X509)* ca = nullptr;
+ if (!PKCS12_parse(p12.get(), passphrase_.c_str(), &pkey, &cert, &ca)) {
+logging::LOG_ERROR(logger_) << "Failed to parse certificate file " <<
certificate_ << " as PKCS#12, " << getLatestOpenSSLErrorString();
+return false;
+ }
+ utils::tls::EVP_PKEY_unique_ptr pkey_ptr{pkey};
+ utils::tls::X509_unique_ptr cert_ptr{cert};
+ const auto ca_deleter = gsl::finally([ca] { sk_X509_pop_free(ca, X509_free);
});
+
+ if (SSL_CTX_use_certificate(ctx, cert) != 1) {
+logging::LOG_ERROR(logger_) << "Failed to set certificate from " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+ while (ca != nullptr && sk_X509_num(ca) > 0) {
+utils::tls::X509_unique_ptr cacert{sk_X509_pop(ca)};
+if (SSL_CTX_add_extra_chain_cert(ctx, cacert.get()) != 1) {
+ logging::LOG_ERROR(logger_) << "Failed to set additional certificate
from " << certificate_ << ", " << getLatestOpenSSLErrorString();
+ return false;
+}
+cacert.release(); // a successful SSL_CTX_add_extra_chain_cert() takes
ownership of cacert
+ }
+ if (SSL_CTX_use_PrivateKey(ctx, pkey) != 1) {
+logging::LOG_ERROR(logger_) << "Failed to set private key from " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
return false;
}
return true;
}
-#endif
+
+bool SSLContextService::addPemCertificateToSSLContext(SSL_CTX* ctx) const {
+ if (SSL_CTX_use_certificate_chain_file(ctx, certificate_.c_str()) <= 0) {
+logging::LOG_ERROR(logger_) << "Could not load client certificate " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+
+ if (!IsNullOrEmpty(passphrase_)) {
+void* passphrase = const_cast(&passphrase_);
+SSL_CTX_set_default_passwd_cb_userdata(ctx, passphrase);
+SSL_CTX_set_default_passwd_cb(ctx, minifi::utils::tls::pemPassWordCb);
+ }
+
+ if (!IsNullOrEmpty(private_key_)) {
+int retp = SSL_CTX_use_PrivateKey_file(ctx, private_key_.c_str(),
SSL_FILETYPE_PEM);
+if (retp != 1) {
+ logging::LOG_ERROR(logger_) << "Could not load private key, " << retp <<
" on " << private_key_ << ", " << getLatestOpenSSLErrorString();
+ return false;
+}
+ }
+
+ return true;
+}
+
+bool
SSLContextService::addClientCertificateFromSystemStoreToSSLContext(SSL_CTX*
ctx) const {
+#ifdef WIN32
+ utils::tls::WindowsCertStoreLocation store_location{cert_store_location_};
+ HCERTSTORE hCertStore = CertOpenStore(CERT_STORE_PROV_SYSTEM_A, 0, NULL,
+CERT_STORE_OPEN_EXISTING_FLAG |
CERT_STORE_READONLY_FLAG | store_location.getBitfieldValue(),
+client_cert_store_.data());
+ if (!hCertStore) {
+logger_->log_error("Could not open system certificate store %s/%s (client
certificates)", cert_store_location_, clie
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
szaszm commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538632918
##
File path: libminifi/src/utils/tls/ExtendedKeyUsage.cpp
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifdef OPENSSL_SUPPORT
+
+#include "utils/tls/ExtendedKeyUsage.h"
+
+#include
+
+#include
+#include
+#include
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace tls {
+
+namespace {
+
+struct KeyValuePair {
+ const char* key;
+ uint8_t value;
+};
+constexpr std::array EXT_KEY_USAGE_NAME_TO_BIT_POS{{
+KeyValuePair{"Server Authentication", 1},
+KeyValuePair{"Client Authentication", 2},
+KeyValuePair{"Code Signing", 3},
+KeyValuePair{"Secure Email", 4},
+KeyValuePair{"Time Stamping", 8},
+KeyValuePair{"OCSP Signing", 9}
+}};
+
+} // namespace
+
+void EXTENDED_KEY_USAGE_deleter::operator()(EXTENDED_KEY_USAGE* key_usage)
const { EXTENDED_KEY_USAGE_free(key_usage); }
+
+ExtendedKeyUsage::ExtendedKeyUsage() :
logger_(core::logging::LoggerFactory::getLogger()) {}
+
+ExtendedKeyUsage::ExtendedKeyUsage(const EXTENDED_KEY_USAGE& key_usage_asn1) :
ExtendedKeyUsage{} {
+ const int num_oids = sk_ASN1_OBJECT_num(&key_usage_asn1);
+ for (int i = 0; i < num_oids; ++i) {
+const ASN1_OBJECT* const oid = sk_ASN1_OBJECT_value(&key_usage_asn1, i);
+assert(oid && oid->length > 0);
+const unsigned char bit_pos = oid->data[oid->length - 1];
Review comment:
My problem is that I couldn't find a source that describes the structure
of extended key usage asn1 objects in detail. I don't understand why we index
to the last element (byte?) of the object and why it contains the bit position
that we use below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
fgerlits commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538631795
##
File path: libminifi/src/controllers/SSLContextService.cpp
##
@@ -128,16 +149,239 @@ bool SSLContextService::configure_ssl_context(SSL_CTX
*ctx) {
}
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, nullptr);
- int retp = SSL_CTX_load_verify_locations(ctx, ca_certificate_.c_str(), 0);
- if (retp == 0) {
-logging::LOG_ERROR(logger_) << "Can not load CA certificate, Exiting, " <<
getLatestOpenSSLErrorString();
+ if (!IsNullOrEmpty(ca_certificate_)) {
+if (SSL_CTX_load_verify_locations(ctx, ca_certificate_.c_str(), 0) == 0) {
+ logging::LOG_ERROR(logger_) << "Cannot load CA certificate, exiting, "
<< getLatestOpenSSLErrorString();
+ return false;
+}
+ }
+
+ if (use_system_cert_store_ && IsNullOrEmpty(certificate_)) {
+if (!addClientCertificateFromSystemStoreToSSLContext(ctx)) {
+ return false;
+}
+ }
+
+ if (use_system_cert_store_ && IsNullOrEmpty(ca_certificate_)) {
+if (!addServerCertificatesFromSystemStoreToSSLContext(ctx)) {
+ return false;
+}
+ }
+
+ return true;
+}
+
+bool SSLContextService::addP12CertificateToSSLContext(SSL_CTX* ctx) const {
+ const auto fp_deleter = [](BIO* ptr) { BIO_free(ptr); };
+ std::unique_ptr fp(BIO_new(BIO_s_file()),
fp_deleter);
+ if (fp == nullptr) {
+logging::LOG_ERROR(logger_) << "Failed create new file BIO, " <<
getLatestOpenSSLErrorString();
+return false;
+ }
+ if (BIO_read_filename(fp.get(), certificate_.c_str()) <= 0) {
+logging::LOG_ERROR(logger_) << "Failed to read certificate file " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+ const auto p12_deleter = [](PKCS12* ptr) { PKCS12_free(ptr); };
+ std::unique_ptr p12(d2i_PKCS12_bio(fp.get(),
nullptr), p12_deleter);
+ if (p12 == nullptr) {
+logging::LOG_ERROR(logger_) << "Failed to DER decode certificate file " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+
+ EVP_PKEY* pkey = nullptr;
+ X509* cert = nullptr;
+ STACK_OF(X509)* ca = nullptr;
+ if (!PKCS12_parse(p12.get(), passphrase_.c_str(), &pkey, &cert, &ca)) {
+logging::LOG_ERROR(logger_) << "Failed to parse certificate file " <<
certificate_ << " as PKCS#12, " << getLatestOpenSSLErrorString();
+return false;
+ }
+ utils::tls::EVP_PKEY_unique_ptr pkey_ptr{pkey};
+ utils::tls::X509_unique_ptr cert_ptr{cert};
+ const auto ca_deleter = gsl::finally([ca] { sk_X509_pop_free(ca, X509_free);
});
+
+ if (SSL_CTX_use_certificate(ctx, cert) != 1) {
+logging::LOG_ERROR(logger_) << "Failed to set certificate from " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+ while (ca != nullptr && sk_X509_num(ca) > 0) {
+utils::tls::X509_unique_ptr cacert{sk_X509_pop(ca)};
+if (SSL_CTX_add_extra_chain_cert(ctx, cacert.get()) != 1) {
+ logging::LOG_ERROR(logger_) << "Failed to set additional certificate
from " << certificate_ << ", " << getLatestOpenSSLErrorString();
+ return false;
+}
+cacert.release(); // a successful SSL_CTX_add_extra_chain_cert() takes
ownership of cacert
+ }
+ if (SSL_CTX_use_PrivateKey(ctx, pkey) != 1) {
+logging::LOG_ERROR(logger_) << "Failed to set private key from " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
return false;
}
return true;
}
-#endif
+
+bool SSLContextService::addPemCertificateToSSLContext(SSL_CTX* ctx) const {
+ if (SSL_CTX_use_certificate_chain_file(ctx, certificate_.c_str()) <= 0) {
+logging::LOG_ERROR(logger_) << "Could not load client certificate " <<
certificate_ << ", " << getLatestOpenSSLErrorString();
+return false;
+ }
+
+ if (!IsNullOrEmpty(passphrase_)) {
+void* passphrase = const_cast(&passphrase_);
+SSL_CTX_set_default_passwd_cb_userdata(ctx, passphrase);
+SSL_CTX_set_default_passwd_cb(ctx, minifi::utils::tls::pemPassWordCb);
+ }
+
+ if (!IsNullOrEmpty(private_key_)) {
+int retp = SSL_CTX_use_PrivateKey_file(ctx, private_key_.c_str(),
SSL_FILETYPE_PEM);
+if (retp != 1) {
+ logging::LOG_ERROR(logger_) << "Could not load private key, " << retp <<
" on " << private_key_ << ", " << getLatestOpenSSLErrorString();
+ return false;
+}
+ }
+
+ return true;
+}
+
+bool
SSLContextService::addClientCertificateFromSystemStoreToSSLContext(SSL_CTX*
ctx) const {
+#ifdef WIN32
+ utils::tls::WindowsCertStoreLocation store_location{cert_store_location_};
+ HCERTSTORE hCertStore = CertOpenStore(CERT_STORE_PROV_SYSTEM_A, 0, NULL,
+CERT_STORE_OPEN_EXISTING_FLAG |
CERT_STORE_READONLY_FLAG | store_location.getBitfieldValue(),
+client_cert_store_.data());
+ if (!hCertStore) {
+logger_->log_error("Could not open system certificate store %s/%s (client
certificates)", cert_store_location_, cl
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
fgerlits commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538629206
##
File path: libminifi/src/utils/tls/ExtendedKeyUsage.cpp
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifdef OPENSSL_SUPPORT
+
+#include "utils/tls/ExtendedKeyUsage.h"
+
+#include
+
+#include
+#include
+#include
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace tls {
+
+namespace {
+
+struct KeyValuePair {
+ const char* key;
+ uint8_t value;
+};
+constexpr std::array EXT_KEY_USAGE_NAME_TO_BIT_POS{{
+KeyValuePair{"Server Authentication", 1},
+KeyValuePair{"Client Authentication", 2},
+KeyValuePair{"Code Signing", 3},
+KeyValuePair{"Secure Email", 4},
+KeyValuePair{"Time Stamping", 8},
+KeyValuePair{"OCSP Signing", 9}
+}};
+
+} // namespace
+
+void EXTENDED_KEY_USAGE_deleter::operator()(EXTENDED_KEY_USAGE* key_usage)
const { EXTENDED_KEY_USAGE_free(key_usage); }
+
+ExtendedKeyUsage::ExtendedKeyUsage() :
logger_(core::logging::LoggerFactory::getLogger()) {}
+
+ExtendedKeyUsage::ExtendedKeyUsage(const EXTENDED_KEY_USAGE& key_usage_asn1) :
ExtendedKeyUsage{} {
+ const int num_oids = sk_ASN1_OBJECT_num(&key_usage_asn1);
+ for (int i = 0; i < num_oids; ++i) {
+const ASN1_OBJECT* const oid = sk_ASN1_OBJECT_value(&key_usage_asn1, i);
+assert(oid && oid->length > 0);
+const unsigned char bit_pos = oid->data[oid->length - 1];
Review comment:
I have made some code improvements: rename variable, add comment, use
`std::bitset`, which make this code less incomprehensible, I hope.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
fgerlits commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538627427
##
File path: libminifi/src/utils/tls/ExtendedKeyUsage.cpp
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifdef OPENSSL_SUPPORT
+
+#include "utils/tls/ExtendedKeyUsage.h"
+
+#include
+
+#include
+#include
+#include
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace tls {
+
+namespace {
+
+struct KeyValuePair {
+ const char* key;
+ uint8_t value;
+};
+constexpr std::array EXT_KEY_USAGE_NAME_TO_BIT_POS{{
+KeyValuePair{"Server Authentication", 1},
+KeyValuePair{"Client Authentication", 2},
+KeyValuePair{"Code Signing", 3},
+KeyValuePair{"Secure Email", 4},
+KeyValuePair{"Time Stamping", 8},
+KeyValuePair{"OCSP Signing", 9}
+}};
Review comment:
I have added a comment with a link to the RFC
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on pull request #953: MINIFICPP-1421 - Disable C2JstackTest
hunyadi-dev commented on pull request #953:
URL: https://github.com/apache/nifi-minifi-cpp/pull/953#issuecomment-740697359
@arpadboda
This is the inheritance hierarchy:
```c++
class VerifyC2DescribeJstack : public VerifyC2Describe {
class VerifyC2Describe : public VerifyC2Base {
class VerifyC2Base : public HTTPIntegrationBase {
class HTTPIntegrationBase : public IntegrationBase {
https://github.com/apache/nifi-minifi-cpp/blob/main/libminifi/test/integration/IntegrationBase.h#L82
https://github.com/apache/nifi-minifi-cpp/blob/main/libminifi/include/FlowController.h#L333-L339
```
Meaning that `runAssertions()` can only be run if the scheduler was already
initialized and as far as I understand, it is the initialization of the logger
that produces the `logger got sinks from ...` message meaning that the
assertion is always true (and will never wait).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (MINIFICPP-1422) MiNiFi should be able to get certs from the Openssl truststore on Linux
Ferenc Gerlits created MINIFICPP-1422:
-
Summary: MiNiFi should be able to get certs from the Openssl
truststore on Linux
Key: MINIFICPP-1422
URL: https://issues.apache.org/jira/browse/MINIFICPP-1422
Project: Apache NiFi MiNiFi C++
Issue Type: New Feature
Reporter: Ferenc Gerlits
Minifi is able to read the server and client certificates necessary to connect
to the C2 server from the Windows truststore (MINIFICPP-1401), but this does
not work on Linux.
On Linux, the natural way would be to use Openssl's own truststore.
The server certificate works, to some degree: if {{server-cert.pem}} is the
server certificate, then you can install it like this:
{noformat}
$ cd ${OPENSSL_CACERT_DIR}
$ cp /path/to/server-cert.pem ./
$ CERTIFICATE_HASH=`openssl x509 -noout -hash -in server-cert.pem`
$ ln -s server-cert.pem ${CERTIFICATE_HASH}.0
$ chmod 755 ${OPENSSL_CACERT_DIR}
$ chmod 600 ${OPENSSL_CACERT_DIR}/server-cert.pem{noformat}
After this, if you unset {{nifi.security.client.ca.certificate}} and set
{{nifi.security.use.system.cert.store=true}}, then Minifi will read the server
certificate from {{OPENSSL_CACERT_DIR}}.
But the default {{OPENSSL_CACERT_DIR}} depends on where Minifi was compiled,
eg. it could be
{{/home/myuser/src/minifi/build/thirdparty/libressl-install/etc/ssl/certs}},
which is not nice. The default location should be changed to something more
sensible, and there needs to be a way to override it.
I don't know how to add the client certificate + key to the Openssl truststore,
so that will need to be investigated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (NIFI-7896) Add support for GET request in ListenHTTP
[ https://issues.apache.org/jira/browse/NIFI-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Gyori closed NIFI-7896. - > Add support for GET request in ListenHTTP > - > > Key: NIFI-7896 > URL: https://issues.apache.org/jira/browse/NIFI-7896 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently the ListenHTTP processor does not accept GET requests. > Enhance the ListenHTTP processor to support GET requests that are sent to > /holds/ids. > The response for GET requests should be "200 OK", and the response body > should contain the IDs of uncommitted holds (in JSON format), or empty JSON > list if there are no uncommitted holds. > If SSL is used and the client certificate's distinguished name does not match > the "Authorized DN Pattern" set on the processor, "200 OK" with an empty list > should be returned. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7896) Add support for GET request in ListenHTTP
[ https://issues.apache.org/jira/browse/NIFI-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Gyori resolved NIFI-7896. --- Resolution: Implemented > Add support for GET request in ListenHTTP > - > > Key: NIFI-7896 > URL: https://issues.apache.org/jira/browse/NIFI-7896 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently the ListenHTTP processor does not accept GET requests. > Enhance the ListenHTTP processor to support GET requests that are sent to > /holds/ids. > The response for GET requests should be "200 OK", and the response body > should contain the IDs of uncommitted holds (in JSON format), or empty JSON > list if there are no uncommitted holds. > If SSL is used and the client certificate's distinguished name does not match > the "Authorized DN Pattern" set on the processor, "200 OK" with an empty list > should be returned. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7896) Add support for GET request in ListenHTTP
[ https://issues.apache.org/jira/browse/NIFI-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Gyori updated NIFI-7896: -- Status: Open (was: Patch Available) Support for GET has been handled in https://issues.apache.org/jira/browse/NIFI-7922 . > Add support for GET request in ListenHTTP > - > > Key: NIFI-7896 > URL: https://issues.apache.org/jira/browse/NIFI-7896 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently the ListenHTTP processor does not accept GET requests. > Enhance the ListenHTTP processor to support GET requests that are sent to > /holds/ids. > The response for GET requests should be "200 OK", and the response body > should contain the IDs of uncommitted holds (in JSON format), or empty JSON > list if there are no uncommitted holds. > If SSL is used and the client certificate's distinguished name does not match > the "Authorized DN Pattern" set on the processor, "200 OK" with an empty list > should be returned. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] pgyori commented on pull request #4581: NIFI-7896: Added support for GET requests in ListenHTTP
pgyori commented on pull request #4581: URL: https://github.com/apache/nifi/pull/4581#issuecomment-740668143 Support for GET has been implemented in https://github.com/apache/nifi/pull/4603 Closing this pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pgyori closed pull request #4581: NIFI-7896: Added support for GET requests in ListenHTTP
pgyori closed pull request #4581: URL: https://github.com/apache/nifi/pull/4581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on pull request #924: MINIFICPP-1389 - Upgrade librdkafka to version 1.5.0
hunyadi-dev commented on pull request #924: URL: https://github.com/apache/nifi-minifi-cpp/pull/924#issuecomment-740658317 @fgerlits @szaszm updated the transitive licenses. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #947: MINIFICPP-1401 Read certificates from the Windows system store
szaszm commented on a change in pull request #947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/947#discussion_r538274424
##
File path: libminifi/src/utils/tls/ExtendedKeyUsage.cpp
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifdef OPENSSL_SUPPORT
+
+#include "utils/tls/ExtendedKeyUsage.h"
+
+#include
+
+#include
+#include
+#include
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace tls {
+
+namespace {
+
+struct KeyValuePair {
+ const char* key;
+ uint8_t value;
+};
+constexpr std::array EXT_KEY_USAGE_NAME_TO_BIT_POS{{
+KeyValuePair{"Server Authentication", 1},
+KeyValuePair{"Client Authentication", 2},
+KeyValuePair{"Code Signing", 3},
+KeyValuePair{"Secure Email", 4},
+KeyValuePair{"Time Stamping", 8},
+KeyValuePair{"OCSP Signing", 9}
+}};
Review comment:
A source for these constants would be nice. I've found them in RFC3280
on page 109.
##
File path: libminifi/src/utils/tls/ExtendedKeyUsage.cpp
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifdef OPENSSL_SUPPORT
+
+#include "utils/tls/ExtendedKeyUsage.h"
+
+#include
+
+#include
+#include
+#include
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace tls {
+
+namespace {
+
+struct KeyValuePair {
+ const char* key;
+ uint8_t value;
+};
+constexpr std::array EXT_KEY_USAGE_NAME_TO_BIT_POS{{
+KeyValuePair{"Server Authentication", 1},
+KeyValuePair{"Client Authentication", 2},
+KeyValuePair{"Code Signing", 3},
+KeyValuePair{"Secure Email", 4},
+KeyValuePair{"Time Stamping", 8},
+KeyValuePair{"OCSP Signing", 9}
+}};
+
+} // namespace
+
+void EXTENDED_KEY_USAGE_deleter::operator()(EXTENDED_KEY_USAGE* key_usage)
const { EXTENDED_KEY_USAGE_free(key_usage); }
+
+ExtendedKeyUsage::ExtendedKeyUsage() :
logger_(core::logging::LoggerFactory::getLogger()) {}
+
+ExtendedKeyUsage::ExtendedKeyUsage(const EXTENDED_KEY_USAGE& key_usage_asn1) :
ExtendedKeyUsage{} {
+ const int num_oids = sk_ASN1_OBJECT_num(&key_usage_asn1);
+ for (int i = 0; i < num_oids; ++i) {
+const ASN1_OBJECT* const oid = sk_ASN1_OBJECT_value(&key_usage_asn1, i);
+assert(oid && oid->length > 0);
+const unsigned char bit_pos = oid->data[oid->length - 1];
Review comment:
Could you provide some structure references for mortals like myself to
be able to see what makes e.g. `oid->data[oid->length - 1]` be the bit to be
set in the extended key usage bitset?
##
File path: libminifi/src/controllers/SSLContextService.cpp
##
@@ -128,16 +149,239 @@ bool SSLContextService::configure_ssl_context(SSL_CTX
*ctx) {
}
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, nullptr);
- int retp = SSL_CTX_load_verify_locations(ctx, ca_certificate_.c_str(), 0);
- if (retp == 0) {
-logging::LOG_ERROR(logger_) << "Can not load CA certificate, Exiting, " <<
getLatestOpenSSLErrorString();
+ if (!IsNullOrEmpty(ca_certificate_)) {
+if (SSL_CTX_load_verify_locations(ctx, ca_certificate_.c_str(), 0) == 0) {
+ logging::LOG_ERROR(logger_) << "Cannot load CA certificate, exiting, "
<< getLatestOpenSSLErrorString();
+ return false;
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
exceptionfactory commented on a change in pull request #4715:
URL: https://github.com/apache/nifi/pull/4715#discussion_r538391803
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -64,8 +68,8 @@ class ConfigEncryptionTool {
public String outputLoginIdentityProvidersPath
public String authorizersPath
public String outputAuthorizersPath
-public String flowXmlPath
-public String outputFlowXmlPath
+public static String flowXmlPath
+public static String outputFlowXmlPath
Review comment:
Is there a reason for making these static?
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -93,6 +97,7 @@ class ConfigEncryptionTool {
private boolean handlingFlowXml = false
private boolean ignorePropertiesFiles = false
private boolean translatingCli = false
+private boolean isflowXmlGZipped = false
Review comment:
Instead of setting and track this variable, is it possible to check the
class type of the InputStream returned from loadFlowXml? That would avoid
introducing another state variable into the class.
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
##
@@ -3806,19 +3812,23 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
// Verify the flow definition
def verifyTool = new ConfigEncryptionTool()
verifyTool.isVerbose = true
-verifyTool.flowXmlPath = workingFlowXmlFile.path
-String updatedFlowXmlContent = verifyTool.loadFlowXml()
+InputStream updatedFlowXmlContent =
verifyTool.loadFlowXml(workingFlowXmlFile.path)
// Check that the flow.xml.gz content changed
assert updatedFlowXmlContent != ORIGINAL_FLOW_XML_CONTENT
// Verify that the cipher texts decrypt correctly
logger.info("Original flow.xml.gz cipher texts:
${originalFlowCipherTexts}")
-def flowCipherTexts = updatedFlowXmlContent.findAll(WFXCTR)
-logger.info("Updated flow.xml.gz cipher texts:
${flowCipherTexts}")
-assert flowCipherTexts.size() == CIPHER_TEXT_COUNT
-flowCipherTexts.every {
-assert ConfigEncryptionTool.decryptFlowElement(it,
newFlowPassword) == "thisIsABadPassword"
+
+// TODO: Nathan needs to change this findAll to search the
updated flow xml file and stream search it.
+def updatedFlowCipherTexts =
findFieldsInStream(updatedFlowXmlContent, WFXCTR)
+//def flowCipherTexts = updatedFlowXmlContent.findAll(WFXCTR)
Review comment:
Recommend removing this comment since the implementation has changed
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
##
@@ -3911,19 +3921,20 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
// Verify the flow definition
def verifyTool = new ConfigEncryptionTool()
verifyTool.isVerbose = true
-verifyTool.flowXmlPath = workingFlowXmlFile.path
-String updatedFlowXmlContent = verifyTool.loadFlowXml()
+InputStream migratedFlowXmlContent =
verifyTool.loadFlowXml(workingFlowXmlFile.path)
// Check that the flow.xml.gz cipher texts did change (new
salt)
-assert updatedFlowXmlContent != ORIGINAL_FLOW_XML_CONTENT
+assert migratedFlowXmlContent != ORIGINAL_FLOW_XML_CONTENT
// Verify that the cipher texts decrypt correctly
logger.info("Original flow.xml.gz cipher texts:
${originalFlowCipherTexts}")
-def flowCipherTexts = updatedFlowXmlContent.findAll(WFXCTR)
-logger.info("Updated flow.xml.gz cipher texts:
${flowCipherTexts}")
-assert flowCipherTexts.size() == CIPHER_TEXT_COUNT
-flowCipherTexts.every {
-assert ConfigEncryptionTool.decryptFlowElement(it,
newFlowPassword) == "thisIsABadPassword"
+def migratedFlowCipherTexts =
findFieldsInStream(migratedFlowXmlContent, WFXCTR)
+logger.info("Updated flow.xml.gz cipher texts:
${migratedFlowCipherTexts}")
+assert migratedFlowCipherTexts.size() == CIPHER_TEXT_COUNT
+migratedFlowCipherTexts.each {
+String decryptedValue =
ConfigEncryptionTool.decryptFlowElement(it, newFlowPassword)
+logger.info("Decrypted value of migrated " +
workingFlowXmlFile.path + " was: " + decryptedValue)
Review comment:
Recommend changing this log to u
[GitHub] [nifi] markap14 commented on pull request #4714: NIFI-7801 Adding support for HTTP based Splunk put and indexed acknowledgement
markap14 commented on pull request #4714: URL: https://github.com/apache/nifi/pull/4714#issuecomment-740642509 Will Review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on pull request #837: MINIFICPP-1121 - Upgrade spdlog to version 1.8.0
szaszm commented on pull request #837: URL: https://github.com/apache/nifi-minifi-cpp/pull/837#issuecomment-740638136 There are some compilation errors caused by the last change This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-1419) Stabilize docker tests in Github Actions
[ https://issues.apache.org/jira/browse/MINIFICPP-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi reassigned MINIFICPP-1419: Assignee: Gabor Gyimesi > Stabilize docker tests in Github Actions > > > Key: MINIFICPP-1419 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1419 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > test_zero_file.py tests sometimes time out and fail in the CI environment, so > it should be stabilized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1419) Stabilize docker tests in Github Actions
[ https://issues.apache.org/jira/browse/MINIFICPP-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi resolved MINIFICPP-1419. -- Resolution: Fixed > Stabilize docker tests in Github Actions > > > Key: MINIFICPP-1419 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1419 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > test_zero_file.py tests sometimes time out and fail in the CI environment, so > it should be stabilized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1121) Upgrade spdlog
[
https://issues.apache.org/jira/browse/MINIFICPP-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi resolved MINIFICPP-1121.
-
Fix Version/s: (was: 1.0.0)
0.9.0
Resolution: Fixed
> Upgrade spdlog
> --
>
> Key: MINIFICPP-1121
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1121
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Marton Szasz
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Fix For: 0.9.0
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Or version of spdlog is 2+ years old. The new spdlog version uses a new
> version of the cppformat (back then)/fmt (now) formatting library.
> We should consider directly depending on {{fmt}} since we already have it as
> a transitive dependency and it would be useful for e.g. formatting
> exception/error messages, etc.
>
> *Update (hunyadi):*
> Seems like we have to skip version 1.0 with upgrading. There are quite a lot
> of non-documented breaking changes, for example this commit:
>
> [https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
> breaks ::instance() on stdout/stderr sinks. Unfortunately, changes like this
> in spdlog are not documented and the codebase is kept up-to-date with commits
> sent directly to the central repository.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1344) Investigate and fix FlowConfiguration payload URLs
[ https://issues.apache.org/jira/browse/MINIFICPP-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Hunyadi resolved MINIFICPP-1344. - Resolution: Fixed > Investigate and fix FlowConfiguration payload URLs > -- > > Key: MINIFICPP-1344 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1344 > Project: Apache NiFi MiNiFi C++ > Issue Type: Task >Affects Versions: 0.7.0 >Reporter: Adam Hunyadi >Assignee: Adam Hunyadi >Priority: Minor > Labels: MiNiFi-CPP-Hygiene > Fix For: 0.8.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > *Background:* > [[This line from > FlowConfiguration::updateFromPayload]|https://github.com/apache/nifi-minifi-cpp/blob/main/libminifi/src/core/FlowConfiguration.cpp#L92] > does not match the [[NiFi Registry REST API > docs]|http://nifi.apache.org/docs/nifi-registry-docs/rest-api/index.html]. > Upon checking how this function was invoked in test, I found that an example > invocation has this string passed as target url: > "https://localhost:57454/update";. > *Proposal:* > This functionality might be broken and unused. It might need some > investigation to check how this function is expected to be invoked. Until > then, fixing the REST endpoint seems like the correct and safe thing to do. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (MINIFICPP-1121) Upgrade spdlog
[
https://issues.apache.org/jira/browse/MINIFICPP-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1121:
Comment: was deleted
(was: (flag) Flag added
Waiting for review)
> Upgrade spdlog
> --
>
> Key: MINIFICPP-1121
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1121
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Marton Szasz
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Fix For: 1.0.0
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Or version of spdlog is 2+ years old. The new spdlog version uses a new
> version of the cppformat (back then)/fmt (now) formatting library.
> We should consider directly depending on {{fmt}} since we already have it as
> a transitive dependency and it would be useful for e.g. formatting
> exception/error messages, etc.
>
> *Update (hunyadi):*
> Seems like we have to skip version 1.0 with upgrading. There are quite a lot
> of non-documented breaking changes, for example this commit:
>
> [https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
> breaks ::instance() on stdout/stderr sinks. Unfortunately, changes like this
> in spdlog are not documented and the codebase is kept up-to-date with commits
> sent directly to the central repository.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1405) Fix improperly defined warning flags for civetweb
[
https://issues.apache.org/jira/browse/MINIFICPP-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi resolved MINIFICPP-1405.
-
Fix Version/s: 0.9.0
Resolution: Fixed
> Fix improperly defined warning flags for civetweb
> -
>
> Key: MINIFICPP-1405
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1405
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: $ uname -a
> Linux laptop 4.15.0-123-generic #126~16.04.1-Ubuntu SMP Wed Oct 21 13:51:51
> UTC 2020 i686 i686 i686 GNU/Linux
> $ clang-11 --version
> Ubuntu clang version
> 11.0.0-++20200721055954+cebd637c886-1~exp1~20200721161335.13
> Target: i686-pc-linux-gnu
> Thread model: posix
> $ cmake --version
> cmake version 3.19.0-rc2
> $ ninja --version
> 1.10.0.git.kitware.jobserver-1
>Reporter: Ivan Serdyuk
>Assignee: Adam Hunyadi
>Priority: Major
> Labels: Linux, MiNiFi-CPP-Hygiene, Ubuntu_16.04, linux, ubuntu
> Fix For: 0.9.0
>
> Attachments: build.ninja
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> *Background:*
> Building civetweb is unstable on certain environment (our CI jobs show errors
> on builds on Ubuntu builds only).
> {code:bash|title=Example build failure with Ninja and clang-11}
> [1/445] Performing build step for 'civetweb-external'
> FAILED:
> civetweb-external-prefix/src/civetweb-external-stamp/civetweb-external-build
> thirdparty/civetweb-install/lib/libcivetweb.a
> thirdparty/civetweb-install/lib/libcivetweb-cpp.a
> cd
> /home/oceanfish81/Desktop/nifi-minifi-cpp/build/civetweb-external-prefix/src/civetweb-external-build
> && /usr/local/bin/cmake --build . && /usr/local/bin/cmake -E touch
> /home/oceanfish81/Desktop/nifi-minifi-cpp/build/civetweb-external-prefix/src/civetweb-external-stamp/civetweb-external-build
> [1/6] Building CXX object src/CMakeFiles/civetweb-cpp.dir/CivetServer.cpp.o
> FAILED: src/CMakeFiles/civetweb-cpp.dir/CivetServer.cpp.o
> /usr/bin/clang++-11 -DNO_SSL_DL -DUSE_STACK_SIZE=102400
> -I/home/oceanfish81/Desktop/nifi-minifi-cpp/build/thirdparty/libressl-install/include
>
> -I/home/oceanfish81/Desktop/nifi-minifi-cpp/build/thirdparty/civetweb-src/include
> -std=c++11 -std=c++14 -Wall -Wextra -Wshadow -Wmissing-prototypes
> -Weverything /W4 -Wno-padded /Wd4820 -Wno-unused-macros
> -Wno-format-nonliteral /WX -pedantic-errors -fvisibility=hidden -Os -DNDEBUG
> -fPIC -MD -MT src/CMakeFiles/civetweb-cpp.dir/CivetServer.cpp.o -MF
> src/CMakeFiles/civetweb-cpp.dir/CivetServer.cpp.o.d -o
> src/CMakeFiles/civetweb-cpp.dir/CivetServer.cpp.o -c
> /home/oceanfish81/Desktop/nifi-minifi-cpp/build/thirdparty/civetweb-src/src/CivetServer.cpp
> clang: error: no such file or directory: '/W4'
> clang: error: no such file or directory: '/Wd4820'
> clang: error: no such file or directory: '/WX'
> [2/6] Building C object src/CMakeFiles/c-executable.dir/main.c.o
> FAILED: src/CMakeFiles/c-executable.dir/main.c.o
> /usr/bin/clang-11 -DNO_SSL_DL -DUSE_STACK_SIZE=102400
> -I/home/oceanfish81/Desktop/nifi-minifi-cpp/build/thirdparty/libressl-install/include
>
> -I/home/oceanfish81/Desktop/nifi-minifi-cpp/build/thirdparty/civetweb-src/include
> -std=c11 -Wall -Wextra -Wshadow -Wconversion -Wmissing-prototypes
> -Weverything -Wparentheses /W4 -Wno-padded -Wno-unused-macros
> -Wno-reserved-id-macros -Wno-format-nonliteral -Wno-date-time -Wno-cast-qual
> /Wd4820 -pedantic-errors -fvisibility=hidden -Os -DNDEBUG -fPIE -MD -MT
> src/CMakeFiles/c-executable.dir/main.c.o -MF
> src/CMakeFiles/c-executable.dir/main.c.o.d -o
> src/CMakeFiles/c-executable.dir/main.c.o -c
> /home/oceanfish81/Desktop/nifi-minifi-cpp/build/thirdparty/civetweb-src/src/main.c
> clang: error: no such file or directory: '/W4'
> clang: error: no such file or directory: '/Wd4820'
> ninja: build stopped: subcommand failed.
> ninja: build stopped: subcommand failed.
> {code}
> *Proposal:*
> Update civetweb version, see if the problem persists. If it still does, patch
> out all MSVC warning flags from the civetweb cmake file.
> *Update:*
> Seems like civetweb had an issue with checking available compiler flags. I
> opened a PR to fix it in their repo, and backported the change to our version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1421) Investigate and fix C2JstackTest
[
https://issues.apache.org/jira/browse/MINIFICPP-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1421:
Issue Type: Bug (was: Improvement)
> Investigate and fix C2JstackTest
>
>
> Key: MINIFICPP-1421
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1421
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Fix For: 0.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Background:*
> C2JstackTest is currently looks for a log line that contains the word
> "SchedulingAgent". This line however is only present due to the
> LogTestController logs that log lines from the SchedulingAgents will not be
> shown.
> {code:bash|title=Filtered test output}
> ➜ ./extensions/http-curl/tests/C2JstackTest
> ../src/libminifi/test/resources/TestHTTPGet.yml
> ../src/libminifi/test/resources/ |& grep "SchedulingAgent"
>
> [2020-12-08 10:48:51.955]
> [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
> org::apache::nifi::minifi::SchedulingAgent logger got sinks from namespace
> root and level error from namespace root
> [2020-12-08 10:48:51.955]
> [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
> org::apache::nifi::minifi::ThreadedSchedulingAgent logger got sinks from
> namespace root and level error from namespace root
> [2020-12-08 10:48:51.955]
> [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
> org::apache::nifi::minifi::TimerDrivenSchedulingAgent logger got sinks from
> namespace root and level error from namespace root
> {code}
> Enabling all sinks still did not produce any relevant log line that would be
> worth matching against:
> {code:c++}
> #include
> "../../extensions/standard-processors/controllers/UnorderedMapKeyValueStoreService.h"
> #include
> "../../extensions/standard-processors/controllers/UnorderedMapPersistableKeyValueStoreService.h"
> #include "../../extensions/http-curl/processors/InvokeHTTP.h"
> #include "../../extensions/http-curl/processors/InvokeHTTP.h"
> #include "../../extensions/standard-processors/processors/LogAttribute.h"
> #include "../../extensions/standard-processors/processors/LogAttribute.h"
> #include "../../libminifi/include/c2/ControllerSocketProtocol.h"
> #include "../../libminifi/include/c2/ControllerSocketProtocol.h"
> #include "../../extensions/standard-processors/processors/AppendHostInfo.h"
> #include "../../extensions/standard-processors/processors/AppendHostInfo.h"
> #include "../../extensions/standard-processors/processors/ExecuteProcess.h"
> #include "../../extensions/standard-processors/processors/ExecuteProcess.h"
> #include "../../extensions/standard-processors/processors/ExtractText.h"
> #include "../../extensions/standard-processors/processors/ExtractText.h"
> #include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
> #include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
> #include "../../extensions/standard-processors/processors/GetFile.h"
> #include "../../extensions/standard-processors/processors/GetFile.h"
> #include "../../extensions/standard-processors/processors/GetTCP.h"
> #include "../../extensions/standard-processors/processors/GetTCP.h"
> #include "../../extensions/standard-processors/processors/HashContent.h"
> #include "../../extensions/standard-processors/processors/HashContent.h"
> #include "../../extensions/standard-processors/processors/ListenSyslog.h"
> #include "../../extensions/standard-processors/processors/ListenSyslog.h"
> #include "../../extensions/standard-processors/processors/PutFile.h"
> #include "../../extensions/standard-processors/processors/PutFile.h"
> #include "../../extensions/standard-processors/processors/RetryFlowFile.h"
> #include "../../extensions/standard-processors/processors/RetryFlowFile.h"
> #include "../../extensions/standard-processors/processors/RouteOnAttribute.h"
> #include "../../extensions/standard-processors/processors/RouteOnAttribute.h"
> #include "../../extensions/standard-processors/processors/TailFile.h"
> #include "../../extensions/standard-processors/processors/TailFile.h"
> #include "../../extensions/standard-processors/processors/UpdateAttribute.h"
> #include "../../extensions/standard-processors/processors/UpdateAttribute.h"
> #include "../../extensions/http-curl/protocols/AgentPrinter.h"
> #include "../../extensions/http-curl/protocols/AgentPrinter.h"
> #include "../../extensions/http-curl/protocols/RESTReceiver.h"
> #include "../../extensions/http-curl/protocols/RESTReceiver.h"
> #include "../../extensions/cive
[jira] [Resolved] (MINIFICPP-1420) Add alternate mirrors for libsodium
[
https://issues.apache.org/jira/browse/MINIFICPP-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi resolved MINIFICPP-1420.
-
Resolution: Fixed
> Add alternate mirrors for libsodium
> ---
>
> Key: MINIFICPP-1420
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1420
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.8.0
>Reporter: Adam Hunyadi
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Fix For: 0.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Backgroud:*
> The official libsodium downloads page is unstable, causing clean builds to
> fail with the following error:
> {code:bash|title=Libsodium server not available}
> CMake Error at
> /home/hunyadix/Documents/Projects/nifi-minifi-cpp_1/build/libsodium-external-prefix/src/libsodium-external-stamp/download-libsodium-external.cmake:170
> (message):
> Each download failed!
>
>
>
>
>
>
>
> error: downloading
> 'https://download.libsodium.org/libsodium/releases/libsodium-1.0.18.tar.gz'
> failed
>
> status_code: 6
>
>
>
> status_string: "Couldn't resolve host name"
>
>
>
> log:
>
>
>
> --- LOG BEGIN ---
>
>
>
> getaddrinfo(3) failed for download.libsodium.org:443
>
>
>
>
>
>
>
> Couldn't resolve host 'download.libsodium.org'
>
>
>
>
>
>
>
> Closing connection 0
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --- LOG END ---
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #949: MINIFICPP-1419 Stabilize docker tests in Github Actions
arpadboda closed pull request #949: URL: https://github.com/apache/nifi-minifi-cpp/pull/949 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #940: MINIFICPP-1373 - Implement ConsumeKafka
hunyadi-dev commented on a change in pull request #940:
URL: https://github.com/apache/nifi-minifi-cpp/pull/940#discussion_r538358865
##
File path: libminifi/test/TestBase.h
##
@@ -200,7 +199,7 @@ class LogTestController {
}
my_properties_->set("logger.root", "ERROR,ostream");
my_properties_->set("logger." + core::getClassName(),
"INFO");
-my_properties_->set("logger." +
core::getClassName(), "DEBUG");
+my_properties_->set("logger." +
core::getClassName(), "INFO");
Review comment:
This brakes C2JstackTest. Once #953 is merged, rebasing on it should
fix this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #940: MINIFICPP-1373 - Implement ConsumeKafka
hunyadi-dev commented on a change in pull request #940:
URL: https://github.com/apache/nifi-minifi-cpp/pull/940#discussion_r538356594
##
File path: extensions/librdkafka/ConsumeKafka.h
##
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "core/Processor.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "rdkafka.h"
+#include "rdkafka_utils.h"
+#include "KafkaConnection.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+
+// The upper limit for Max Poll Time is 4 seconds. This is because Watchdog
would potentially start
+// reporting issues with the processor health otherwise
+class ConsumeKafkaMaxPollTimeValidator : public TimePeriodValidator {
+ public:
+ ConsumeKafkaMaxPollTimeValidator(const std::string &name) // NOLINT
+ : TimePeriodValidator(name) {
+ }
+ ~ConsumeKafkaMaxPollTimeValidator() override = default;
+
+ ValidationResult validate(const std::string& subject, const std::string&
input) const override {
+uint64_t value;
+TimeUnit timeUnit;
+uint64_t value_as_ms;
+return
ValidationResult::Builder::createBuilder().withSubject(subject).withInput(input).isValid(
+core::TimePeriodValue::StringToTime(input, value, timeUnit) &&
+org::apache::nifi::minifi::core::Property::ConvertTimeUnitToMS(value,
timeUnit, value_as_ms) &&
+0 < value_as_ms && value_as_ms <= 4000).build();
+ }
+};
+
+} // namespace core
+namespace processors {
+
+class ConsumeKafka : public core::Processor {
+ public:
+ static constexpr char const* ProcessorName = "ConsumeKafka";
+
+ // Supported Properties
+ static core::Property KafkaBrokers;
+ static core::Property SecurityProtocol;
+ static core::Property TopicNames;
+ static core::Property TopicNameFormat;
+ static core::Property HonorTransactions;
+ static core::Property GroupID;
+ static core::Property OffsetReset;
+ static core::Property KeyAttributeEncoding;
+ static core::Property MessageDemarcator;
+ static core::Property MessageHeaderEncoding;
+ static core::Property HeadersToAddAsAttributes;
+ static core::Property DuplicateHeaderHandling;
+ static core::Property MaxPollRecords;
+ static core::Property MaxPollTime;
+ static core::Property SessionTimeout;
+
+ // Supported Relationships
+ static const core::Relationship Success;
+
+ // Security Protocol allowable values
+ static constexpr char const* SECURITY_PROTOCOL_PLAINTEXT = "PLAINTEXT";
+ static constexpr char const* SECURITY_PROTOCOL_SSL = "SSL";
+ static constexpr char const* SECURITY_PROTOCOL_SASL_PLAINTEXT =
"SASL_PLAINTEXT";
+ static constexpr char const* SECURITY_PROTOCOL_SASL_SSL = "SASL_SSL";
+
+ // Topic Name Format allowable values
+ static constexpr char const* TOPIC_FORMAT_NAMES = "Names";
+ static constexpr char const* TOPIC_FORMAT_PATTERNS = "Patterns";
+
+ // Offset Reset allowable values
+ static constexpr char const* OFFSET_RESET_EARLIEST = "earliest";
+ static constexpr char const* OFFSET_RESET_LATEST = "latest";
+ static constexpr char const* OFFSET_RESET_NONE = "none";
+
+ // Key Attribute Encoding allowable values
+ static constexpr char const* KEY_ATTR_ENCODING_UTF_8 = "UTF-8";
+ static constexpr char const* KEY_ATTR_ENCODING_HEX = "Hex";
+
+ // Message Header Encoding allowable values
+ static constexpr char const* MSG_HEADER_ENCODING_UTF_8 = "UTF-8";
+ static constexpr char const* MSG_HEADER_ENCODING_HEX = "Hex";
+
+ // Duplicate Header Handling allowable values
+ static constexpr char const* MSG_HEADER_KEEP_FIRST = "Keep First";
+ static constexpr char const* MSG_HEADER_KEEP_LATEST = "Keep Latest";
+ static constexpr char const* MSG_HEADER_COMMA_SEPARATED_MERGE =
"Comma-separated Merge";
+
+ // Flowfile attributes written
+ static constexpr char const* KAFKA_COUNT_ATTR = "kafka.count"; // Always 1
until we start supporting merging from batches
+ static constexpr char const* KAFKA_MESSAGE_KEY_ATTR = "kafka.key";
+ static constexpr char const* KAFKA_OFFSET_ATTR = "kafka.offset";
+ static constexpr char const* KAFKA_PARTITION_ATTR = "kafka.partition";
+ static constexpr cha
[jira] [Resolved] (MINIFICPP-1409) Add default credential provider chain support to AWS authentication
[ https://issues.apache.org/jira/browse/MINIFICPP-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi resolved MINIFICPP-1409. -- Resolution: Fixed > Add default credential provider chain support to AWS authentication > --- > > Key: MINIFICPP-1409 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1409 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently access key and secret key authentication is supported for AWS > authentication through properties or credentials property file. Supporting > the default credential provider chain of AWS would allow customers to use > environment variable, profile config file or instance profiles (in that > order) for AWS authentication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] hunyadi-dev opened a new pull request #953: MINIFICPP-1421 - Disable C2JstackTest
hunyadi-dev opened a new pull request #953: URL: https://github.com/apache/nifi-minifi-cpp/pull/953 C2JstackTest is currently looks for a log line that contains the word "SchedulingAgent". This line however is only present due to the LogTestController logs that log lines from the SchedulingAgents will not be shown. ```bash ➜ ./extensions/http-curl/tests/C2JstackTest ../src/libminifi/test/resources/TestHTTPGet.yml ../src/libminifi/test/resources/ |& grep "SchedulingAgent" [2020-12-08 10:48:51.955] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::SchedulingAgent logger got sinks from namespace root and level error from namespace root [2020-12-08 10:48:51.955] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::ThreadedSchedulingAgent logger got sinks from namespace root and level error from namespace root [2020-12-08 10:48:51.955] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::TimerDrivenSchedulingAgent logger got sinks from namespace root and level error from namespace root ``` Enabling all sinks still did not produce any relevant log line that would be worth matching against: ```c++ #include "../../extensions/standard-processors/controllers/UnorderedMapKeyValueStoreService.h" #include "../../extensions/standard-processors/controllers/UnorderedMapPersistableKeyValueStoreService.h" #include "../../extensions/http-curl/processors/InvokeHTTP.h" #include "../../extensions/http-curl/processors/InvokeHTTP.h" #include "../../extensions/standard-processors/processors/LogAttribute.h" #include "../../extensions/standard-processors/processors/LogAttribute.h" #include "../../libminifi/include/c2/ControllerSocketProtocol.h" #include "../../libminifi/include/c2/ControllerSocketProtocol.h" #include "../../extensions/standard-processors/processors/AppendHostInfo.h" #include "../../extensions/standard-processors/processors/AppendHostInfo.h" #include "../../extensions/standard-processors/processors/ExecuteProcess.h" #include "../../extensions/standard-processors/processors/ExecuteProcess.h" #include "../../extensions/standard-processors/processors/ExtractText.h" #include "../../extensions/standard-processors/processors/ExtractText.h" #include "../../extensions/standard-processors/processors/GenerateFlowFile.h" #include "../../extensions/standard-processors/processors/GenerateFlowFile.h" #include "../../extensions/standard-processors/processors/GetFile.h" #include "../../extensions/standard-processors/processors/GetFile.h" #include "../../extensions/standard-processors/processors/GetTCP.h" #include "../../extensions/standard-processors/processors/GetTCP.h" #include "../../extensions/standard-processors/processors/HashContent.h" #include "../../extensions/standard-processors/processors/HashContent.h" #include "../../extensions/standard-processors/processors/ListenSyslog.h" #include "../../extensions/standard-processors/processors/ListenSyslog.h" #include "../../extensions/standard-processors/processors/PutFile.h" #include "../../extensions/standard-processors/processors/PutFile.h" #include "../../extensions/standard-processors/processors/RetryFlowFile.h" #include "../../extensions/standard-processors/processors/RetryFlowFile.h" #include "../../extensions/standard-processors/processors/RouteOnAttribute.h" #include "../../extensions/standard-processors/processors/RouteOnAttribute.h" #include "../../extensions/standard-processors/processors/TailFile.h" #include "../../extensions/standard-processors/processors/TailFile.h" #include "../../extensions/standard-processors/processors/UpdateAttribute.h" #include "../../extensions/standard-processors/processors/UpdateAttribute.h" #include "../../extensions/http-curl/protocols/AgentPrinter.h" #include "../../extensions/http-curl/protocols/AgentPrinter.h" #include "../../extensions/http-curl/protocols/RESTReceiver.h" #include "../../extensions/http-curl/protocols/RESTReceiver.h" #include "../../extensions/civetweb/processors/ListenHTTP.h" #include "../../extensions/civetweb/processors/ListenHTTP.h" (...) LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestController::getInstance().setTrace(); LogTestControl
[jira] [Updated] (MINIFICPP-1421) Investigate and fix C2JstackTest
[
https://issues.apache.org/jira/browse/MINIFICPP-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1421:
Description:
*Background:*
C2JstackTest is currently looks for a log line that contains the word
"SchedulingAgent". This line however is only present due to the
LogTestController logs that log lines from the SchedulingAgents will not be
shown.
{code:bash|title=Filtered test output}
➜ ./extensions/http-curl/tests/C2JstackTest
../src/libminifi/test/resources/TestHTTPGet.yml
../src/libminifi/test/resources/ |& grep "SchedulingAgent"
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::SchedulingAgent logger got sinks from namespace root
and level error from namespace root
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::ThreadedSchedulingAgent logger got sinks from
namespace root and level error from namespace root
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::TimerDrivenSchedulingAgent logger got sinks from
namespace root and level error from namespace root
{code}
Enabling all sinks still did not produce any relevant log line that would be
worth matching against:
{code:c++}
#include
"../../extensions/standard-processors/controllers/UnorderedMapKeyValueStoreService.h"
#include
"../../extensions/standard-processors/controllers/UnorderedMapPersistableKeyValueStoreService.h"
#include "../../extensions/http-curl/processors/InvokeHTTP.h"
#include "../../extensions/http-curl/processors/InvokeHTTP.h"
#include "../../extensions/standard-processors/processors/LogAttribute.h"
#include "../../extensions/standard-processors/processors/LogAttribute.h"
#include "../../libminifi/include/c2/ControllerSocketProtocol.h"
#include "../../libminifi/include/c2/ControllerSocketProtocol.h"
#include "../../extensions/standard-processors/processors/AppendHostInfo.h"
#include "../../extensions/standard-processors/processors/AppendHostInfo.h"
#include "../../extensions/standard-processors/processors/ExecuteProcess.h"
#include "../../extensions/standard-processors/processors/ExecuteProcess.h"
#include "../../extensions/standard-processors/processors/ExtractText.h"
#include "../../extensions/standard-processors/processors/ExtractText.h"
#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
#include "../../extensions/standard-processors/processors/GetFile.h"
#include "../../extensions/standard-processors/processors/GetFile.h"
#include "../../extensions/standard-processors/processors/GetTCP.h"
#include "../../extensions/standard-processors/processors/GetTCP.h"
#include "../../extensions/standard-processors/processors/HashContent.h"
#include "../../extensions/standard-processors/processors/HashContent.h"
#include "../../extensions/standard-processors/processors/ListenSyslog.h"
#include "../../extensions/standard-processors/processors/ListenSyslog.h"
#include "../../extensions/standard-processors/processors/PutFile.h"
#include "../../extensions/standard-processors/processors/PutFile.h"
#include "../../extensions/standard-processors/processors/RetryFlowFile.h"
#include "../../extensions/standard-processors/processors/RetryFlowFile.h"
#include "../../extensions/standard-processors/processors/RouteOnAttribute.h"
#include "../../extensions/standard-processors/processors/RouteOnAttribute.h"
#include "../../extensions/standard-processors/processors/TailFile.h"
#include "../../extensions/standard-processors/processors/TailFile.h"
#include "../../extensions/standard-processors/processors/UpdateAttribute.h"
#include "../../extensions/standard-processors/processors/UpdateAttribute.h"
#include "../../extensions/http-curl/protocols/AgentPrinter.h"
#include "../../extensions/http-curl/protocols/AgentPrinter.h"
#include "../../extensions/http-curl/protocols/RESTReceiver.h"
#include "../../extensions/http-curl/protocols/RESTReceiver.h"
#include "../../extensions/civetweb/processors/ListenHTTP.h"
#include "../../extensions/civetweb/processors/ListenHTTP.h"
(...)
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace()
[jira] [Updated] (MINIFICPP-1421) Investigate and fix C2JstackTest
[
https://issues.apache.org/jira/browse/MINIFICPP-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1421:
Description:
*Background:*
C2JstackTest is currently looks for a log line that contains the word
"SchedulingAgent". This line however is only present due to the
LogTestController logs that log lines from the SchedulingAgents will not be
shown.
{code:bash|title=Filtered test output}
➜ ./extensions/http-curl/tests/C2JstackTest
../src/libminifi/test/resources/TestHTTPGet.yml
../src/libminifi/test/resources/ |& grep "SchedulingAgent"
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::SchedulingAgent logger got sinks from namespace root
and level error from namespace root
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::ThreadedSchedulingAgent logger got sinks from
namespace root and level error from namespace root
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::TimerDrivenSchedulingAgent logger got sinks from
namespace root and level error from namespace root
{code}
Enabling all sinks still did not produce any relevant log line that would be
worth matching against:
{code:c++|title: Enabling all sinks}
#include
"../../extensions/standard-processors/controllers/UnorderedMapKeyValueStoreService.h"
#include
"../../extensions/standard-processors/controllers/UnorderedMapPersistableKeyValueStoreService.h"
#include "../../extensions/http-curl/processors/InvokeHTTP.h"
#include "../../extensions/http-curl/processors/InvokeHTTP.h"
#include "../../extensions/standard-processors/processors/LogAttribute.h"
#include "../../extensions/standard-processors/processors/LogAttribute.h"
#include "../../libminifi/include/c2/ControllerSocketProtocol.h"
#include "../../libminifi/include/c2/ControllerSocketProtocol.h"
#include "../../extensions/standard-processors/processors/AppendHostInfo.h"
#include "../../extensions/standard-processors/processors/AppendHostInfo.h"
#include "../../extensions/standard-processors/processors/ExecuteProcess.h"
#include "../../extensions/standard-processors/processors/ExecuteProcess.h"
#include "../../extensions/standard-processors/processors/ExtractText.h"
#include "../../extensions/standard-processors/processors/ExtractText.h"
#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
#include "../../extensions/standard-processors/processors/GetFile.h"
#include "../../extensions/standard-processors/processors/GetFile.h"
#include "../../extensions/standard-processors/processors/GetTCP.h"
#include "../../extensions/standard-processors/processors/GetTCP.h"
#include "../../extensions/standard-processors/processors/HashContent.h"
#include "../../extensions/standard-processors/processors/HashContent.h"
#include "../../extensions/standard-processors/processors/ListenSyslog.h"
#include "../../extensions/standard-processors/processors/ListenSyslog.h"
#include "../../extensions/standard-processors/processors/PutFile.h"
#include "../../extensions/standard-processors/processors/PutFile.h"
#include "../../extensions/standard-processors/processors/RetryFlowFile.h"
#include "../../extensions/standard-processors/processors/RetryFlowFile.h"
#include "../../extensions/standard-processors/processors/RouteOnAttribute.h"
#include "../../extensions/standard-processors/processors/RouteOnAttribute.h"
#include "../../extensions/standard-processors/processors/TailFile.h"
#include "../../extensions/standard-processors/processors/TailFile.h"
#include "../../extensions/standard-processors/processors/UpdateAttribute.h"
#include "../../extensions/standard-processors/processors/UpdateAttribute.h"
#include "../../extensions/http-curl/protocols/AgentPrinter.h"
#include "../../extensions/http-curl/protocols/AgentPrinter.h"
#include "../../extensions/http-curl/protocols/RESTReceiver.h"
#include "../../extensions/http-curl/protocols/RESTReceiver.h"
#include "../../extensions/civetweb/processors/ListenHTTP.h"
#include "../../extensions/civetweb/processors/ListenHTTP.h"
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::ge
[jira] [Created] (MINIFICPP-1421) Investigate and fix C2JstackTest
Adam Hunyadi created MINIFICPP-1421:
---
Summary: Investigate and fix C2JstackTest
Key: MINIFICPP-1421
URL: https://issues.apache.org/jira/browse/MINIFICPP-1421
Project: Apache NiFi MiNiFi C++
Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Adam Hunyadi
Fix For: 0.8.0
*Background:*
C2JstackTest is currently looks for a log line that contains the word
"SchedulingAgent". This line however is only present due to the
LogTestController logs that log lines from the SchedulingAgents will not be
shown.
{code:bash|title=Filtered test output}
➜ ./extensions/http-curl/tests/C2JstackTest
../src/libminifi/test/resources/TestHTTPGet.yml
../src/libminifi/test/resources/ |& grep "SchedulingAgent"
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::SchedulingAgent logger got sinks from namespace root
and level error from namespace root
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::ThreadedSchedulingAgent logger got sinks from
namespace root and level error from namespace root
[2020-12-08 10:48:51.955]
[org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug]
org::apache::nifi::minifi::TimerDrivenSchedulingAgent logger got sinks from
namespace root and level error from namespace root
{code}
*Proposal:*
As the test breaks if once we disable the inclusion of "logger got sinks" by
default on test, we should turn off this test and at some point investigate
what exactly the test was meant to check. One could probably check out the
version of the MiNiFi this was introduced in and see if something important was
captured by checking against this log line.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
