[jira] [Created] (NIFI-7713) During JoltTransform Charaset for foreign language not working
Praneeth Gajji created NIFI-7713: Summary: During JoltTransform Charaset for foreign language not working Key: NIFI-7713 URL: https://issues.apache.org/jira/browse/NIFI-7713 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.11.4 Environment: Windows 10,pachi Nifi UI Reporter: Praneeth Gajji Fix For: 1.11.4 Attachments: JoltTransformJson_processor.png Hi I have a record where some text is in foreing language like Russia which is supported by utf-8 only . I have take processorJolttransformJson and tried to convert text and after convrsion I see wrong data getting populated for Russian text . When I see processor configuration I dont have any option to specify default charset i.e utf-8 , Attached screenshot of JoltTransformJson and options available to the processor kindly assist -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-3096) Issues with NTLM authentication in GetHttp and InvokeHttp processors
[ https://issues.apache.org/jira/browse/NIFI-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172765#comment-17172765 ] Eduardo Mota Fontes commented on NIFI-3096: --- There is a implementation of InvokeHTTP with NTLM support here [https://github.com/peetkes/nifi-http-processor] It would be nice to do a merge. > Issues with NTLM authentication in GetHttp and InvokeHttp processors > > > Key: NIFI-3096 > URL: https://issues.apache.org/jira/browse/NIFI-3096 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Michael Kalika >Assignee: Karthik Narayanan >Priority: Major > > GetHttp and InvokeHttp do not support HTTP endpoints protected with NTLM > authentication. This functionality is needed for working with services and > APIs hosted in Windows systems. > Here's a log with error: > 2016-11-23 17:45:22,272 WARN [Timer-Driven Process Thread-7] > o.a.http.impl.auth.HttpAuthenticator NEGOTIATE authentication error: No valid > credentials provided (Mechanism level: No valid credentials provided > (Mechanism > level: Failed to find any Kerberos tgt)) > 2016-11-23 17:45:22,272 WARN [Timer-Driven Process Thread-7] > o.a.http.impl.auth.HttpAuthenticator NTLM authentication error: Credentials > cannot be used for NTLM authentication: > org.apache.http.auth.UsernamePasswordCredentials > 2016-11-23 17:45:22,274 ERROR [Timer-Driven Process > Thread-7] o.a.nifi.processors.standard.GetHTTP > GetHTTP[id=91ce7dfd-0158-1000-1c53-cf24cf132983] received status code > 401:Unauthorized from http://internal.api.ep/dosomething > and a discussion about this issue > https://community.hortonworks.com/questions/68097/issues-with-gethttp-and-ntlm-authentication-in-nif.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] alopresto edited a comment on pull request #4458: NIFI-7572: Force ScriptedTransformRecord to recompile the script when started
alopresto edited a comment on pull request #4458: URL: https://github.com/apache/nifi/pull/4458#issuecomment-670258777 Sorry for the delay, had some unrelated test failures occurring because of networking changes on my machine. Verified the expected behavior, contrib-check, and all tests. Thanks Matt. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy LoPresto updated NIFI-7572:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Closing again after Matt Burgess resolved the "recompile script on change"
issue.
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in some very specific way, but NiFi doesn't make this as
> easy as it should. There are methods using ExecuteScript,
> InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
> instance.
> But each of these requires that the Script writer understand a lot about NiFi
> and how to expose properties, create Property Descriptors, etc. and for
> fairly simple transformation we end up with scripts where the logic takes
> fewer lines of code than the boilerplate.
> We should expose a Processor that allows a user to write a script that takes
> a Record and transforms that Record in some way. The processor should be
> configured with the following:
> * Record Reader (required)
> * Record Writer (required)
> * Script Language (required)
> * Script Body or Script File (one and only one of these required)
> The script should implement a single method along the lines of:
> {code:java}
> Record transform(Record input) throws Exception; {code}
> If the script returns null, the input Record should be dropped. Otherwise,
> whatever Record is returned should be written to the Record Writer.
> The processor should have two relationships: "success" and "failure."
> The script should not be allowed to expose any properties or define any
> relationships. The point is to keep the script focused purely on processing
> the record itself.
> It's not entirely clear to me how easy the Record API works with some of the
> scripting languages. The Record object does expose a method named toMap()
> that returns a Map containing the underlying key/value pairs.
> However, the values in that Map may themselves be Records. It might make
> sense to expose a new method toNormalizedMap() or something along those lines
> that would return a Map where the values have been
> recursively normalized, in much the same way that we do for
> JoltTransformRecord. This would perhaps allow for cleaner syntax, but I'm not
> a scripting expert so I can't say for sure whether such a method is necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] alopresto closed pull request #4458: NIFI-7572: Force ScriptedTransformRecord to recompile the script when started
alopresto closed pull request #4458: URL: https://github.com/apache/nifi/pull/4458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172754#comment-17172754
]
ASF subversion and git services commented on NIFI-7572:
---
Commit 6488db13762fd42ce6c51f574e2e47ac29006ab5 in nifi's branch
refs/heads/main from Matt Burgess
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6488db1 ]
NIFI-7572: Force ScriptedTransformRecord to recompile the script when started
This closes #4458.
Signed-off-by: Andy LoPresto
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in some very specific way, but NiFi doesn't make this as
> easy as it should. There are methods using ExecuteScript,
> InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
> instance.
> But each of these requires that the Script writer understand a lot about NiFi
> and how to expose properties, create Property Descriptors, etc. and for
> fairly simple transformation we end up with scripts where the logic takes
> fewer lines of code than the boilerplate.
> We should expose a Processor that allows a user to write a script that takes
> a Record and transforms that Record in some way. The processor should be
> configured with the following:
> * Record Reader (required)
> * Record Writer (required)
> * Script Language (required)
> * Script Body or Script File (one and only one of these required)
> The script should implement a single method along the lines of:
> {code:java}
> Record transform(Record input) throws Exception; {code}
> If the script returns null, the input Record should be dropped. Otherwise,
> whatever Record is returned should be written to the Record Writer.
> The processor should have two relationships: "success" and "failure."
> The script should not be allowed to expose any properties or define any
> relationships. The point is to keep the script focused purely on processing
> the record itself.
> It's not entirely clear to me how easy the Record API works with some of the
> scripting languages. The Record object does expose a method named toMap()
> that returns a Map containing the underlying key/value pairs.
> However, the values in that Map may themselves be Records. It might make
> sense to expose a new method toNormalizedMap() or something along those lines
> that would return a Map where the values have been
> recursively normalized, in much the same way that we do for
> JoltTransformRecord. This would perhaps allow for cleaner syntax, but I'm not
> a scripting expert so I can't say for sure whether such a method is necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] alopresto commented on pull request #4458: NIFI-7572: Force ScriptedTransformRecord to recompile the script when started
alopresto commented on pull request #4458: URL: https://github.com/apache/nifi/pull/4458#issuecomment-670258777 Sorry for the delay, had some unrelated test failures occurring because of networking changes on my machine. Verified the expected behavior, contrib-check, and all tests. Thanks Matt. https://user-images.githubusercontent.com/798465/89596304-abfdd380-d80b-11ea-9f83-c8ebafd1bd85.png";> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 opened a new pull request #4459: NIFI-7592: Revert
mattyb149 opened a new pull request #4459:
URL: https://github.com/apache/nifi/pull/4459
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Reverts the previous commits for NIFI-7592
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7708) Add Azure SDK for Java to LICENCE and NOTICE files
[ https://issues.apache.org/jira/browse/NIFI-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7708: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add Azure SDK for Java to LICENCE and NOTICE files > -- > > Key: NIFI-7708 > URL: https://issues.apache.org/jira/browse/NIFI-7708 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Major > Labels: Azure, adls > Fix For: 1.12.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > License and notice files should be extended with information regarding the > usage of Azure SDK for Java and its transitive dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-3566) Node fails to pull flow.xml from NCM, purges content repo
[
https://issues.apache.org/jira/browse/NIFI-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172685#comment-17172685
]
Brandon Rhys DeVries commented on NIFI-3566:
Nope, sounds like that should cover it!
> Node fails to pull flow.xml from NCM, purges content repo
> -
>
> Key: NIFI-3566
> URL: https://issues.apache.org/jira/browse/NIFI-3566
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: Brandon Rhys DeVries
>Priority: Minor
>
> We have an instance were a node was removed from a cluster to address a
> production data flow issue. During this process, changes were made such that
> it's flow.xml was different from the cluster (different run states). The
> general procedure we follow in this case is to remove the Node's flow.xml,
> and let is pull the "correct" / consistent one from the NCM. However, in
> this case, something prevented the NCM's flow.xml from propagating to the
> Node. the Node ended up with an empty flow.xml... and then proceeded to
> purge all of the content repo with the warning "{} maps to unknown FlowFile
> Queue {}; this record will be discarded"\[1].
> In cases like this, we should see if we can be a bit more friendly.
> Specifically, in our case, it would have been preferable to shut down rather
> than delete the content repo. It would seem to me that if an admin
> intentionally removes the flow.xml, it would not be unreasonable to make it
> their responsibility to also remove the content repo (and possibly
> others...). But cases in which a network hiccup can cause 100% data loss on
> a node seem bad.
> \[1]
> https://github.com/apache/nifi/blob/rel/nifi-0.7.1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java#L717
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-3566) Node fails to pull flow.xml from NCM, purges content repo
[
https://issues.apache.org/jira/browse/NIFI-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172657#comment-17172657
]
Mark Payne commented on NIFI-3566:
--
OK gotcha, thanks [~devriesb]. So there are kind of two issues for this Jira,
if I am understanding correctly. (1) There was a problem where the node failed
to pull the flow from NCM and ended up with an empty flow. I don't believe that
is a problem in 1.x. (2) If the node does end up with an empty flow, it blows
away the data. This is still a concern with 1.x.
This second issue, though, ties together with NIFI-7706 and NIFI-5702. Since
those are both being addressed in 1.12.0, I think this makes sense to close
this Jira.
Do you agree? Or have I misunderstood?
> Node fails to pull flow.xml from NCM, purges content repo
> -
>
> Key: NIFI-3566
> URL: https://issues.apache.org/jira/browse/NIFI-3566
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: Brandon Rhys DeVries
>Priority: Minor
>
> We have an instance were a node was removed from a cluster to address a
> production data flow issue. During this process, changes were made such that
> it's flow.xml was different from the cluster (different run states). The
> general procedure we follow in this case is to remove the Node's flow.xml,
> and let is pull the "correct" / consistent one from the NCM. However, in
> this case, something prevented the NCM's flow.xml from propagating to the
> Node. the Node ended up with an empty flow.xml... and then proceeded to
> purge all of the content repo with the warning "{} maps to unknown FlowFile
> Queue {}; this record will be discarded"\[1].
> In cases like this, we should see if we can be a bit more friendly.
> Specifically, in our case, it would have been preferable to shut down rather
> than delete the content repo. It would seem to me that if an admin
> intentionally removes the flow.xml, it would not be unreasonable to make it
> their responsibility to also remove the content repo (and possibly
> others...). But cases in which a network hiccup can cause 100% data loss on
> a node seem bad.
> \[1]
> https://github.com/apache/nifi/blob/rel/nifi-0.7.1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java#L717
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] goosalex edited a comment on pull request #4410: NIFI-7642: KafkaConsumers - improve batching into FlowFiles under high latency conditions
goosalex edited a comment on pull request #4410: URL: https://github.com/apache/nifi/pull/4410#issuecomment-670169916 Thanks @pvillard31 for having a look at it. We were noticing too small Flowfiles compared to the "maxPollRecords" when handling production Flows ( up to over 250'000msg/s ), on a secured Kafka cluster in a non trivial network layout. The fixed 10ms timeout per single poll roundtrip led to undeterministic polling results, reporting "no more records" after ~200 messages on average, never mind "maxPollRecords" and "maxWatingMilllis" being very high. (we would have preferred ~1msg/FF) In a simple, unsecured Lab setting, we could not reproduce the issue. As I understand it, "maxWaitmillis" is the maximum time a Nifi-KafkaConsumer-Processor can use up to fill a FlowFile, whereas x in kafkaConsumer.poll(x) is the timeout for a single Kafka-transaction. Perhaps we should not put the user supplied-maWaitMillis directly here, but also a lower bound of the previously hard-coded 10ms ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] goosalex commented on pull request #4410: NIFI-7642: KafkaConsumers - improve batching into FlowFiles under high latency conditions
goosalex commented on pull request #4410: URL: https://github.com/apache/nifi/pull/4410#issuecomment-670169916 Thanks @pvillard31 for having a look at it. We were noticing too small Flowfiles compared to the "maxPollRecords" when handling production Flows ( up to over 250'000msg/s ), on a secured Kafka cluster in a non trivial network layout. The fixed 10ms timeout per single poll roundtrip led to undeterministic polling results, reporting "no more records" after ~200 messages on average, never mind "maxPollRecords" and "maxWatingMilllis" beeing very high. In a simple, unsecured Lab setting, we could not reproduce the issue. As I understand it, "maxWaitmillis" is the maximum time a Nifi-KafkaConsumer-Processor can use up to fill a FlowFile, whereas x in kafkaConsumer.poll(x) is the timeout for a single Kafka-transaction. Perhaps we should not put the user supplied-maWaitMillis directly here, but also a lower bound of the previously hard-coded 10ms ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFIREG-411) Add additional authenitcation methods to nifi registry client
Bryan Bende created NIFIREG-411: --- Summary: Add additional authenitcation methods to nifi registry client Key: NIFIREG-411 URL: https://issues.apache.org/jira/browse/NIFIREG-411 Project: NiFi Registry Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Bryan Bende Assignee: Bryan Bende Currently nifi-registry-client only allows client certs with an optional proxied entity. We should add the ability to obtain an access token by providing login credentials and be able to pass that token back on future requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFIREG-408) Add clients for tenants and policies
[ https://issues.apache.org/jira/browse/NIFIREG-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Doran resolved NIFIREG-408. - Fix Version/s: 0.8.0 Resolution: Done > Add clients for tenants and policies > > > Key: NIFIREG-408 > URL: https://issues.apache.org/jira/browse/NIFIREG-408 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.8.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Add clients to nifi-registry-client for tenants and policies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-registry] kevdoran closed pull request #292: NIFIREG-408 Adding clients for tenants and policies
kevdoran closed pull request #292: URL: https://github.com/apache/nifi-registry/pull/292 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-registry] kevdoran commented on pull request #292: NIFIREG-408 Adding clients for tenants and policies
kevdoran commented on pull request #292: URL: https://github.com/apache/nifi-registry/pull/292#issuecomment-670132940 Hey, @bbende. I agree that it is a testing issue, and also that this change, although it doe not fix the root cause, does make the test seemingly pass reliably. +1 from me, will merge. We can monitor the testing setup to see if we need to fi the root cause at some point... thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-registry] bbende commented on pull request #292: NIFIREG-408 Adding clients for tenants and policies
bbende commented on pull request #292: URL: https://github.com/apache/nifi-registry/pull/292#issuecomment-670089065 I can't figure out why that error seems to happen only on the updateUserGroup call, but I tried a few things and so far switching the order in the test where it does "create, retrieve, update, delete", instead of "create, update, retrieve, delete", seems to make it consistently pass. I feel like this is more some testing environmental issue, since even in the previous scenario, the create and update were working on the server side, but for some reason the connection was closed by the time the client tried to read the response. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7708) Add Azure SDK for Java to LICENCE and NOTICE files
[ https://issues.apache.org/jira/browse/NIFI-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172582#comment-17172582 ] ASF subversion and git services commented on NIFI-7708: --- Commit 5cc7c0485aedfac7ad731aba5fcfbec67e8806f2 in nifi's branch refs/heads/main from Peter Gyori [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5cc7c04 ] NIFI-7708: Add Azure SDK for Java (and transitive dependencies) to LICENCE and NOTICE files This closes #4452 Signed-off-by: Joey Frazee > Add Azure SDK for Java to LICENCE and NOTICE files > -- > > Key: NIFI-7708 > URL: https://issues.apache.org/jira/browse/NIFI-7708 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Major > Labels: Azure, adls > Fix For: 1.12.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > License and notice files should be extended with information regarding the > usage of Azure SDK for Java and its transitive dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] jfrazee closed pull request #4452: NIFI-7708: Add Azure SDK for Java (and transitive dependencies) to LI…
jfrazee closed pull request #4452: URL: https://github.com/apache/nifi/pull/4452 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] jfrazee commented on pull request #4452: NIFI-7708: Add Azure SDK for Java (and transitive dependencies) to LI…
jfrazee commented on pull request #4452: URL: https://github.com/apache/nifi/pull/4452#issuecomment-670055233 @pgyori LGTM. I did a check of the included Jars and this looks correct. Re-building the rebase and then will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7592) Allow NiFi to be started without a GUI/REST interface
[ https://issues.apache.org/jira/browse/NIFI-7592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7592: --- Status: Patch Available (was: Reopened) > Allow NiFi to be started without a GUI/REST interface > - > > Key: NIFI-7592 > URL: https://issues.apache.org/jira/browse/NIFI-7592 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.12.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > In conjunction with MINIFI-422 (bringing MiNiFi into the NiFi codebase), it > would be necessary to allow a NiFi build to run without having the GUI and > REST API components required. For normal NiFi releases, the GUI would be > included in the assembly, but this Jira proposes to reorganize and refactor > the framework code to allow NiFi to run flows and such without requiring the > web bundle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] alopresto commented on pull request #4458: NIFI-7572: Force ScriptedTransformRecord to recompile the script when started
alopresto commented on pull request #4458: URL: https://github.com/apache/nifi/pull/4458#issuecomment-670010842 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #4457: NIFI-7592: Make nifi-framework-nar the parent of the NiFiServer NARs, fix dependencies
markap14 commented on pull request #4457: URL: https://github.com/apache/nifi/pull/4457#issuecomment-669983009 Thanks @mattyb149 . Seems to be an issue with clustering here though. I see the following error: ``` 2020-08-06 10:51:38,130 WARN [Process Cluster Protocol Request-3] org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered exception due to: java.lang.IllegalAccessError: tried to access method org.apache.nifi.cluster.coordination.node.NodeConnectionStatus.updateIdGenerator(J)V from class org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator java.lang.IllegalAccessError: tried to access method org.apache.nifi.cluster.coordination.node.NodeConnectionStatus.updateIdGenerator(J)V from class org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator at org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.handleNodeStatusChange(NodeClusterCoordinator.java:1089) at org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.handle(NodeClusterCoordinator.java:1020) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:176) at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:131) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` I also noticed the following: ``` work $ find . -name nifi-framework-cluster\* ./nar/framework/nifi-framework-nar-1.12.0-SNAPSHOT.nar-unpacked/NAR-INF/bundled-dependencies/nifi-framework-cluster-protocol-1.12.0-SNAPSHOT.jar ./nar/extensions/nifi-server-nar-1.12.0-SNAPSHOT.nar-unpacked/NAR-INF/bundled-dependencies/nifi-framework-cluster-1.12.0-SNAPSHOT.jar ./nar/extensions/nifi-server-nar-1.12.0-SNAPSHOT.nar-unpacked/NAR-INF/bundled-dependencies/nifi-framework-cluster-protocol-1.12.0-SNAPSHOT.jar ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-7572:
---
Status: Patch Available (was: Reopened)
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in some very specific way, but NiFi doesn't make this as
> easy as it should. There are methods using ExecuteScript,
> InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
> instance.
> But each of these requires that the Script writer understand a lot about NiFi
> and how to expose properties, create Property Descriptors, etc. and for
> fairly simple transformation we end up with scripts where the logic takes
> fewer lines of code than the boilerplate.
> We should expose a Processor that allows a user to write a script that takes
> a Record and transforms that Record in some way. The processor should be
> configured with the following:
> * Record Reader (required)
> * Record Writer (required)
> * Script Language (required)
> * Script Body or Script File (one and only one of these required)
> The script should implement a single method along the lines of:
> {code:java}
> Record transform(Record input) throws Exception; {code}
> If the script returns null, the input Record should be dropped. Otherwise,
> whatever Record is returned should be written to the Record Writer.
> The processor should have two relationships: "success" and "failure."
> The script should not be allowed to expose any properties or define any
> relationships. The point is to keep the script focused purely on processing
> the record itself.
> It's not entirely clear to me how easy the Record API works with some of the
> scripting languages. The Record object does expose a method named toMap()
> that returns a Map containing the underlying key/value pairs.
> However, the values in that Map may themselves be Records. It might make
> sense to expose a new method toNormalizedMap() or something along those lines
> that would return a Map where the values have been
> recursively normalized, in much the same way that we do for
> JoltTransformRecord. This would perhaps allow for cleaner syntax, but I'm not
> a scripting expert so I can't say for sure whether such a method is necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-7706) When NiFi is restarted, if FlowFile is queued for a connection that no longer exists, the FlowFile remains but its data does not
[
https://issues.apache.org/jira/browse/NIFI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Bende resolved NIFI-7706.
---
Resolution: Fixed
> When NiFi is restarted, if FlowFile is queued for a connection that no longer
> exists, the FlowFile remains but its data does not
>
>
> Key: NIFI-7706
> URL: https://issues.apache.org/jira/browse/NIFI-7706
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.12.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When NiFi is restarted, if there is data in the FlowFile Repository that maps
> to a given connection, it is placed back into the queue for that connection.
> If the connection does not exist, it logs a WARNING like the following:
> {quote}
> nifi-app_2020-08-04_12.0.log:2020-08-04 12:18:19,840 WARN [main]
> o.a.n.c.r.WriteAheadFlowFileRepository Encountered Repository Record (id=137)
> with Queue identifier ba3261a3-0173-1000-4582-5ee7cffe1e8b but no Queue
> exists with that ID. Dropping this FlowFile
> {quote}
> The data is not actually dropped - it remains within the NiFi instance and if
> the flow is restored to have the connection and NiFi is restarted, the
> FlowFile will be placed back into the queue. However, when the connection
> does not exist, the Claimant Count is not incremented for the Content Claim.
> As a result, the data may be reclaimed by the Content Repository. As a
> result, when the flow is restored and NiFi restarted, the data exists in the
> queue but its content is gone.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-5702) FlowFileRepo should not discard data (at least not by default)
[ https://issues.apache.org/jira/browse/NIFI-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-5702: -- Resolution: Fixed Status: Resolved (was: Patch Available) > FlowFileRepo should not discard data (at least not by default) > -- > > Key: NIFI-5702 > URL: https://issues.apache.org/jira/browse/NIFI-5702 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1, 1.9.2 >Reporter: Brandon Rhys DeVries >Assignee: Mark Payne >Priority: Major > Fix For: 1.12.0 > > > The WriteAheadFlowFileRepository currently discards data it cannot find a > queue for. Unfortunately, we have run in to issues where, when rejoining a > node to a cluster, the flow.xml.gz can go "missing". This results in the > instance creating a new, empty, flow.xml.gz and then continuing on... and not > finding queues for any of its existing data, dropping it all. Regardless of > the circumstances leading to an empty (or unexpectedly modified) flow.xml.gz, > dropping data without user input seems less than ideal. > Internally, my group has added a property > "remove.orphaned.flowfiles.on.startup", defaulting to "false". On > startup, rather than silently dropping data, the repo will throw an exception > preventing startup. The operator can then choose to either "fix" any > unexpected issues with the flow.xml.gz, or they can set the above property to > "true" which restores the original behavior allowing the system to be > restarted. When set to "true" this property also results in a warning > message indicating that in this configuration the repo can drop data without > (advance) warning. > > > [1] > https://github.com/apache/nifi/blob/support/nifi-1.7.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java#L596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4454: NIFI-7706, NIFI-5702: Allow NiFi to keep FlowFiles if their queue is …
asfgit closed pull request #4454: URL: https://github.com/apache/nifi/pull/4454 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-5702) FlowFileRepo should not discard data (at least not by default)
[ https://issues.apache.org/jira/browse/NIFI-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172443#comment-17172443 ] ASF subversion and git services commented on NIFI-5702: --- Commit eca7f153d0b23d39f14e5c65d1f96c7f663b2bbe in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=eca7f15 ] NIFI-7706, NIFI-5702: Allow NiFi to keep FlowFiles if their queue is unknown. This way, if a Flow is inadvertently removed, updated, etc., and NiFi is restarted, the data will not be dropped by default. The old mechanism of dropping data is exposed via a property This closes #4454. Signed-off-by: Bryan Bende > FlowFileRepo should not discard data (at least not by default) > -- > > Key: NIFI-5702 > URL: https://issues.apache.org/jira/browse/NIFI-5702 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1, 1.9.2 >Reporter: Brandon Rhys DeVries >Assignee: Mark Payne >Priority: Major > Fix For: 1.12.0 > > > The WriteAheadFlowFileRepository currently discards data it cannot find a > queue for. Unfortunately, we have run in to issues where, when rejoining a > node to a cluster, the flow.xml.gz can go "missing". This results in the > instance creating a new, empty, flow.xml.gz and then continuing on... and not > finding queues for any of its existing data, dropping it all. Regardless of > the circumstances leading to an empty (or unexpectedly modified) flow.xml.gz, > dropping data without user input seems less than ideal. > Internally, my group has added a property > "remove.orphaned.flowfiles.on.startup", defaulting to "false". On > startup, rather than silently dropping data, the repo will throw an exception > preventing startup. The operator can then choose to either "fix" any > unexpected issues with the flow.xml.gz, or they can set the above property to > "true" which restores the original behavior allowing the system to be > restarted. When set to "true" this property also results in a warning > message indicating that in this configuration the repo can drop data without > (advance) warning. > > > [1] > https://github.com/apache/nifi/blob/support/nifi-1.7.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java#L596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7706) When NiFi is restarted, if FlowFile is queued for a connection that no longer exists, the FlowFile remains but its data does not
[
https://issues.apache.org/jira/browse/NIFI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172442#comment-17172442
]
ASF subversion and git services commented on NIFI-7706:
---
Commit eca7f153d0b23d39f14e5c65d1f96c7f663b2bbe in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=eca7f15 ]
NIFI-7706, NIFI-5702: Allow NiFi to keep FlowFiles if their queue is unknown.
This way, if a Flow is inadvertently removed, updated, etc., and NiFi is
restarted, the data will not be dropped by default. The old mechanism of
dropping data is exposed via a property
This closes #4454.
Signed-off-by: Bryan Bende
> When NiFi is restarted, if FlowFile is queued for a connection that no longer
> exists, the FlowFile remains but its data does not
>
>
> Key: NIFI-7706
> URL: https://issues.apache.org/jira/browse/NIFI-7706
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.12.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When NiFi is restarted, if there is data in the FlowFile Repository that maps
> to a given connection, it is placed back into the queue for that connection.
> If the connection does not exist, it logs a WARNING like the following:
> {quote}
> nifi-app_2020-08-04_12.0.log:2020-08-04 12:18:19,840 WARN [main]
> o.a.n.c.r.WriteAheadFlowFileRepository Encountered Repository Record (id=137)
> with Queue identifier ba3261a3-0173-1000-4582-5ee7cffe1e8b but no Queue
> exists with that ID. Dropping this FlowFile
> {quote}
> The data is not actually dropped - it remains within the NiFi instance and if
> the flow is restored to have the connection and NiFi is restarted, the
> FlowFile will be placed back into the queue. However, when the connection
> does not exist, the Claimant Count is not incremented for the Content Claim.
> As a result, the data may be reclaimed by the Content Repository. As a
> result, when the flow is restored and NiFi restarted, the data exists in the
> queue but its content is gone.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] mattyb149 opened a new pull request #4458: NIFI-7572: Force ScriptedTransformRecord to recompile the script when started
mattyb149 opened a new pull request #4458:
URL: https://github.com/apache/nifi/pull/4458
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
The reference to the compiled script was not being cleared out before
restarting the processor, so any changes made to the script were not applied.
This PR simply resets the compiled script to null when the processor is
scheduled.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] bbende commented on pull request #4454: NIFI-7706, NIFI-5702: Allow NiFi to keep FlowFiles if their queue is …
bbende commented on pull request #4454: URL: https://github.com/apache/nifi/pull/4454#issuecomment-669978352 Looks good, verified the functionality and ran system tests. There are a couple of checkstyle issues, but I'll resolve locally and include when merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7705) UI - update frontend deps
[ https://issues.apache.org/jira/browse/NIFI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Fellows updated NIFI-7705: -- Resolution: Fixed Status: Resolved (was: Patch Available) > UI - update frontend deps > - > > Key: NIFI-7705 > URL: https://issues.apache.org/jira/browse/NIFI-7705 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.12.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7705) UI - update frontend deps
[ https://issues.apache.org/jira/browse/NIFI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172434#comment-17172434 ] ASF subversion and git services commented on NIFI-7705: --- Commit 6596fb1f8724d6163db3c4f899a57049a8355274 in nifi's branch refs/heads/main from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6596fb1 ] [NIFI-7705] update fronted deps make variable and parameters tables position relative Signed-off-by: Rob Fellows This closes #4449 > UI - update frontend deps > - > > Key: NIFI-7705 > URL: https://issues.apache.org/jira/browse/NIFI-7705 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.12.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4449: [NIFI-7705] update frontend deps
asfgit closed pull request #4449: URL: https://github.com/apache/nifi/pull/4449 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 opened a new pull request #4457: NIFI-7592: Make nifi-framework-nar the parent of the NiFiServer NARs, fix dependencies
mattyb149 opened a new pull request #4457:
URL: https://github.com/apache/nifi/pull/4457
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
The components in the framework NAR were not being loaded, and after fixing
that, some Jersey classes were not being loaded correctly, so this PR
reorganizes the NAR structure to make nifi-framework-nar the parent of
nifi-server-nar and nifi-headless-nar (the two NiFiServer implementations),
delegating the Jersey dependencies up to the nifi-framework-nar classloader.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #849: MINIFICPP-1261 - Mark ScopeGuard class deprecated
arpadboda closed pull request #849: URL: https://github.com/apache/nifi-minifi-cpp/pull/849 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] joewitt commented on pull request #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
joewitt commented on pull request #4169: URL: https://github.com/apache/nifi/pull/4169#issuecomment-669963201 @si-sun the thread safety point is still of interest. That we are synchronizing initialization is good. But is the class itself thread safe for concurrent usage? I've just looked briefly on their code/javadocs and not seeing any indicator yet. We should know this and if it is safe then great. If it is not then we need to make the class TriggerSerially so that it wont allow multiple threads. Also, the windows and osx tests failed on the last build. I suspect the osx issue is just environment issues on github ci that we see time to time. But the windows builds have been quite stable. Have you built this on windows? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] si-sun commented on pull request #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
si-sun commented on pull request #4169: URL: https://github.com/apache/nifi/pull/4169#issuecomment-669957057 bibistroc also found that the file size attribute wasn't consistent with other processors, so he was kind enough to add this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] si-sun commented on a change in pull request #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
si-sun commented on a change in pull request #4169:
URL: https://github.com/apache/nifi/pull/4169#discussion_r466446066
##
File path:
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java
##
@@ -0,0 +1,521 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.mserref.NtStatus;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileAllInformation;
+import com.hierynomus.msfscc.fileinformation.FileBasicInformation;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.InputStream;
+
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.List;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Collections;
+import java.util.ListIterator;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.ArrayList;
+import java.util.Locale;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.regex.Pattern;
+
+@TriggerWhenEmpty
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"samba, smb, cifs, files, get"})
+@CapabilityDescription("Reads file from a samba network location to FlowFiles.
" +
+"Use this processor instead of a cifs mounts if share access control is
important.")
+@SeeAlso({PutSmbFile.class})
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The filename
is set to the name of the file on the network share"),
+@WritesAttribute(attribute = "path", description = "The path is set to
the relative path of the file's network share name. For example, "
++ "if the input is set to hostname\\share\\tmp, files
picked up from \\tmp will have the path attribute set to tmp"),
+@WritesAttribute(attribute = "file.creationTime", description = "The
date and time that the file was created. May not work on all file systems"),
+@WritesAttribute(attribute = "file.lastModifiedTime", description =
"The date and time that the file was last modified. May not work on all "
++ "file systems"),
+@WritesAttribute(attribute = "file.lastAccessTime", description = "The
date and time that the file was last accessed. May not work on all "
++ "file systems"
[jira] [Created] (MINIFICPP-1324) Fix transiently failing tests in MINIFI
Adam Hunyadi created MINIFICPP-1324: --- Summary: Fix transiently failing tests in MINIFI Key: MINIFICPP-1324 URL: https://issues.apache.org/jira/browse/MINIFICPP-1324 Project: Apache NiFi MiNiFi C++ Issue Type: Epic Affects Versions: 0.7.0 Reporter: Adam Hunyadi Fix For: 1.0.0 *Background:* The CI jobs are currently unreliable as in they produce quite a lot of transient failures when running the tests. *Propopsal:* We should go through the recent GitHub Actions CI jobs and check if we can fix each of the recent transient errors. Any non-trivial fix should have its Jira linked to this epic. *Acceptance criteria:* We should have at least 97% confidence of getting no transient error on a single CI job for a total of min. 70% chance of getting no failures on any of the 12 CI platforms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #859: MINIFICPP-1313 Do a multifile lookup in onSchedule
arpadboda closed pull request #859: URL: https://github.com/apache/nifi-minifi-cpp/pull/859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1121) Upgrade spdlog
[
https://issues.apache.org/jira/browse/MINIFICPP-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1121:
Fix Version/s: 1.0.0
> Upgrade spdlog
> --
>
> Key: MINIFICPP-1121
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1121
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Marton Szasz
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Fix For: 1.0.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Or version of spdlog is 2+ years old. The new spdlog version uses a new
> version of the cppformat (back then)/fmt (now) formatting library.
> We should consider directly depending on {{fmt}} since we already have it as
> a transitive dependency and it would be useful for e.g. formatting
> exception/error messages, etc.
>
> *Update (hunyadi):*
> Seems like we have to skip version 1.0 with upgrading. There are quite a lot
> of non-documented breaking changes, for example this commit:
>
> [https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
> breaks ::instance() on stdout/stderr sinks. Unfortunately, changes like this
> in spdlog are not documented and the codebase is kept up-to-date with commits
> sent directly to the central repository.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1121) Upgrade spdlog
[
https://issues.apache.org/jira/browse/MINIFICPP-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1121:
Affects Version/s: 0.7.0
> Upgrade spdlog
> --
>
> Key: MINIFICPP-1121
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1121
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Marton Szasz
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Or version of spdlog is 2+ years old. The new spdlog version uses a new
> version of the cppformat (back then)/fmt (now) formatting library.
> We should consider directly depending on {{fmt}} since we already have it as
> a transitive dependency and it would be useful for e.g. formatting
> exception/error messages, etc.
>
> *Update (hunyadi):*
> Seems like we have to skip version 1.0 with upgrading. There are quite a lot
> of non-documented breaking changes, for example this commit:
>
> [https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
> breaks ::instance() on stdout/stderr sinks. Unfortunately, changes like this
> in spdlog are not documented and the codebase is kept up-to-date with commits
> sent directly to the central repository.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] turcsanyip commented on a change in pull request #4425: NIFI-7663 Allow user to empty all queues in a Process Group
turcsanyip commented on a change in pull request #4425:
URL: https://github.com/apache/nifi/pull/4425#discussion_r466332998
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java
##
@@ -484,6 +485,41 @@
*/
List findAllConnections();
+/**
+ * Initiates a request to drop all FlowFiles in all connections under this
process group (recursively).
+ * This method returns a DropFlowFileStatus that can be used to determine
the current state of the request.
+ * Additionally, the DropFlowFileStatus provides a request identifier that
can then be
+ * passed to the {@link #getDropAllFlowFilesStatus(String)} and {@link
#cancelDropAllFlowFiles(String)}
+ * methods in order to obtain the status later or cancel a request
+ *
+ * @param requestIdentifier the identifier of the Drop FlowFile Request
+ * @param requestor the entity that is requesting that the FlowFiles be
dropped; this will be
+ *included in the Provenance Events that are generated.
+ *
+ * @return the status of the drop request, or null if there
is no
+ * connection in the process group.
+ */
+DropFlowFileStatus dropAllFlowFiles(String requestIdentifier, String
requestor);
+
+/**
+ * Returns the current status of a Drop AlL FlowFiles Request that was
initiated via the
Review comment:
Typo: All
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java
##
@@ -484,6 +485,41 @@
*/
List findAllConnections();
+/**
+ * Initiates a request to drop all FlowFiles in all connections under this
process group (recursively).
+ * This method returns a DropFlowFileStatus that can be used to determine
the current state of the request.
+ * Additionally, the DropFlowFileStatus provides a request identifier that
can then be
+ * passed to the {@link #getDropAllFlowFilesStatus(String)} and {@link
#cancelDropAllFlowFiles(String)}
+ * methods in order to obtain the status later or cancel a request
+ *
+ * @param requestIdentifier the identifier of the Drop FlowFile Request
+ * @param requestor the entity that is requesting that the FlowFiles be
dropped; this will be
+ *included in the Provenance Events that are generated.
+ *
+ * @return the status of the drop request, or null if there
is no
+ * connection in the process group.
+ */
+DropFlowFileStatus dropAllFlowFiles(String requestIdentifier, String
requestor);
+
+/**
+ * Returns the current status of a Drop AlL FlowFiles Request that was
initiated via the
+ * {@link #dropAllFlowFiles(String, String)} method with the given
identifier
+ *
+ * @param requestIdentifier the identifier of the Drop FlowFile Request
Review comment:
Drop All FlowFiles Request
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java
##
@@ -1260,6 +1273,87 @@ public Connection findConnection(final String id) {
return findAllConnections(this);
}
+@Override
+public DropFlowFileStatus dropAllFlowFiles(String requestIdentifier,
String requestor) {
+return handleDropAllFlowFiles(requestIdentifier, queue ->
queue.dropFlowFiles(requestIdentifier, requestor));
+}
+
+@Override
+public DropFlowFileStatus getDropAllFlowFilesStatus(String
requestIdentifier) {
+return handleDropAllFlowFiles(requestIdentifier, queue ->
queue.getDropFlowFileStatus(requestIdentifier));
+}
+
+@Override
+public DropFlowFileStatus cancelDropAllFlowFiles(String requestIdentifier)
{
+return handleDropAllFlowFiles(requestIdentifier, queue ->
queue.cancelDropFlowFileRequest(requestIdentifier));
+}
+
+private DropFlowFileStatus handleDropAllFlowFiles(String dropRequestId,
Function function) {
+DropFlowFileStatus resultDropFlowFileStatus;
+
+List connections = findAllConnections(this);
+
+DropFlowFileRequest aggregateDropFlowFileStatus = new
DropFlowFileRequest(dropRequestId);
+aggregateDropFlowFileStatus.setState(null);
+
+AtomicBoolean processedAtLeastOne = new AtomicBoolean(false);
+
+connections.stream()
+.map(Connection::getFlowFileQueue)
+.map(function::apply)
+.forEach(additionalDropFlowFileStatus -> {
+aggregate(aggregateDropFlowFileStatus,
additionalDropFlowFileStatus);
+processedAtLeastOne.set(true);
+});
+
+if (processedAtLeastOne.get()) {
+resultDropFlowFileStatus = aggregateDropFlowFileStatus;
+} else {
+resultDropFlowFileStatus = null;
+}
+
+return
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #859: MINIFICPP-1313 Do a multifile lookup in onSchedule
fgerlits commented on a change in pull request #859:
URL: https://github.com/apache/nifi-minifi-cpp/pull/859#discussion_r466425578
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -378,13 +380,13 @@ void TailFile::onSchedule(const
std::shared_ptr &context,
} else {
throw minifi::Exception(ExceptionType::PROCESSOR_EXCEPTION, "File to
tail must be a fully qualified file");
}
+
+recoverState(context);
}
std::string rolling_filename_pattern_glob;
context->getProperty(RollingFilenamePattern.getName(),
rolling_filename_pattern_glob);
rolling_filename_pattern_ =
utils::file::PathUtils::globToRegex(rolling_filename_pattern_glob);
Review comment:
I think it should be a separate PR, but yes, invalidating the state on
changes of key properties is not done well, and needs to be fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #859: MINIFICPP-1313 Do a multifile lookup in onSchedule
arpadboda commented on a change in pull request #859:
URL: https://github.com/apache/nifi-minifi-cpp/pull/859#discussion_r466420134
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -378,13 +380,13 @@ void TailFile::onSchedule(const
std::shared_ptr &context,
} else {
throw minifi::Exception(ExceptionType::PROCESSOR_EXCEPTION, "File to
tail must be a fully qualified file");
}
+
+recoverState(context);
}
std::string rolling_filename_pattern_glob;
context->getProperty(RollingFilenamePattern.getName(),
rolling_filename_pattern_glob);
rolling_filename_pattern_ =
utils::file::PathUtils::globToRegex(rolling_filename_pattern_glob);
Review comment:
Most probably unrelated to the current change, but it makes me wonder if
changing the value of this property should invalidate the state. I tend to say
yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #860: MINIFICPP-1314 Fix an intermittent test failure in ConsumeWindowsEventLogTests
arpadboda closed pull request #860: URL: https://github.com/apache/nifi-minifi-cpp/pull/860 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #850: MINIFICPP-1203 - Replace linter reported tabs with two spaces, remove few comments
arpadboda closed pull request #850: URL: https://github.com/apache/nifi-minifi-cpp/pull/850 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-1323) Encrypt sensitive properties of agent's configuration
Ferenc Gerlits created MINIFICPP-1323:
-
Summary: Encrypt sensitive properties of agent's configuration
Key: MINIFICPP-1323
URL: https://issues.apache.org/jira/browse/MINIFICPP-1323
Project: Apache NiFi MiNiFi C++
Issue Type: Improvement
Reporter: Ferenc Gerlits
Assignee: Ferenc Gerlits
Fix For: 0.9.0
Encrypt sensitive information like passwords in the {{minifi.properties}} file.
As the first step, encrypt {{nifi.security.client.pass.phrase}} and
{{nifi.rest.api.password}}, but allow the user to specify further properties to
be encrypted.
Also as an initial implementation, we can follow what NiFi does in its
{{encrypt-config.sh}} script: replace the sensitive properties with an
encrypted version, and place the encryption key in the {{bootstrap.conf}} file
(which should be secured using file system permissions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] jayaaditya edited a comment on pull request #4450: NIFI-7681 - Add update-bucket-policy command, add option to specify timeout and fix documentation to include previously implemented c
jayaaditya edited a comment on pull request #4450: URL: https://github.com/apache/nifi/pull/4450#issuecomment-669899024 @bbende Thanks for your quick review! I have incorporated the changes you had suggested. Could you please review again? Thanks once again! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] jayaaditya edited a comment on pull request #4450: NIFI-7681 - Add update-bucket-policy command, add option to specify timeout and fix documentation to include previously implemented c
jayaaditya edited a comment on pull request #4450: URL: https://github.com/apache/nifi/pull/4450#issuecomment-669899024 @bbende Thanks for your quick review! I have incorporated the changes you have suggested. Could you please review again? Thanks once again! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] jayaaditya commented on pull request #4450: NIFI-7681 - Add update-bucket-policy command, add option to specify timeout and fix documentation to include previously implemented commands
jayaaditya commented on pull request #4450: URL: https://github.com/apache/nifi/pull/4450#issuecomment-669899024 @bbende Thanks for your quick review! I have incorporated the changes you have suggested. Could you please review gain? Thanks once again! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] benkelly opened a new pull request #4456: Added IntelliJ generated `.ipr` to `.gitignore`
benkelly opened a new pull request #4456:
URL: https://github.com/apache/nifi/pull/4456
Description of PR
- IntelliJ generates a `.ipr` Project File when exec the `mvn` goals
`mvn idea:idea` and `mvn idea:project`
- [Maven `idea`
Doc](https://maven.apache.org/plugins/maven-idea-plugin/usage.html)
Signed-off-by: ben
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on pull request #857: MINIFICPP-1309 - RAII based resourceClaim management
adamdebreceni commented on pull request #857: URL: https://github.com/apache/nifi-minifi-cpp/pull/857#issuecomment-669821844 Indeed I don't remember seeing `RepoTests` fail, although on my fork's CI run it did not fail This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #848: MINIFICPP-1268 Fix compiler warnings on Windows (VS2017)
szaszm commented on a change in pull request #848:
URL: https://github.com/apache/nifi-minifi-cpp/pull/848#discussion_r466282492
##
File path: libminifi/include/io/CRCStream.h
##
@@ -224,7 +222,7 @@ template
int CRCStream::readData(uint8_t *buf, int buflen) {
int ret = child_stream_->read(buf, buflen);
if (ret > 0) {
-crc_ = crc32(crc_, buf, ret);
+crc_ = crc32(gsl::narrow(crc_), buf, ret);
Review comment:
Oh, I assumed it must be 32 bit, because it's crc32. Great, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #857: MINIFICPP-1309 - RAII based resourceClaim management
adamdebreceni commented on a change in pull request #857:
URL: https://github.com/apache/nifi-minifi-cpp/pull/857#discussion_r466275322
##
File path: libminifi/test/persistence-tests/PersistenceTests.cpp
##
@@ -257,8 +258,26 @@ TEST_CASE("Persisted flowFiles are updated on
modification", "[TestP1]") {
config->set(minifi::Configure::nifi_flowfile_repository_directory_default,
utils::file::FileUtils::concat_path(dir, "flowfile_repository"));
std::shared_ptr prov_repo =
std::make_shared();
- std::shared_ptr ff_repository =
std::make_shared("flowFileRepository");
- std::shared_ptr content_repo =
std::make_shared();
+ std::shared_ptr ff_repository;
+ std::shared_ptr content_repo;
+ SECTION("FlowFileRepository"){
+ff_repository =
std::make_shared("flowFileRepository");
+SECTION("VolatileContentRepository"){
+ content_repo =
std::make_shared();
+}
+SECTION("FileSystemContenRepository"){
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #857: MINIFICPP-1309 - RAII based resourceClaim management
adamdebreceni commented on a change in pull request #857:
URL: https://github.com/apache/nifi-minifi-cpp/pull/857#discussion_r466275440
##
File path: libminifi/include/ResourceClaim.h
##
@@ -47,6 +47,8 @@ extern void setDefaultDirectory(std::string);
// ResourceClaim Class
class ResourceClaim : public std::enable_shared_from_this {
Review comment:
removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on pull request #857: MINIFICPP-1309 - RAII based resourceClaim management
fgerlits commented on pull request #857: URL: https://github.com/apache/nifi-minifi-cpp/pull/857#issuecomment-669814958 Is the Windows (GitHub actions) test failure related to this change? It doesn't look like one of the usual flaky tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #861: MINIFICPP-1312 - Fix flaky FlowControllerTest
fgerlits commented on a change in pull request #861:
URL: https://github.com/apache/nifi-minifi-cpp/pull/861#discussion_r466255118
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -79,6 +79,17 @@ Flow Controller:
Remote Processing Groups:
)";
+template
+bool verifyWithBusyWait(std::chrono::milliseconds timeout, Fn&& fn) {
+ auto start = std::chrono::steady_clock::now();
+ while
(std::chrono::duration_cast(std::chrono::steady_clock::now()
- start) < timeout) {
Review comment:
minor, but I don't think you need the `duration_cast`: durations in
different units can be compared
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #861: MINIFICPP-1312 - Fix flaky FlowControllerTest
adamdebreceni commented on a change in pull request #861:
URL: https://github.com/apache/nifi-minifi-cpp/pull/861#discussion_r466250542
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -133,12 +144,9 @@ TEST_CASE("Flow shutdown waits for a while",
"[TestFlow2]") {
testController.startFlow();
// wait for the source processor to enqueue its flowFiles
- int tryCount = 0;
- while (tryCount++ < 10 && root->getTotalFlowFileCount() != 3) {
-std::this_thread::sleep_for(std::chrono::milliseconds{20});
- }
+ busy_wait(std::chrono::milliseconds{50}, [&] {return
root->getTotalFlowFileCount() != 0;});
Review comment:
changed it to `verifyWithBusyWait`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #861: MINIFICPP-1312 - Fix flaky FlowControllerTest
adamdebreceni commented on a change in pull request #861:
URL: https://github.com/apache/nifi-minifi-cpp/pull/861#discussion_r466186234
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -215,20 +220,19 @@ TEST_CASE("Extend the waiting period during shutdown",
"[TestFlow4]") {
testController.startFlow();
// wait for the source processor to enqueue its flowFiles
- int tryCount = 0;
- while (tryCount++ < 10 && root->getTotalFlowFileCount() != 3) {
-std::this_thread::sleep_for(std::chrono::milliseconds{20});
- }
+ busy_wait(std::chrono::milliseconds{50}, [&] {return
root->getTotalFlowFileCount() != 0;});
- REQUIRE(root->getTotalFlowFileCount() == 3);
+ REQUIRE(root->getTotalFlowFileCount() != 0);
REQUIRE(sourceProc->trigger_count.load() == 1);
std::thread shutdownThread([&]{
execSinkPromise.set_value();
controller->stop(true);
});
- while (controller->isRunning()) {
+ int extendCount = 0;
+ while (extendCount++ < 5 && controller->isRunning()) {
Review comment:
note that here we are not simply waiting for the controller to shutdown,
but we are verifying if we can progressively delay the shutdown by adjusting
the timeout parameter, shorter sleeps seem to be relatively more imprecise than
longer ones on the agents, I understand now that this might cause confusion,
will add some check to better convey its purpose
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #861: MINIFICPP-1312 - Fix flaky FlowControllerTest
adamdebreceni commented on a change in pull request #861:
URL: https://github.com/apache/nifi-minifi-cpp/pull/861#discussion_r466186234
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -215,20 +220,19 @@ TEST_CASE("Extend the waiting period during shutdown",
"[TestFlow4]") {
testController.startFlow();
// wait for the source processor to enqueue its flowFiles
- int tryCount = 0;
- while (tryCount++ < 10 && root->getTotalFlowFileCount() != 3) {
-std::this_thread::sleep_for(std::chrono::milliseconds{20});
- }
+ busy_wait(std::chrono::milliseconds{50}, [&] {return
root->getTotalFlowFileCount() != 0;});
- REQUIRE(root->getTotalFlowFileCount() == 3);
+ REQUIRE(root->getTotalFlowFileCount() != 0);
REQUIRE(sourceProc->trigger_count.load() == 1);
std::thread shutdownThread([&]{
execSinkPromise.set_value();
controller->stop(true);
});
- while (controller->isRunning()) {
+ int extendCount = 0;
+ while (extendCount++ < 5 && controller->isRunning()) {
Review comment:
note that here we are not simply waiting for the controller to shutdown,
but we are verifying if we can progressively delay the shutdown by adjusting
the timeout parameter, shorter sleeps seem to be relatively more imprecise than
longer ones on the agents, I understand now that this loop might not be the
best so I will unroll it to properly convey its purpose
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
