[GitHub] [nifi] exceptionfactory commented on pull request #6995: NIFI-7060 populate NiFi and NiFi Registry properties files using env vars in Docker Containers
exceptionfactory commented on PR #6995: URL: https://github.com/apache/nifi/pull/6995#issuecomment-1449258278 Thanks for the thoughtful reply @ChrisSamo632. For the simple things first, it seems like the following items could be pulled out to a simpler PR: - address shellcheck errors/warnings in the Docker Image start scripts - update maven-antrun-plugin for dockermaven builds The changes for NiFi Registry also seem self-contained, but might involve a little more review. Loading properties from files would be a helpful general improvement to properties loading. I could see that as either an incremental improvement, or part of a more generalized adjustment to property loading. The work around Sensitive Property Providers for loading values from secrets managers introduced some abstractions for NiFiProperties, which may need to be evaluated as part of the approach. One of the short-comings of the current Docker container is the need to define environment variables and map them to properties in the script. This requires manual updates to support additional properties. One the major advantages of moving environment variable loading into the application would be supporting convention-based mapping. This is a standard feature of Spring Boot, and provides a good pattern. With bootstrap.conf, it should be possible to select the property sources, perhaps the default is the current file properties, but the Docker image would set the value use convention-based environment variables. NiFi Registry is the only application component that uses Spring Boot, so I'm not suggesting we need to move in that direction, just that we should able to follow the same basic idea of convention-based environment variable mapping. Those are my initial thoughts. I'm sure others like @kevdoran have some ideas along these lines. With that, if you're interested in carving out some of these things, I think that would be useful, and we should put some thought into a new Jira issue that encompasses the goals of NIFI-7060. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] levilentz commented on pull request #6848: NIFI-5642 QueryCassandra processor : output FlowFiles as soon fetch_size is reached
levilentz commented on PR #6848: URL: https://github.com/apache/nifi/pull/6848#issuecomment-1449251032 Leaving a comment here so it doesn't go stale. Let me know if this needs any changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-6250) Add Kerberos authentication support to InvokeHTTP Processor
[ https://issues.apache.org/jira/browse/NIFI-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-6250: --- Affects Version/s: (was: 1.9.2) > Add Kerberos authentication support to InvokeHTTP Processor > --- > > Key: NIFI-6250 > URL: https://issues.apache.org/jira/browse/NIFI-6250 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > > InvokeHTTP should support authenticating using Kerberos. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-6250) Add Kerberos authentication support to InvokeHTTP Processor
[ https://issues.apache.org/jira/browse/NIFI-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-6250: -- Assignee: (was: Peter Wicks) > Add Kerberos authentication support to InvokeHTTP Processor > --- > > Key: NIFI-6250 > URL: https://issues.apache.org/jira/browse/NIFI-6250 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Peter Wicks >Priority: Minor > > InvokeHTTP should support authenticating using Kerberos. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on pull request #6990: NIFI-11205 Adding deprecation log for redundant Kerberos properties in Kafka processors
exceptionfactory commented on PR #6990: URL: https://github.com/apache/nifi/pull/6990#issuecomment-1448952310 Thanks for the reply @nandorsoma. The deprecation logs are a tool, sometimes suggesting alternatives, but sometimes no alternatives are available. The log message indicating an alternative is what caught my attention. Adding the DeprecationNotice to the 2_0 components would remove the need to log warnings for the properties on those components. That would make use of the current implementation for the 2_6 components. Will look for some additional feedback before moving in one direction or the other. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] nandorsoma commented on pull request #6990: NIFI-11205 Adding deprecation log for redundant Kerberos properties in Kafka processors
nandorsoma commented on PR #6990: URL: https://github.com/apache/nifi/pull/6990#issuecomment-1448936748 Thank you for the review, @exceptionfactory! I didn't know that the logger should reference the componentClass, but it is a good idea to pass it instead of the processorName. Regarding the new properties in the 2_0 processors... Maybe I misunderstood the concept of the deprecation logs. I didn’t know we needed to provide an alternative for the deprecated component on the same main version. Is it really a requirement? The message is unlucky because it suggests that, but with a little more detail, it would be clear. If it is unacceptable, I will favor removing 2_0 processors because I think it would be wrong to create a precedence to add a non-bugfix change to this branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1517: MINIFICPP-2048 ARM64 support part 1
szaszm commented on code in PR #1517: URL: https://github.com/apache/nifi-minifi-cpp/pull/1517#discussion_r1120777278 ## thirdparty/ossp-uuid/ossp-uuid-update-config-guess.patch: ## @@ -0,0 +1,5538 @@ +diff --git a/config.guess b/config.guess +index ca2a03c..69188da 100755 +--- a/config.guess b/config.guess Review Comment: Can we update the whole package instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11212) default schedule time in processor changed
[ https://issues.apache.org/jira/browse/NIFI-11212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694734#comment-17694734 ] Joe Witt commented on NIFI-11212: - Agree with [~mosermw]. That said I think we went a little overboard on a processor or two but i think they were already corrected. Bottom line here is for these processors we made by default run less often - we want people to think about what they need for their case because otherwise our defaults while fast...also are wasting resources in most cases. > default schedule time in processor changed > -- > > Key: NIFI-11212 > URL: https://issues.apache.org/jira/browse/NIFI-11212 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: naveen kumar saharan >Priority: Major > > We observed, for the processors like generate flowfile, lists3 the default > schedule time has changed from 0 sec to 1 min -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11212) default schedule time in processor changed
[ https://issues.apache.org/jira/browse/NIFI-11212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael W Moser resolved NIFI-11212. Resolution: Not A Bug > default schedule time in processor changed > -- > > Key: NIFI-11212 > URL: https://issues.apache.org/jira/browse/NIFI-11212 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: naveen kumar saharan >Priority: Major > > We observed, for the processors like generate flowfile, lists3 the default > schedule time has changed from 0 sec to 1 min -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11212) default schedule time in processor changed
[ https://issues.apache.org/jira/browse/NIFI-11212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694733#comment-17694733 ] Michael W Moser commented on NIFI-11212: This change was intentional, under NIFI-10628, for components that could cause high resource usage when Run Schedule is left at the old default 0 sec. > default schedule time in processor changed > -- > > Key: NIFI-11212 > URL: https://issues.apache.org/jira/browse/NIFI-11212 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: naveen kumar saharan >Priority: Major > > We observed, for the processors like generate flowfile, lists3 the default > schedule time has changed from 0 sec to 1 min -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1518: MINIFICPP-2057 Improve the performance of ListFile
szaszm commented on code in PR #1518:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1518#discussion_r1120710054
##
extensions/standard-processors/processors/ListFile.h:
##
@@ -81,19 +81,15 @@ class ListFile : public core::Processor {
private:
struct ListedFile : public utils::ListedObject {
[[nodiscard]] std::chrono::time_point
getLastModified() const override {
- return
std::chrono::time_point_cast(utils::file::FileUtils::to_sys(last_modified_time));
+ return last_modified_time;
}
[[nodiscard]] std::string getKey() const override {
- return absolute_path.string();
+ return full_file_path.string();
}
-std::filesystem::path filename;
-std::filesystem::path absolute_path;
-std::filesystem::file_time_type last_modified_time;
-std::filesystem::path relative_path;
+std::chrono::time_point last_modified_time;
Review Comment:
Does this change have any implications for reusing existing state from an
older minifi version before the change? In other words, does this change the
contents or the interpretation of the stored state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits opened a new pull request, #1518: MINIFICPP-2057 Improve the performance of ListFile
fgerlits opened a new pull request, #1518: URL: https://github.com/apache/nifi-minifi-cpp/pull/1518 On a sample test data of 20,000 files on Windows, `onTrigger` now takes ~2.5 sec instead of ~7.5 sec in the case when all 20,000 files have been processed already. The main changes are: * except for the file name (with path) and modification time, only query anything else (eg. file size) when and if we need it; * check first if the file is new, and only later if it matches the selection criteria; * instead of collecting the file names in a vector first and then processing them in a loop, we process the files directly in the `list_dir` callback (without this, the `onTrigger` duration was ~3.5 sec). NOTE: there are three changes in behavior: 1. When storing the list of files with the latest modification time, we used to store the path only, which doesn't uniquely identify the file. I'm pretty sure this was a bug, now we store path + filename, and there is a new unit test to verify that this works. 2. If we are not able to find the modification time of the file (apparently, this can happen on Windows), we used to skip the file. Now we list the file, with the modification time set to the epoch of the `system_clock` . 3. If the `Path Filter` is set, we used to list all files in the `Input Directory` root, plus all files in subdirectories the names of which match the filter. Now we only list files in matching subdirectories, only including the root if the filter regex includes it (eg., `"good_subdir|ugly_subdir|\."`). Also, `Path Filter` is ignored if `Recurse Subdirectories` is false -- in this case, all files in the Input Directory are listed, but none in the subdirectories. The new behavior matches the current behavior of NiFi; I think the old behavior matched the behavior of an earlier version of NiFi. --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6996: NIFI-11232 Fix buffer handling in ContentClaimInputStream
exceptionfactory commented on PR #6996: URL: https://github.com/apache/nifi/pull/6996#issuecomment-1448769027 > If that works I want to get the unit test into a state where it is mergeable. Thanks, adding the unit test to the PR would be very helpful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] Chrzi commented on pull request #6996: NIFI-11232 Fix buffer handling in ContentClaimInputStream
Chrzi commented on PR #6996: URL: https://github.com/apache/nifi/pull/6996#issuecomment-1448764306 I have confirmed the behaviour using the unit test provided in the Issue, which I also used to reproduce the bug. At the moment I am having issues executing `mvn clean install -P contrib-check` (the contrib-check fails) and just asked on Slack for help. If that works I want to get the unit test into a state where it is mergeable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11233) adjust Github runner configuration to use HTTP connection pooling
[
https://issues.apache.org/jira/browse/NIFI-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694718#comment-17694718
]
Paul Grey commented on NIFI-11233:
--
{noformat}
Error: Failed to execute goal on project nifi-hadoop-utils: Could not resolve
dependencies for project org.apache.nifi:nifi-hadoop-utils:jar:2.0.0-SNAPSHOT:
Failed to collect dependencies at org.apache.hadoop:hadoop-common:jar:3.3.4:
Failed to read artifact descriptor for
org.apache.hadoop:hadoop-common:jar:3.3.4: Could not transfer artifact
org.apache.hadoop:hadoop-common:pom:3.3.4 from/to central
(https://repo.maven.apache.org/maven2): Connection reset -> [Help 1]
{noformat}
> adjust Github runner configuration to use HTTP connection pooling
> -
>
> Key: NIFI-11233
> URL: https://issues.apache.org/jira/browse/NIFI-11233
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Paul Grey
>Assignee: Paul Grey
>Priority: Minor
>
> Multiple recent Github automation build failures indicate a problem
> downloading a POM for a dependency. Investigate adjustments to Github CI
> configuration to alleviate this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11233) adjust Github runner configuration to use HTTP connection pooling
Paul Grey created NIFI-11233: Summary: adjust Github runner configuration to use HTTP connection pooling Key: NIFI-11233 URL: https://issues.apache.org/jira/browse/NIFI-11233 Project: Apache NiFi Issue Type: Improvement Reporter: Paul Grey Multiple recent Github automation build failures indicate a problem downloading a POM for a dependency. Investigate adjustments to Github CI configuration to alleviate this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11233) adjust Github runner configuration to use HTTP connection pooling
[ https://issues.apache.org/jira/browse/NIFI-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Grey reassigned NIFI-11233: Assignee: Paul Grey > adjust Github runner configuration to use HTTP connection pooling > - > > Key: NIFI-11233 > URL: https://issues.apache.org/jira/browse/NIFI-11233 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > > Multiple recent Github automation build failures indicate a problem > downloading a POM for a dependency. Investigate adjustments to Github CI > configuration to alleviate this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] ChrisSamo632 commented on pull request #6995: NIFI-7060 populate NiFi and NiFi Registry properties files using env vars in Docker Containers
ChrisSamo632 commented on PR #6995: URL: https://github.com/apache/nifi/pull/6995#issuecomment-1448750600 @exceptionfactory I agree that there's utility in discussing some of the changes being made herein, it was the main driver for raising a Draft PR (rather than pushing through with more updates), so glad you noticed and thanks for commenting. I agree that an instance-based loading mechanism would make more sense and I'd been considering that as I took forward the (Docker Image-focussed, but not strictly limited) current env var approach, however I didn't want to go through making such changes without some discussion as I think there needs to be agreement on a common approach of how to load properties and what order of precedence is to be used (i.e. properties file vs. env var vs. defaults in property loader code, etc.) - this seems like it's worth a ticket in its own right (whether that's NIFI-7060 that's re-shaped *or* a new ticket to be raised). Other things that have been covered in this PR so far that we may/not want to retain: - [ ] address `shellcheck` errors/warnings in the Docker Image start scripts - [ ] load (sensitive) properties from files - do we keep this as a Docker Image-focussed approach or look to generalise this for other deployments too (I'm not sure how much sense it makes as likely any such files are just as readable as the `nifi/-registry.properties` files themselves on such systems, but in Docker/k8s the injection of a sensitive value as a file does make things a little more secure as the values can't be seen by a simple inspection of the running container's env) - [ ] update `maven-antrun-plugin` for `dockermaven` builds - [ ] refactor the Docker Image start scripts for `nifi-registry` - this fixed a few bugs/inconsistencies between the current `dockermaven` and `dockerhub` scripts currently on `main`, including: - missing `needClientAuth` settings - inability to configure the `database` for the Registry Flow Provider - missing LDAP `Referral Strategy` config - invalid OIDC properties (copied from `nifi`, so missing the `.registry` part of the property name *and* some properties are missing/present when unused) I'm not precious about keeping this PR open (after the discussion) and/or linking it to a different Jira ticket in future to cover the points we want to keep from this existing work vs. raising/reshaping remaining changes in NIFI-7060/a new ticket -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] Chrzi opened a new pull request, #6996: NIFI-11232 Fix buffer handling in ContentClaimInputStream
Chrzi opened a new pull request, #6996: URL: https://github.com/apache/nifi/pull/6996 # Summary [NIFI-11232](https://issues.apache.org/jira/browse/NIFI-11232) Under some circumstances the buffer is not recreated correctly in ContentClaimInputStream leading to a situation where data is read from the buffer and the buffer trying to read from a closed stream. More details in the issue. One case where this can happen is in a custom processor using Apache Tika. Passing the InputStream created of a flowfile by the processSession to Tika exposes the bug, since it uses the mark and reset feature extensively. This PR creates a new buffer when the stream is marked to buffer the readLimit (which is the primary cause why it was introduced in NIFI-10888). It also throws the buffer away during reset, if a reset within the buffer is not possible. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (NIFI-11232) FlowFileAccessException using ContentClaimInputStream
[
https://issues.apache.org/jira/browse/NIFI-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694679#comment-17694679
]
Christian Wahl edited comment on NIFI-11232 at 2/28/23 6:30 PM:
Sure here is a test case (added to TestContentClaimInputStream) that causes
this error in the Release commit of 1.20
I also added the whole file with that test case as an attachment.
{code:java}
@Test
public void test_NIFI_11232() throws IOException {
final InputStream original = new BufferedInputStream(new
ByteArrayInputStream(new byte[10_000]));
// use an offset greater than the first two reads
final ContentClaimInputStream in = new ContentClaimInputStream(repo,
contentClaim, 3L, original, new NopPerformanceTracker());
in.mark(1);
in.read();
in.reset();
in.mark(1);
in.read();
// in this reset the "original" stream is closed, but the buffer is still
there and using that one
in.reset();
// This fails because it tries to read from the buffer and the buffer from
the closed original stream.
// To force the buffer to read from the original stream a value greater
than the buffer size is required (8192)
in.readNBytes(9000);
} {code}
In the ContentClaimInputStream I identfied two problematic parts causing this
behaviour.
Line 176, bytesConsumed is only ever incremented and does not reflect the bytes
read since the last mark
{code:java}
if (bufferedIn != null && bytesConsumed <= markReadLimit)
{code}
Line 217-221, the currentOffset is first of all reset to the claim offset and
is therefore not usable to calculate the bytes read since the last mark and
currentOffset can be easily bigger than the markReadLimit (since the
markReadLimit is not the markOffset)
{code:java}
currentOffset = claimOffset;
if (markReadLimit > 0) {
final int limitLeft = (int) (markReadLimit - currentOffset);
if (limitLeft > 0) { {code}
was (Author: JIRAUSER299040):
Sure here is a test case (added to TestContentClaimInputStream) that causes
this error in the Release commit of 1.20
I also added the whole file with that test case as an attachment.
{code:java}
@Test
public void test_NIFI_11232() throws IOException {
final InputStream original = new BufferedInputStream(new
ByteArrayInputStream(new byte[10_000]));
// use an offset greater than the first two reads
final ContentClaimInputStream in = new ContentClaimInputStream(repo,
contentClaim, 3L, original, new NopPerformanceTracker());
in.mark(1);
in.read();
in.reset();
in.mark(1);
in.read();
// in this reset the "original" stream is closed, but the buffer is still
there and using that one
in.reset();
// this fails because it tries to read from the buffer
// and the buffer from the closed original stream
// to force the buffer to read from the original stream a value greater
than the buffer size is required (8192)
in.readNBytes(9000);
} {code}
In the ContentClaimInputStream I identfied two problematic parts causing this
behaviour.
Line 176, bytesConsumed is only ever incremented and does not reflect the bytes
read since the last mark
{code:java}
if (bufferedIn != null && bytesConsumed <= markReadLimit)
{code}
Line 217-221, the currentOffset is first of all reset to the claim offset and
is therefore not usable to calculate the bytes read since the last mark and
currentOffset can be easily bigger than the markReadLimit (since the
markReadLimit is not the markOffset)
{code:java}
currentOffset = claimOffset;
if (markReadLimit > 0) {
final int limitLeft = (int) (markReadLimit - currentOffset);
if (limitLeft > 0) { {code}
> FlowFileAccessException using ContentClaimInputStream
> -
>
> Key: NIFI-11232
> URL: https://issues.apache.org/jira/browse/NIFI-11232
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.20.0
>Reporter: Christian Wahl
>Priority: Major
> Attachments: TestContentClaimInputStream.java
>
>
> NIFI-10888 introduced a BufferedInputStream inside of the
> ContentClaimInputStream to speed up rewinding in small flow files (<1MB).
> Under some circumstances it can happen in reset that the delegate stream is
> closed and a new delegate stream is created, but the bufferedIn is not
> recreated with the new delegate.
> During the next read this leads to a situation where it tries to read from
> bufferedIn and bufferedIn in turn tries to read from the old and closed
> delegate stream causing an IOException or FlowFileAccessException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-11158) PutSalesforceObject processor improvements
[
https://issues.apache.org/jira/browse/NIFI-11158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Turcsanyi resolved NIFI-11158.
Fix Version/s: 2.0.0
Resolution: Fixed
> PutSalesforceObject processor improvements
> --
>
> Key: NIFI-11158
> URL: https://issues.apache.org/jira/browse/NIFI-11158
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Zsihovszki Krisztina
>Assignee: Zsihovszki Krisztina
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Found a few minor issues while checking PutSalesforceObject processor in the
> scope of NiFi 1.20 RC1 validation.
>
> * The session.penalize call is missing when flowfile is routed to FAILURE
> relationship.
> * It would be also useful if an "error.message" attribute was put on the
> outgoing flowfile containing info about the occurred error.
> * No provenance event is generated for successful put actions.
> * SeeAlso tags are missing (there is also a QuerySalesforceObject processor).
>
> I used an already fetched Salesforce record for testing. It contained null
> fields. When tried to upload that record (with a new name) to Salesforce, I
> got an NPE (flowfile was routed to FAILURE):
>
> {code:java}
> 2023-02-09 10:12:47,176 ERROR [Timer-Driven Process Thread-2]
> o.a.n.p.salesforce.PutSalesforceObject
> PutSalesforceObject[id=351e77fb-0186-1000-2c1f-d14e9aad775b] Failed to put
> records to Salesforce.
> java.lang.NullPointerException: null
> at java.base/java.util.Objects.requireNonNull(Objects.java:221)
> at
> java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178)
> at
> java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> org.apache.nifi.processors.salesforce.util.RecordExtender.getExtendedRecord(RecordExtender.java:71)
> at
> org.apache.nifi.processors.salesforce.PutSalesforceObject.onTrigger(PutSalesforceObject.java:164)
> {code}
> I believe it is not a common use case to upload a record with null value
> field but the code might be improved filter out/ignore null value fields and
> handle this situation without error.
>
> A short example to reproduce the above error with NPE (it's a BusinessBrand
> object).
> {code:java}
> {
> "Name": "=p'C!QWv",
> "OrgId": "S/}[R\"Tf",
> "ParentId": null
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11158) PutSalesforceObject processor improvements
[
https://issues.apache.org/jira/browse/NIFI-11158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694681#comment-17694681
]
ASF subversion and git services commented on NIFI-11158:
Commit 60c02225d5694efe8f6dab151aa441448fb215c7 in nifi's branch
refs/heads/main from krisztina-zsihovszki
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=60c02225d5 ]
NIFI-11158 PutSalesforceObject processor improvements
This closes #6959.
Reviewed-by: Lehel
Reviewed-by: Mark Bathori
Signed-off-by: Peter Turcsanyi
> PutSalesforceObject processor improvements
> --
>
> Key: NIFI-11158
> URL: https://issues.apache.org/jira/browse/NIFI-11158
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Zsihovszki Krisztina
>Assignee: Zsihovszki Krisztina
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Found a few minor issues while checking PutSalesforceObject processor in the
> scope of NiFi 1.20 RC1 validation.
>
> * The session.penalize call is missing when flowfile is routed to FAILURE
> relationship.
> * It would be also useful if an "error.message" attribute was put on the
> outgoing flowfile containing info about the occurred error.
> * No provenance event is generated for successful put actions.
> * SeeAlso tags are missing (there is also a QuerySalesforceObject processor).
>
> I used an already fetched Salesforce record for testing. It contained null
> fields. When tried to upload that record (with a new name) to Salesforce, I
> got an NPE (flowfile was routed to FAILURE):
>
> {code:java}
> 2023-02-09 10:12:47,176 ERROR [Timer-Driven Process Thread-2]
> o.a.n.p.salesforce.PutSalesforceObject
> PutSalesforceObject[id=351e77fb-0186-1000-2c1f-d14e9aad775b] Failed to put
> records to Salesforce.
> java.lang.NullPointerException: null
> at java.base/java.util.Objects.requireNonNull(Objects.java:221)
> at
> java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178)
> at
> java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> org.apache.nifi.processors.salesforce.util.RecordExtender.getExtendedRecord(RecordExtender.java:71)
> at
> org.apache.nifi.processors.salesforce.PutSalesforceObject.onTrigger(PutSalesforceObject.java:164)
> {code}
> I believe it is not a common use case to upload a record with null value
> field but the code might be improved filter out/ignore null value fields and
> handle this situation without error.
>
> A short example to reproduce the above error with NPE (it's a BusinessBrand
> object).
> {code:java}
> {
> "Name": "=p'C!QWv",
> "OrgId": "S/}[R\"Tf",
> "ParentId": null
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] asfgit closed pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
asfgit closed pull request #6959: NIFI-11158 PutSalesforceObject processor improvements URL: https://github.com/apache/nifi/pull/6959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ChrisSamo632 opened a new pull request, #6995: NIFI-7060 populate NiFi and NiFi Registry properties files using env vars in Docker Containers
ChrisSamo632 opened a new pull request, #6995: URL: https://github.com/apache/nifi/pull/6995 # Summary [NIFI-7060](https://issues.apache.org/jira/browse/NIFI-7060) populate NiFi and NiFi Registry properties files using env vars in Docker Containers Refreshes the original work by @yarox in #4008, rebased against latest `main` and extended to cover `apache/nifi-registry` along with `apache/nifi` images. - Allows for the specification of general env vars to cover any properties to be updated or added before instance startup. - Caters for existing env vars where they don't match the property names to be populated. - Allows for the injection of sensitive properties using files and referencing those via an env var, e.g. for passwords, sensitive keys or secrets - Fixes various inconsistencies between `dockermaven` and `dockerhub` scripts in NiFi and NiFi Registry (also tidies up some issues around OIDC env vars for Registry) TODO: - [ ] update documentation - [ ] build and test images for different env var use cases (e.g. mixed-case properties) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11232) FlowFileAccessException using ContentClaimInputStream
[ https://issues.apache.org/jira/browse/NIFI-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Wahl updated NIFI-11232: -- Attachment: TestContentClaimInputStream.java > FlowFileAccessException using ContentClaimInputStream > - > > Key: NIFI-11232 > URL: https://issues.apache.org/jira/browse/NIFI-11232 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.20.0 >Reporter: Christian Wahl >Priority: Major > Attachments: TestContentClaimInputStream.java > > > NIFI-10888 introduced a BufferedInputStream inside of the > ContentClaimInputStream to speed up rewinding in small flow files (<1MB). > Under some circumstances it can happen in reset that the delegate stream is > closed and a new delegate stream is created, but the bufferedIn is not > recreated with the new delegate. > During the next read this leads to a situation where it tries to read from > bufferedIn and bufferedIn in turn tries to read from the old and closed > delegate stream causing an IOException or FlowFileAccessException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11232) FlowFileAccessException using ContentClaimInputStream
[
https://issues.apache.org/jira/browse/NIFI-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694679#comment-17694679
]
Christian Wahl commented on NIFI-11232:
---
Sure here is a test case (added to TestContentClaimInputStream) that causes
this error in the Release commit of 1.20
I also added the whole file with that test case as an attachment.
{code:java}
@Test
public void test_NIFI_11232() throws IOException {
final InputStream original = new BufferedInputStream(new
ByteArrayInputStream(new byte[10_000]));
// use an offset greater than the first two reads
final ContentClaimInputStream in = new ContentClaimInputStream(repo,
contentClaim, 3L, original, new NopPerformanceTracker());
in.mark(1);
in.read();
in.reset();
in.mark(1);
in.read();
// in this reset the "original" stream is closed, but the buffer is still
there and using that one
in.reset();
// this fails because it tries to read from the buffer
// and the buffer from the closed original stream
// to force the buffer to read from the original stream a value greater
than the buffer size is required (8192)
in.readNBytes(9000);
} {code}
In the ContentClaimInputStream I identfied two problematic parts causing this
behaviour.
Line 176, bytesConsumed is only ever incremented and does not reflect the bytes
read since the last mark
{code:java}
if (bufferedIn != null && bytesConsumed <= markReadLimit)
{code}
Line 217-221, the currentOffset is first of all reset to the claim offset and
is therefore not usable to calculate the bytes read since the last mark and
currentOffset can be easily bigger than the markReadLimit (since the
markReadLimit is not the markOffset)
{code:java}
currentOffset = claimOffset;
if (markReadLimit > 0) {
final int limitLeft = (int) (markReadLimit - currentOffset);
if (limitLeft > 0) { {code}
> FlowFileAccessException using ContentClaimInputStream
> -
>
> Key: NIFI-11232
> URL: https://issues.apache.org/jira/browse/NIFI-11232
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.20.0
>Reporter: Christian Wahl
>Priority: Major
> Attachments: TestContentClaimInputStream.java
>
>
> NIFI-10888 introduced a BufferedInputStream inside of the
> ContentClaimInputStream to speed up rewinding in small flow files (<1MB).
> Under some circumstances it can happen in reset that the delegate stream is
> closed and a new delegate stream is created, but the bufferedIn is not
> recreated with the new delegate.
> During the next read this leads to a situation where it tries to read from
> bufferedIn and bufferedIn in turn tries to read from the old and closed
> delegate stream causing an IOException or FlowFileAccessException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
szaszm commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r1120572943
##
extensions/rocksdb-repos/FlowFileRepository.cpp:
##
@@ -200,6 +200,20 @@ bool FlowFileRepository::initialize(const
std::shared_ptr )
}
logger_->log_debug("NiFi FlowFile Repository Directory %s", directory_);
+ compaction_period_ = DEFAULT_COMPACTION_PERIOD;
+ if (auto compaction_period_str =
configure->get(Configure::nifi_flowfile_repository_compaction_period)) {
+if (auto compaction_period =
TimePeriodValue::fromString(compaction_period_str.value())) {
+ compaction_period_ = compaction_period->getMilliseconds();
+ if (compaction_period_.count() == 0) {
+logger_->log_warn("Setting '%s' to 0 disables forced compaction",
Configure::nifi_dbcontent_repository_compaction_period);
+ }
+} else {
+ logger_->log_error("Malformed property '%s', expected time period, using
default", Configure::nifi_flowfile_repository_compaction_period);
+}
+ } else {
+logger_->log_info("Using default compaction period");
Review Comment:
I think this can use a lower log level. It would also be nice to include the
value of this default.
```suggestion
logger_->log_debug("Using default compaction period of %" PRId64 " ms",
compaction_period_.count());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-7060) Replace properties in nifi.properties by environment variables
[
https://issues.apache.org/jira/browse/NIFI-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694675#comment-17694675
]
ASF subversion and git services commented on NIFI-7060:
---
Commit 6c1913af56471a8b762f42237860882ba4b2e445 in nifi's branch
refs/heads/NIFI-7060 from Chris Sampson
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6c1913af56 ]
NIFI-7060 update NiFi and NiFi Registry Docker Image start scripts to populate
properties from env vars
> Replace properties in nifi.properties by environment variables
> --
>
> Key: NIFI-7060
> URL: https://issues.apache.org/jira/browse/NIFI-7060
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Configuration, Docker
>Reporter: Adrian Martin
>Priority: Minor
> Labels: beginner, easyfix, features
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{start.sh}} replaces some properties defined in {{nifi.properties}} with a
> set of environment variables.
> This adds some flexibility as the users are able to modify properties without
> touching the {{nifi.properties}} file but it can lead to some frustration
> when users think they can replace any property by defining an environment
> variable with the same name. Also, when users want to provide their own
> {{nifi.properties}} file as some properties will be overwritten.
> I believe we should only replace a property when there is an environment
> variable defined with the same name (we should use the same names that appear
> in the {{nifi.properties}} file in uppercase and replacing any {{.}} with
> {{_}}). Defaults should be taken from the {{nifi.properties}} file.
> This way users can replace the properties file with a custom one, choose to
> define environment variables, or both.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] turcsanyip commented on pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
turcsanyip commented on PR #6959: URL: https://github.com/apache/nifi/pull/6959#issuecomment-1448653284 @krisztina-zsihovszki Thanks for the improvements! @mark-bathori @Lehel44 Thanks for your reviews! Merging to main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a diff in pull request #6769: NIFI-10955 - Added JASN1Reader the ability to try to adjust for unsupported ASN features
tpalfy commented on code in PR #6769: URL: https://github.com/apache/nifi/pull/6769#discussion_r1113403669 ## nifi-nar-bundles/nifi-asn1-bundle/nifi-asn1-services/src/main/java/org/apache/nifi/jasn1/JASN1Reader.java: ## @@ -134,17 +135,32 @@ public class JASN1Reader extends AbstractConfigurableComponent implements Record .required(false) .build(); +private static final PropertyDescriptor PREPROCESS_OUTPUT_DIRECTORY = new PropertyDescriptor.Builder() Review Comment: I'll add a "switch" property. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a diff in pull request #6769: NIFI-10955 - Added JASN1Reader the ability to try to adjust for unsupported ASN features
tpalfy commented on code in PR #6769:
URL: https://github.com/apache/nifi/pull/6769#discussion_r1113397901
##
nifi-nar-bundles/nifi-asn1-bundle/nifi-asn1-services/src/main/java/org/apache/nifi/jasn1/preprocess/NiFiASNPreprocessor.java:
##
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.jasn1.preprocess;
+
+import java.util.List;
+
+public interface NiFiASNPreprocessor {
Review Comment:
I'll remove the NiFi prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a diff in pull request #6769: NIFI-10955 - Added JASN1Reader the ability to try to adjust for unsupported ASN features
tpalfy commented on code in PR #6769:
URL: https://github.com/apache/nifi/pull/6769#discussion_r1120547354
##
nifi-nar-bundles/nifi-asn1-bundle/nifi-asn1-services/src/test/java/org/apache/nifi/jasn1/JASN1ReaderTest.java:
##
@@ -84,6 +84,7 @@ public void testCanLoadClassCompiledFromAsn() throws
Exception {
ConfigurationContext context = mock(ConfigurationContext.class,
RETURNS_DEEP_STUBS);
when(context.getProperty(ASN_FILES).isSet()).thenReturn(true);
when(context.getProperty(ASN_FILES).evaluateAttributeExpressions().getValue()).thenReturn(Paths.get("src",
"test", "resources", "test.asn").toString());
+//
when(context.getProperty(ASN_FILES).evaluateAttributeExpressions().getValue()).thenReturn(Paths.get("src",
"test", "resources", "test_error.asn").toString());
Review Comment:
It shouldn't be there at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a diff in pull request #6769: NIFI-10955 - Added JASN1Reader the ability to try to adjust for unsupported ASN features
tpalfy commented on code in PR #6769: URL: https://github.com/apache/nifi/pull/6769#discussion_r1120545224 ## nifi-nar-bundles/nifi-asn1-bundle/nifi-asn1-services/src/main/resources/docs/org.apache.nifi.jasn1.JASN1Reader/additionalDetails.html: ## @@ -122,5 +122,61 @@ Troubleshooting + +Additional Preprocessing + + +NiFi doesn't support every feature that the ASN standard allows. To alleviate problems when encountering ASN files with unsupported features, +NiFi can do additional preprocessing steps that creates modified versions of the provided ASN files, +removing unsupported features in a way that makes them less strict but otherwise should still be compatible with incoming data. +The original files will remain intact and new ones will be created with the same names in a directory set in the 'Additional Preprocessing Output Directory' property. +Please note that this is a best-effort attempt. It is also strongly recommended to compare the resulting ASN files to the originals and make sure they are still appropriate. + + +The following modification are applied: + + Review Comment: I can do that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a diff in pull request #6769: NIFI-10955 - Added JASN1Reader the ability to try to adjust for unsupported ASN features
tpalfy commented on code in PR #6769:
URL: https://github.com/apache/nifi/pull/6769#discussion_r1120544810
##
nifi-nar-bundles/nifi-asn1-bundle/nifi-asn1-services/src/test/java/org/apache/nifi/jasn1/preprocess/NiFiASNPreprocessorEngineTest.java:
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.jasn1.preprocess;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.List;
+import java.util.StringJoiner;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class NiFiASNPreprocessorEngineTest {
Review Comment:
In general I'm not a fan of complex tests. A test method - similar to
production ones - should do one thing and one thing only. If it fails, it
should be narrow enough to know where to look.
In this case due to possible unexpected cross-interaction between the
preprocessors it may have some additional value.
I'll add a very simple one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-11232) FlowFileAccessException using ContentClaimInputStream
[ https://issues.apache.org/jira/browse/NIFI-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694669#comment-17694669 ] Joe Witt commented on NIFI-11232: - [~cwahl] Can you share an example that exposes it by chance? The PR you mentioned you'd provide might well be enough for it to be obvious but a test that reproduces would be awesome. > FlowFileAccessException using ContentClaimInputStream > - > > Key: NIFI-11232 > URL: https://issues.apache.org/jira/browse/NIFI-11232 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.20.0 >Reporter: Christian Wahl >Priority: Major > > NIFI-10888 introduced a BufferedInputStream inside of the > ContentClaimInputStream to speed up rewinding in small flow files (<1MB). > Under some circumstances it can happen in reset that the delegate stream is > closed and a new delegate stream is created, but the bufferedIn is not > recreated with the new delegate. > During the next read this leads to a situation where it tries to read from > bufferedIn and bufferedIn in turn tries to read from the old and closed > delegate stream causing an IOException or FlowFileAccessException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11232) FlowFileAccessException using ContentClaimInputStream
Christian Wahl created NIFI-11232: - Summary: FlowFileAccessException using ContentClaimInputStream Key: NIFI-11232 URL: https://issues.apache.org/jira/browse/NIFI-11232 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.20.0 Reporter: Christian Wahl NIFI-10888 introduced a BufferedInputStream inside of the ContentClaimInputStream to speed up rewinding in small flow files (<1MB). Under some circumstances it can happen during reset that the delegate stream is closed and a new delegate stream is created, but the bufferedIn is not recreated with the new delegate. During the next read this leads to a situation where it tries to read from bufferedIn and bufferedIn in turn tries to read from the old and closed delegate stream causing an IOException or FlowFileAccessException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11232) FlowFileAccessException using ContentClaimInputStream
[ https://issues.apache.org/jira/browse/NIFI-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Wahl updated NIFI-11232: -- Description: NIFI-10888 introduced a BufferedInputStream inside of the ContentClaimInputStream to speed up rewinding in small flow files (<1MB). Under some circumstances it can happen in reset that the delegate stream is closed and a new delegate stream is created, but the bufferedIn is not recreated with the new delegate. During the next read this leads to a situation where it tries to read from bufferedIn and bufferedIn in turn tries to read from the old and closed delegate stream causing an IOException or FlowFileAccessException. was: NIFI-10888 introduced a BufferedInputStream inside of the ContentClaimInputStream to speed up rewinding in small flow files (<1MB). Under some circumstances it can happen during reset that the delegate stream is closed and a new delegate stream is created, but the bufferedIn is not recreated with the new delegate. During the next read this leads to a situation where it tries to read from bufferedIn and bufferedIn in turn tries to read from the old and closed delegate stream causing an IOException or FlowFileAccessException. > FlowFileAccessException using ContentClaimInputStream > - > > Key: NIFI-11232 > URL: https://issues.apache.org/jira/browse/NIFI-11232 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.20.0 >Reporter: Christian Wahl >Priority: Major > > NIFI-10888 introduced a BufferedInputStream inside of the > ContentClaimInputStream to speed up rewinding in small flow files (<1MB). > Under some circumstances it can happen in reset that the delegate stream is > closed and a new delegate stream is created, but the bufferedIn is not > recreated with the new delegate. > During the next read this leads to a situation where it tries to read from > bufferedIn and bufferedIn in turn tries to read from the old and closed > delegate stream causing an IOException or FlowFileAccessException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11161) Simplify Keyed Cipher Property Encryptor
[
https://issues.apache.org/jira/browse/NIFI-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694647#comment-17694647
]
ASF subversion and git services commented on NIFI-11161:
Commit 977310584195a59401ef7472cf01a677be8cd465 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9773105841 ]
NIFI-11161 Simplified KeyedCipherPropertyEncryptor
- Replaced KeyedCipherProvider references with direct Cipher instantiation
This closes #6939
Signed-off-by: Paul Grey
> Simplify Keyed Cipher Property Encryptor
>
>
> Key: NIFI-11161
> URL: https://issues.apache.org/jira/browse/NIFI-11161
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The {{KeyedCipherPropertyEncryptor}} performs encryption and decryption of
> sensitive property values using a Secret Key and AES-GCM. The current
> implementation depends on the {{AESKeyedCipherProvider}} from
> {{{}nifi-security-utils{}}}. All instances of the Property Encryptor use the
> same AES-GCM Cipher Algorithm, so the arguments checks implemented in
> {{AESKeyedCipherProvider}} are not necessary for standard operation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] greyp9 closed pull request #6939: NIFI-11161 Simplify KeyedCipherPropertyEncryptor
greyp9 closed pull request #6939: NIFI-11161 Simplify KeyedCipherPropertyEncryptor URL: https://github.com/apache/nifi/pull/6939 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11210) Missing network read timeout in DistributedMapCacheClientService
[ https://issues.apache.org/jira/browse/NIFI-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Grey updated NIFI-11210: - Status: Patch Available (was: In Progress) > Missing network read timeout in DistributedMapCacheClientService > > > Key: NIFI-11210 > URL: https://issues.apache.org/jira/browse/NIFI-11210 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Noted in Apache/Slack thread: > https://apachenifi.slack.com/archives/C0L9VCD47/p1676907328887569?thread_ts=1671449357.103949=C0L9VCD47 > If a client / server version handshake fails due to a network glitch, the > DistributedMapCacheClientService can be left in a state where it stops > processing new requests. > Remediate by ensuring that network connection construction sets appropriate > timeouts. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] greyp9 opened a new pull request, #6994: NIFI-11210 - add network timeout to distributed cache client
greyp9 opened a new pull request, #6994: URL: https://github.com/apache/nifi/pull/6994 ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 11 - [x] JDK 17 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
Lehel44 commented on PR #6959: URL: https://github.com/apache/nifi/pull/6959#issuecomment-1448470323 Thanks for the changes @krisztina-zsihovszki. LGTM+1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
Lehel44 commented on code in PR #6959:
URL: https://github.com/apache/nifi/pull/6959#discussion_r1120139647
##
nifi-nar-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/main/java/org/apache/nifi/processors/salesforce/util/SalesforceRestService.java:
##
@@ -103,6 +103,10 @@ public InputStream postRecord(String sObjectApiName,
String body) {
return request(request);
}
+public String getVersionedBaseUrl() {
+return baseUrl + "/services/data/v" + version;
+}
+
private InputStream request(Request request) {
Review Comment:
I recommend refactoring this method a little bit.
```private InputStream request(Request request) {
try (Response response = httpClient.newCall(request).execute()) {
if (response.code() < 200 || response.code() > 201) {
throw new ProcessException("Invalid response" +
" Code: " + response.code() +
" Message: " + response.message() +
" Body: " + (response.body() == null ? null :
response.body().string())
);
}
return Objects.requireNonNull(response.body()).byteStream();
} catch(ProcessException e) {
throw e;
} catch (Exception e) {
throw new ProcessException(String.format("Salesforce HTTP
request failed [%s]", request.url()), e);
}
}```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] slambrose opened a new pull request, #6993: NIFI-11231 Stateless NiFi secure parameter context support
slambrose opened a new pull request, #6993: URL: https://github.com/apache/nifi/pull/6993 # Summary [NIFI-11231](https://issues.apache.org/jira/browse/NIFI-11231) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-11231) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X ] Pull Request based on current revision of the `main` branch - [X ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 11 - [X] JDK 17 ### Licensing - [X] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - N/A - [X] New dependencies are documented in applicable `LICENSE` and `NOTICE` files - N/A ### Documentation - [X] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11231) Stateless Nifi Supports Secure Parameter Context Variables
[ https://issues.apache.org/jira/browse/NIFI-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dye357 updated NIFI-11231: -- Summary: Stateless Nifi Supports Secure Parameter Context Variables (was: Stateless Nifi Supports Secure Parameter Contexts) > Stateless Nifi Supports Secure Parameter Context Variables > -- > > Key: NIFI-11231 > URL: https://issues.apache.org/jira/browse/NIFI-11231 > Project: Apache NiFi > Issue Type: Bug > Components: NiFi Stateless >Affects Versions: 1.20.0 >Reporter: Dye357 >Priority: Major > Fix For: 1.21.0 > > > Currently Secure Parameter Context Variables are not supported in Stateless > Nifi - this is due to StatelessFlowManager being stubbed to throw a "Not > Implemented" exception when providing a process group configured with a > secure Parameter Context variable. > Desired functionality: Secure Parameter Context Variables can be used with > stateless Nifi. If a process group contains a Secure Parameter Context > Variable the context is used to hydrate all parameters within the provided > flow including any marked sensitive. > Acceptance Criteria: Process Groups using secure parameters can be used in > stateless Nifi. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11231) Stateless Nifi Supports Secure Parameter Contexts
Dye357 created NIFI-11231: - Summary: Stateless Nifi Supports Secure Parameter Contexts Key: NIFI-11231 URL: https://issues.apache.org/jira/browse/NIFI-11231 Project: Apache NiFi Issue Type: Bug Components: NiFi Stateless Affects Versions: 1.20.0 Reporter: Dye357 Fix For: 1.21.0 Currently Secure Parameter Context Variables are not supported in Stateless Nifi - this is due to StatelessFlowManager being stubbed to throw a "Not Implemented" exception when providing a process group configured with a secure Parameter Context variable. Desired functionality: Secure Parameter Context Variables can be used with stateless Nifi. If a process group contains a Secure Parameter Context Variable the context is used to hydrate all parameters within the provided flow including any marked sensitive. Acceptance Criteria: Process Groups using secure parameters can be used in stateless Nifi. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6990: NIFI-11205 Adding deprecation log for redundant Kerberos properties in Kafka processors
exceptionfactory commented on code in PR #6990:
URL: https://github.com/apache/nifi/pull/6990#discussion_r1120240266
##
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-shared/src/main/java/org/apache/nifi/kafka/shared/validation/KafkaDeprecationValidator.java:
##
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.kafka.shared.validation;
+
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.deprecation.log.DeprecationLogger;
+import org.apache.nifi.deprecation.log.DeprecationLoggerFactory;
+
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.KERBEROS_CREDENTIALS_SERVICE;
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.KERBEROS_KEYTAB;
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.KERBEROS_PRINCIPAL;
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.SELF_CONTAINED_KERBEROS_USER_SERVICE;
+
+public class KafkaDeprecationValidator {
+
+private static final DeprecationLogger deprecationLogger =
DeprecationLoggerFactory.getLogger(KafkaDeprecationValidator.class);
Review Comment:
The deprecation logs should reference the component class, instead of the
`KafkaDeprecationValidator`. This could be implemented by passing the Processor
`Class` reference to the `validate` method, and then getting the
`DeprecationLogger` instance in the method.
##
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-shared/src/main/java/org/apache/nifi/kafka/shared/validation/KafkaDeprecationValidator.java:
##
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.kafka.shared.validation;
+
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.deprecation.log.DeprecationLogger;
+import org.apache.nifi.deprecation.log.DeprecationLoggerFactory;
+
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.KERBEROS_CREDENTIALS_SERVICE;
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.KERBEROS_KEYTAB;
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.KERBEROS_PRINCIPAL;
+import static
org.apache.nifi.kafka.shared.component.KafkaClientComponent.SELF_CONTAINED_KERBEROS_USER_SERVICE;
+
+public class KafkaDeprecationValidator {
+
+private static final DeprecationLogger deprecationLogger =
DeprecationLoggerFactory.getLogger(KafkaDeprecationValidator.class);
+
+public static void validate(String processorName, final String identifier,
final ValidationContext validationContext) {
Review Comment:
Recommend changing the signature to use `Class componentClass` instead of
`String processorName` to support mapping the deprecation warnings to the
component more closely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1515: MINIFICPP-2056 CWEL: support UserData in simple/flattened json
szaszm commented on code in PR #1515:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1515#discussion_r1120199726
##
extensions/windows-event-log/wel/JSONUtils.cpp:
##
@@ -162,9 +186,4 @@ std::string jsonToString(const rapidjson::Document& doc) {
doc.Accept(writer);
return buffer.GetString();
}
-
-} // namespace wel
-} // namespace minifi
-} // namespace nifi
-} // namespace apache
-} // namespace org
+} // namespace org::apache::nifi::minifi::wel
Review Comment:
thanks, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1515: MINIFICPP-2056 CWEL: support UserData in simple/flattened json
martinzink commented on code in PR #1515:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1515#discussion_r1120173953
##
extensions/windows-event-log/wel/JSONUtils.cpp:
##
@@ -162,9 +186,4 @@ std::string jsonToString(const rapidjson::Document& doc) {
doc.Accept(writer);
return buffer.GetString();
}
-
-} // namespace wel
-} // namespace minifi
-} // namespace nifi
-} // namespace apache
-} // namespace org
+} // namespace org::apache::nifi::minifi::wel
Review Comment:
I think the linter failure comes from here `JSONUtils.cpp:189: At least two
spaces is best between code and comments [whitespace/comments] [2]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
Lehel44 commented on code in PR #6959:
URL: https://github.com/apache/nifi/pull/6959#discussion_r1120139647
##
nifi-nar-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/main/java/org/apache/nifi/processors/salesforce/util/SalesforceRestService.java:
##
@@ -103,6 +103,10 @@ public InputStream postRecord(String sObjectApiName,
String body) {
return request(request);
}
+public String getVersionedBaseUrl() {
+return baseUrl + "/services/data/v" + version;
+}
+
private InputStream request(Request request) {
Review Comment:
I recommend refactoring this method a little bit.
```private InputStream request(Request request) {
try (Response response = httpClient.newCall(request).execute()) {
if (response.code() < 200 || response.code() > 201) {
throw new ProcessException("Invalid response" +
" Code: " + response.code() +
" Message: " + response.message() +
" Body: " + (response.body() == null ? null :
response.body().string())
);
}
return Objects.requireNonNull(response.body()).byteStream();
} catch(ProcessException e) {
throw e;
} catch (Exception e) {
throw new ProcessException(String.format("Salesforce HTTP
request failed [%s]", request.url()), e);
}
}```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
Lehel44 commented on code in PR #6959:
URL: https://github.com/apache/nifi/pull/6959#discussion_r1120139647
##
nifi-nar-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/main/java/org/apache/nifi/processors/salesforce/util/SalesforceRestService.java:
##
@@ -103,6 +103,10 @@ public InputStream postRecord(String sObjectApiName,
String body) {
return request(request);
}
+public String getVersionedBaseUrl() {
+return baseUrl + "/services/data/v" + version;
+}
+
private InputStream request(Request request) {
Review Comment:
I recommend refactoring this method a little bit.
`private InputStream request(Request request) {
try (Response response = httpClient.newCall(request).execute()) {
if (response.code() < 200 || response.code() > 201) {
throw new ProcessException("Invalid response" +
" Code: " + response.code() +
" Message: " + response.message() +
" Body: " + (response.body() == null ? null :
response.body().string())
);
}
return Objects.requireNonNull(response.body()).byteStream();
} catch(ProcessException e) {
throw e;
} catch (Exception e) {
throw new ProcessException(String.format("Salesforce HTTP
request failed [%s]", request.url()), e);
}
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
Lehel44 commented on code in PR #6959:
URL: https://github.com/apache/nifi/pull/6959#discussion_r1120120932
##
nifi-nar-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/main/java/org/apache/nifi/processors/salesforce/PutSalesforceObject.java:
##
@@ -170,24 +177,30 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
out.reset();
}
}
-
if (writer.isActiveRecordSet()) {
processRecords(objectType, out, writer, extender);
}
-session.transfer(flowFile, REL_SUCCESS);
-
+ }
+ session.transfer(flowFile, REL_SUCCESS);
+ final long transferMillis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
Review Comment:
The whole bundle does not use the final keyword. Please remove it for
consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
Lehel44 commented on code in PR #6959: URL: https://github.com/apache/nifi/pull/6959#discussion_r1120117231 ## nifi-mock/src/main/java/org/apache/nifi/util/TestRunner.java: ## @@ -1062,4 +1063,6 @@ void assertAttributes( */ void setRunSchedule(long runSchedule); +void assertProvenanceEvent(ProvenanceEventType eventType); Review Comment: Please add Javadoc similar to the other methods. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6959: NIFI-11158 PutSalesforceObject processor improvements
Lehel44 commented on code in PR #6959: URL: https://github.com/apache/nifi/pull/6959#discussion_r1120116567 ## nifi-mock/src/main/java/org/apache/nifi/util/StandardProcessorTestRunner.java: ## @@ -1056,4 +1058,12 @@ public void assertAllConditionsMet(final Relationship relationship, Predicate
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1517: MINIFICPP-2048 ARM64 support part 1
martinzink commented on code in PR #1517:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1517#discussion_r1120105930
##
libminifi/test/unit/StringUtilsTests.cpp:
##
@@ -388,30 +390,22 @@ TEST_CASE("TestStringUtils::testBase64Decode", "[test
base64 decode]") {
REQUIRE("oo" == StringUtils::from_base64("b29vb29v", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("-_-_", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("+/+/", as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",
as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_",
as_string));
+ std::string expected{
+ static_cast(0), static_cast(16), static_cast(-125),
static_cast(16),
+ static_cast(81), static_cast(-121), static_cast(32),
static_cast(-110),
+ static_cast(-117), static_cast(48), static_cast(-45),
static_cast(-113),
+ static_cast(65), static_cast(20), static_cast(-109),
static_cast(81),
+ static_cast(85), static_cast(-105), static_cast(97),
static_cast(-106),
+ static_cast(-101), static_cast(113), static_cast(-41),
static_cast(-97),
+ static_cast(-126), static_cast(24), static_cast(-93),
static_cast(-110),
+ static_cast(89), static_cast(-89), static_cast(-94),
static_cast(-102),
+ static_cast(-85), static_cast(-78), static_cast(-37),
static_cast(-81),
+ static_cast(-61), static_cast(28), static_cast(-77),
static_cast(-45),
+ static_cast(93), static_cast(-73), static_cast(-29),
static_cast(-98),
+ static_cast(-69), static_cast(-13), static_cast(-33),
static_cast(-65)};
Review Comment:
Good idea, unfortunetly this version fails for some reason(it creates an
empty string) on my machine. But I've created [something
similar](https://github.com/apache/nifi-minifi-cpp/pull/1517/commits/8526c237224d7f9a67d88bc0fca6428e1e873083)
based on your idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1517: MINIFICPP-2048 ARM64 support part 1
martinzink commented on code in PR #1517:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1517#discussion_r1120105778
##
thirdparty/ossp-uuid/ossp-uuid-update-config-guess.patch:
##
@@ -0,0 +1,5538 @@
+diff --git a/config.guess b/config.guess
+index ca2a03c..69188da 100755
+--- a/config.guess
b/config.guess
Review Comment:
They are the newer version of the included config.guess and config.sub
https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess
https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub
##
libminifi/test/unit/StringUtilsTests.cpp:
##
@@ -388,30 +390,22 @@ TEST_CASE("TestStringUtils::testBase64Decode", "[test
base64 decode]") {
REQUIRE("oo" == StringUtils::from_base64("b29vb29v", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("-_-_", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("+/+/", as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",
as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_",
as_string));
+ std::string expected{
+ static_cast(0), static_cast(16), static_cast(-125),
static_cast(16),
+ static_cast(81), static_cast(-121), static_cast(32),
static_cast(-110),
+ static_cast(-117), static_cast(48), static_cast(-45),
static_cast(-113),
+ static_cast(65), static_cast(20), static_cast(-109),
static_cast(81),
+ static_cast(85), static_cast(-105), static_cast(97),
static_cast(-106),
+ static_cast(-101), static_cast(113), static_cast(-41),
static_cast(-97),
+ static_cast(-126), static_cast(24), static_cast(-93),
static_cast(-110),
+ static_cast(89), static_cast(-89), static_cast(-94),
static_cast(-102),
+ static_cast(-85), static_cast(-78), static_cast(-37),
static_cast(-81),
+ static_cast(-61), static_cast(28), static_cast(-77),
static_cast(-45),
+ static_cast(93), static_cast(-73), static_cast(-29),
static_cast(-98),
+ static_cast(-69), static_cast(-13), static_cast(-33),
static_cast(-65)};
Review Comment:
Good idea, unfortunetly this version fails for some reason on my machine.
But I've created [something
similar](https://github.com/apache/nifi-minifi-cpp/pull/1517/commits/8526c237224d7f9a67d88bc0fca6428e1e873083)
based on your idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1517: MINIFICPP-2048 ARM64 support part 1
szaszm commented on code in PR #1517:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1517#discussion_r1120058939
##
libminifi/test/unit/StringUtilsTests.cpp:
##
@@ -388,30 +390,22 @@ TEST_CASE("TestStringUtils::testBase64Decode", "[test
base64 decode]") {
REQUIRE("oo" == StringUtils::from_base64("b29vb29v", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("-_-_", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("+/+/", as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",
as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_",
as_string));
+ std::string expected{
+ static_cast(0), static_cast(16), static_cast(-125),
static_cast(16),
+ static_cast(81), static_cast(-121), static_cast(32),
static_cast(-110),
+ static_cast(-117), static_cast(48), static_cast(-45),
static_cast(-113),
+ static_cast(65), static_cast(20), static_cast(-109),
static_cast(81),
+ static_cast(85), static_cast(-105), static_cast(97),
static_cast(-106),
+ static_cast(-101), static_cast(113), static_cast(-41),
static_cast(-97),
+ static_cast(-126), static_cast(24), static_cast(-93),
static_cast(-110),
+ static_cast(89), static_cast(-89), static_cast(-94),
static_cast(-102),
+ static_cast(-85), static_cast(-78), static_cast(-37),
static_cast(-81),
+ static_cast(-61), static_cast(28), static_cast(-77),
static_cast(-45),
+ static_cast(93), static_cast(-73), static_cast(-29),
static_cast(-98),
+ static_cast(-69), static_cast(-13), static_cast(-33),
static_cast(-65)};
Review Comment:
I built the literal with this: https://godbolt.org/z/6o66M18os
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1517: MINIFICPP-2048 ARM64 support part 1
szaszm commented on code in PR #1517: URL: https://github.com/apache/nifi-minifi-cpp/pull/1517#discussion_r1120056512 ## thirdparty/ossp-uuid/ossp-uuid-update-config-guess.patch: ## @@ -0,0 +1,5538 @@ +diff --git a/config.guess b/config.guess +index ca2a03c..69188da 100755 +--- a/config.guess b/config.guess Review Comment: Where is this huge patch coming from? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1517: MINIFICPP-2048 ARM64 support part 1
szaszm commented on code in PR #1517:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1517#discussion_r1120055690
##
libminifi/test/unit/StringUtilsTests.cpp:
##
@@ -388,30 +390,22 @@ TEST_CASE("TestStringUtils::testBase64Decode", "[test
base64 decode]") {
REQUIRE("oo" == StringUtils::from_base64("b29vb29v", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("-_-_", as_string));
REQUIRE("\xfb\xff\xbf" == StringUtils::from_base64("+/+/", as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",
as_string));
- REQUIRE(std::string({ 0, 16, -125, 16,
- 81, -121, 32, -110,
- -117, 48, -45, -113,
- 65, 20, -109, 81,
- 85, -105, 97, -106,
- -101, 113, -41, -97,
- -126, 24, -93, -110,
- 89, -89, -94, -102,
--85, -78, -37, -81,
--61, 28, -77, -45,
- 93, -73, -29, -98,
--69, -13, -33, -65}) ==
StringUtils::from_base64("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_",
as_string));
+ std::string expected{
+ static_cast(0), static_cast(16), static_cast(-125),
static_cast(16),
+ static_cast(81), static_cast(-121), static_cast(32),
static_cast(-110),
+ static_cast(-117), static_cast(48), static_cast(-45),
static_cast(-113),
+ static_cast(65), static_cast(20), static_cast(-109),
static_cast(81),
+ static_cast(85), static_cast(-105), static_cast(97),
static_cast(-106),
+ static_cast(-101), static_cast(113), static_cast(-41),
static_cast(-97),
+ static_cast(-126), static_cast(24), static_cast(-93),
static_cast(-110),
+ static_cast(89), static_cast(-89), static_cast(-94),
static_cast(-102),
+ static_cast(-85), static_cast(-78), static_cast(-37),
static_cast(-81),
+ static_cast(-61), static_cast(28), static_cast(-77),
static_cast(-45),
+ static_cast(93), static_cast(-73), static_cast(-29),
static_cast(-98),
+ static_cast(-69), static_cast(-13), static_cast(-33),
static_cast(-65)};
Review Comment:
Can we represent the same in a more compact way, like this?
```suggestion
std::string expected =
"\x00\x10\x83\x10"
"\x51\x87\x20\x92"
"\x8b\x30\xd3\x8f"
"\x41\x14\x93\x51"
"\x55\x97\x61\x96"
"\x9b\x71\xd7\x9f"
"\x82\x18\xa3\x92"
"\x59\xa7\xa2\x9a"
"\xab\xb2\xdb\xaf"
"\xc3\x1c\xb3\xd3"
"\x5d\xb7\xe3\x9e"
"\xbb\xf3\xdf\xbf";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2058) AWS extension fails to load on ARM64
[
https://issues.apache.org/jira/browse/MINIFICPP-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-2058:
---
Description:
With the [MINIFICPP-2048 ARM64 support part
1|https://github.com/apache/nifi-minifi-cpp/pull/1517] included the AWS
extension builds but the extension fails to load due to missing symbol
{code:java}
Failed to load extension 'minifi-aws' ... aws_cpu_has_feature: symbol not found
{code}
was:
With the [https://github.com/apache/nifi-minifi-cpp/pull/1517] included the AWS
extension builds but the extension fails to load due to missing symbol
{code:java}
Failed to load extension 'minifi-aws' ... aws_cpu_has_feature: symbol not found
{code}
> AWS extension fails to load on ARM64
>
>
> Key: MINIFICPP-2058
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2058
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>
> With the [MINIFICPP-2048 ARM64 support part
> 1|https://github.com/apache/nifi-minifi-cpp/pull/1517] included the AWS
> extension builds but the extension fails to load due to missing symbol
>
> {code:java}
> Failed to load extension 'minifi-aws' ... aws_cpu_has_feature: symbol not
> found
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (MINIFICPP-2058) AWS extension fails to load on ARM64
Martin Zink created MINIFICPP-2058:
--
Summary: AWS extension fails to load on ARM64
Key: MINIFICPP-2058
URL: https://issues.apache.org/jira/browse/MINIFICPP-2058
Project: Apache NiFi MiNiFi C++
Issue Type: Bug
Reporter: Martin Zink
Assignee: Martin Zink
With the [https://github.com/apache/nifi-minifi-cpp/pull/1517] included the AWS
extension builds but the extension fails to load due to missing symbol
{code:java}
Failed to load extension 'minifi-aws' ... aws_cpu_has_feature: symbol not found
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2049) Google cloud extension compile failure on ARM
[ https://issues.apache.org/jira/browse/MINIFICPP-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2049: --- Status: Patch Available (was: Open) https://github.com/apache/nifi-minifi-cpp/pull/1517 > Google cloud extension compile failure on ARM > - > > Key: MINIFICPP-2049 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2049 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > The GCP extension doesnt compile on ARM systems(tested on linux) due to bug > in the currently used crc32c 1.1.1. release. > It seems like this issue is already fixed in the 1.1.2 release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2050) AWS tests compile failure on ARM
[ https://issues.apache.org/jira/browse/MINIFICPP-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2050: --- Status: Patch Available (was: Open) https://github.com/apache/nifi-minifi-cpp/pull/1517 > AWS tests compile failure on ARM > > > Key: MINIFICPP-2050 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2050 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > Some linking issue only on ARM. (linux) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2052) Ambigous signed/unsigned chars
[
https://issues.apache.org/jira/browse/MINIFICPP-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-2052:
---
Status: Patch Available (was: Open)
https://github.com/apache/nifi-minifi-cpp/pull/1517
> Ambigous signed/unsigned chars
> --
>
> Key: MINIFICPP-2052
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2052
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>
> We currently rely on that the char type is signed. (this is usually the case
> on x86_64 systems, and on apple silicon aswell)
> But this is not guarenteed, since the standard does not specify if plain
> {{char}} is signed or unsigned.
> This can cause various issues on ARM systems where the char is unsigned by
> default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2051) Thirdparty ossp-uuid compile failure on ARM
[ https://issues.apache.org/jira/browse/MINIFICPP-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2051: --- Status: Patch Available (was: Open) https://github.com/apache/nifi-minifi-cpp/pull/1517 > Thirdparty ossp-uuid compile failure on ARM > --- > > Key: MINIFICPP-2051 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2051 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > Due to outdated config.guess -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2047) reverse DNS lookup
[ https://issues.apache.org/jira/browse/MINIFICPP-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2047: --- Resolution: Fixed Status: Resolved (was: Patch Available) https://github.com/apache/nifi-minifi-cpp/commit/c4c80e63012fa92c00c7a4f3b5ea383f1ffa55d3 > reverse DNS lookup > -- > > Key: MINIFICPP-2047 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2047 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > We should add this feature so it can improve the readability of the listening > processors such as ListenSyslog, ListenUDP, ListenTCP. > We could add the feature as a standalone processor, include this in the > relevant processors or add an Expression Langauge function. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
szaszm commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r1120017294
##
libminifi/test/rocksdb-tests/DBContentRepositoryTests.cpp:
##
@@ -159,7 +159,7 @@ TEST_CASE("Delete NonExistent Claim", "[TestDBCR4]") {
stream->close();
- content_repo->stop();
+ content_repo->deinitialize();
Review Comment:
Let's do that, or add a comment that deinitialize is a test-only function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1887) Add default Connection size limits
[ https://issues.apache.org/jira/browse/MINIFICPP-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-1887: --- Resolution: Fixed Status: Resolved (was: Patch Available) https://github.com/apache/nifi-minifi-cpp/commit/f27e349b508c502218ca702b78fad2ef931f4f83 > Add default Connection size limits > -- > > Key: MINIFICPP-1887 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1887 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marton Szasz >Assignee: Martin Zink >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
adamdebreceni commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r1120015965
##
libminifi/test/rocksdb-tests/DBContentRepositoryTests.cpp:
##
@@ -159,7 +159,7 @@ TEST_CASE("Delete NonExistent Claim", "[TestDBCR4]") {
stream->close();
- content_repo->stop();
+ content_repo->deinitialize();
Review Comment:
we could drop the deinitialize (or make it private) and allow the tests to
access it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
adamdebreceni commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r1120013803
##
libminifi/test/rocksdb-tests/DBContentRepositoryTests.cpp:
##
@@ -159,7 +159,7 @@ TEST_CASE("Delete NonExistent Claim", "[TestDBCR4]") {
stream->close();
- content_repo->stop();
+ content_repo->deinitialize();
Review Comment:
for the database content repository `stop` was a test-only feature, since it
had no background threads there wasn't
really a point in following in flowfilerepository's footsteps
now that it has a background thread its lifecycle should mimic the
flowfilerepository/provenancerepository's `ctor -> initialize -> start <->
stop -> dtor` pattern IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
szaszm commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r112653
##
libminifi/test/rocksdb-tests/DBContentRepositoryTests.cpp:
##
@@ -159,7 +159,7 @@ TEST_CASE("Delete NonExistent Claim", "[TestDBCR4]") {
stream->close();
- content_repo->stop();
+ content_repo->deinitialize();
Review Comment:
And why do we need a separate stop() and deinitialize(), which closes the DB
after stop. Why can't stop() close the DB like before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1510: MINIFICPP-2047 added reverseDnsLookup to Expression Language
szaszm closed pull request #1510: MINIFICPP-2047 added reverseDnsLookup to Expression Language URL: https://github.com/apache/nifi-minifi-cpp/pull/1510 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1501: MINIFICPP-1887 Add default connection size limits
szaszm closed pull request #1501: MINIFICPP-1887 Add default connection size limits URL: https://github.com/apache/nifi-minifi-cpp/pull/1501 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
adamdebreceni commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r1119993446
##
extensions/rocksdb-repos/DatabaseContentRepository.cpp:
##
@@ -72,13 +87,40 @@ bool DatabaseContentRepository::initialize(const
std::shared_ptropen()) {
+ auto status = opendb->RunCompaction();
+ logger_->log_trace("Compaction triggered: %s", status.ToString());
+} else {
+ logger_->log_error("Failed to open database for compaction");
+}
+ } while (!utils::StoppableThread::wait_stop_requested(compaction_period_));
+}
+
+void DatabaseContentRepository::start() {
+ if (!db_ || !is_valid_) {
+return;
+ }
+ if (compaction_period_.count() != 0) {
+compaction_thread_ = std::make_unique([this] () {
+ runCompaction();
+});
+ }
+}
+
void DatabaseContentRepository::stop() {
if (db_) {
auto opendb = db_->open();
if (opendb) {
opendb->FlushWAL(true);
}
+compaction_thread_.reset();
Review Comment:
the thread's destructor called during reset block until it is joined
##
extensions/rocksdb-repos/DatabaseContentRepository.cpp:
##
@@ -72,13 +87,40 @@ bool DatabaseContentRepository::initialize(const
std::shared_ptropen()) {
+ auto status = opendb->RunCompaction();
+ logger_->log_trace("Compaction triggered: %s", status.ToString());
+} else {
+ logger_->log_error("Failed to open database for compaction");
+}
+ } while (!utils::StoppableThread::wait_stop_requested(compaction_period_));
+}
+
+void DatabaseContentRepository::start() {
+ if (!db_ || !is_valid_) {
+return;
+ }
+ if (compaction_period_.count() != 0) {
+compaction_thread_ = std::make_unique([this] () {
+ runCompaction();
+});
+ }
+}
+
void DatabaseContentRepository::stop() {
if (db_) {
auto opendb = db_->open();
if (opendb) {
opendb->FlushWAL(true);
}
+compaction_thread_.reset();
Review Comment:
the thread's destructor called during reset blocks until it is joined
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
adamdebreceni commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r1119992159
##
libminifi/test/rocksdb-tests/DBContentRepositoryTests.cpp:
##
@@ -159,7 +159,7 @@ TEST_CASE("Delete NonExistent Claim", "[TestDBCR4]") {
stream->close();
- content_repo->stop();
+ content_repo->deinitialize();
Review Comment:
the resource claim stores a shared_ptr to the content repository, "stop()"
previously put the repo into an "empty" state, so that a new repository
instance could open and manage the rocksdb backend, (during normal operations a
ResourceClaim should not outlive the content repo), but these tests depend on
being able to invalidate a previous instance and reusing an old claim, these
tests should be revised
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request, #1517: MINIFICPP-2048 ARM64 support part 1
martinzink opened a new pull request, #1517: URL: https://github.com/apache/nifi-minifi-cpp/pull/1517 MINIFICPP-2049 update crc32c MINIFICPP-2050 AWS tests compilation fix for ARM MINIFICPP-2051 updated ossp-uuid config.guess MINIFICPP-2052 unsigned/signed char fixes With these changes I was able to compile the project on aarch64 rockylinux 9. --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file? - [X] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1516: MINIFICPP-2054 - Periodically run manual compaction
szaszm commented on code in PR #1516:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1516#discussion_r1119956317
##
libminifi/test/rocksdb-tests/DBContentRepositoryTests.cpp:
##
@@ -159,7 +159,7 @@ TEST_CASE("Delete NonExistent Claim", "[TestDBCR4]") {
stream->close();
- content_repo->stop();
+ content_repo->deinitialize();
Review Comment:
Shouldn't this just be called the destructor? What's the reason for keeping
it separate?
##
extensions/rocksdb-repos/DatabaseContentRepository.cpp:
##
@@ -72,13 +87,40 @@ bool DatabaseContentRepository::initialize(const
std::shared_ptropen()) {
+ auto status = opendb->RunCompaction();
+ logger_->log_trace("Compaction triggered: %s", status.ToString());
+} else {
+ logger_->log_error("Failed to open database for compaction");
+}
+ } while (!utils::StoppableThread::wait_stop_requested(compaction_period_));
+}
+
+void DatabaseContentRepository::start() {
+ if (!db_ || !is_valid_) {
+return;
+ }
+ if (compaction_period_.count() != 0) {
+compaction_thread_ = std::make_unique([this] () {
+ runCompaction();
+});
+ }
+}
+
void DatabaseContentRepository::stop() {
if (db_) {
auto opendb = db_->open();
if (opendb) {
opendb->FlushWAL(true);
}
+compaction_thread_.reset();
Review Comment:
I think this should block until the thread is stopped, to avoid accessing
resources that are being cleaned up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni opened a new pull request, #1516: MINIFICPP-2054 - Periodically run manual compaction
adamdebreceni opened a new pull request, #1516: URL: https://github.com/apache/nifi-minifi-cpp/pull/1516 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pvillard31 commented on a diff in pull request #6991: NIFI-11224 - FF attribute support in Where condition of QuerySalesforceObject
pvillard31 commented on code in PR #6991:
URL: https://github.com/apache/nifi/pull/6991#discussion_r1119946978
##
nifi-nar-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/main/java/org/apache/nifi/processors/salesforce/QuerySalesforceObject.java:
##
@@ -326,24 +326,24 @@ protected Collection
customValidate(ValidationContext validati
public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
boolean isCustomQuery =
CUSTOM_QUERY.getValue().equals(context.getProperty(QUERY_TYPE).getValue());
+FlowFile flowFile = session.get();
Review Comment:
Thanks for the review, since we do want to evaluate EL wether the FlowFile
is null or not, I added a null check in the ``processQuery`` part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2050) AWS tests compile failure on ARM
[ https://issues.apache.org/jira/browse/MINIFICPP-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2050: --- Summary: AWS tests compile failure on ARM (was: AWS extension compile failure on ARM) > AWS tests compile failure on ARM > > > Key: MINIFICPP-2050 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2050 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > Some linking issue only on ARM. (linux) -- This message was sent by Atlassian Jira (v8.20.10#820010)
