[jira] [Updated] (NIFI-5725) ConsumeAzureEventHub processor - add custom attribute mapping to SystemProperties and UserProperties
[ https://issues.apache.org/jira/browse/NIFI-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakub Dropia updated NIFI-5725: --- Description: Currently, ConsumeAzureEventHub processor takes from EventData a predefined set of SystemProperties and adds them to FlowFile attributes. But there are cases, where SystemProperties may have contain more information (form example IoT Hub) and would be nice to add possibility to define mapping to them via dynamic property on processor. The same story is with UserProperties which right now are completly ignored and we cannot reach information included there. I would like to propose change for this. was: Currently, GetAzureEventHub processor takes from EventData a predefined set of SystemProperties and adds them to FlowFile attributes. But there are cases, where SystemProperties may have contain more information (form example IoT Hub) and would be nice to add possibility to define mapping to them via dynamic property on processor. The same story is with UserProperties which right now are completly ignored and we cannot reach information included there. I would like to propose change for this. Summary: ConsumeAzureEventHub processor - add custom attribute mapping to SystemProperties and UserProperties (was: GetAzureEventHub processor - add custom attribute mapping to SystemProperties and UserProperties) > ConsumeAzureEventHub processor - add custom attribute mapping to > SystemProperties and UserProperties > > > Key: NIFI-5725 > URL: https://issues.apache.org/jira/browse/NIFI-5725 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Jakub Dropia >Priority: Major > > Currently, ConsumeAzureEventHub processor takes from EventData a predefined > set of SystemProperties and adds them to FlowFile attributes. But there are > cases, where SystemProperties may have contain more information (form example > IoT Hub) and would be nice to add possibility to define mapping to them via > dynamic property on processor. > The same story is with UserProperties which right now are completly ignored > and we cannot reach information included there. > I would like to propose change for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5725) GetAzureEventHub processor - add custom attribute mapping to SystemProperties and UserProperties
Jakub Dropia created NIFI-5725: -- Summary: GetAzureEventHub processor - add custom attribute mapping to SystemProperties and UserProperties Key: NIFI-5725 URL: https://issues.apache.org/jira/browse/NIFI-5725 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Jakub Dropia Currently, GetAzureEventHub processor takes from EventData a predefined set of SystemProperties and adds them to FlowFile attributes. But there are cases, where SystemProperties may have contain more information (form example IoT Hub) and would be nice to add possibility to define mapping to them via dynamic property on processor. The same story is with UserProperties which right now are completly ignored and we cannot reach information included there. I would like to propose change for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5706) Processor ConvertAvroToParquet
[ https://issues.apache.org/jira/browse/NIFI-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656333#comment-16656333 ] ASF GitHub Bot commented on NIFI-5706: -- Github user mohitgargk commented on the issue: https://github.com/apache/nifi/pull/3079 > What is the use case for having Parquet files flow through NiFi? I have a use case where I am writing the parquet files into data lake (in azure) and also need to write them on a network storage device. The intent here is keep the parquet in flowfile, so that it can flow into any sink later on. If we don't have Convert*ToParquet (or similar processors), we would require a hop (an i/o endpoint) that may not give good performance. > Processor ConvertAvroToParquet > --- > > Key: NIFI-5706 > URL: https://issues.apache.org/jira/browse/NIFI-5706 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Mohit >Priority: Major > Labels: pull-request-available > > *Why*? > PutParquet support is limited to HDFS. > PutParquet bypasses the _flowfile_ implementation and writes the file > directly to sink. > We need a processor for parquet that works like _ConvertAvroToOrc_. > *What*? > _ConvertAvroToParquet_ will convert the incoming avro flowfile to a parquet > flowfile. Unlike PutParquet, which writes to the hdfs file system, processor > ConvertAvroToParquet would write into the flowfile, which can be pipelined to > put into other sinks, like _local_, _S3, Azure data lake_ etc. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3079: NIFI-5706 : Added ConvertAvroToParquet processor
Github user mohitgargk commented on the issue: https://github.com/apache/nifi/pull/3079 > What is the use case for having Parquet files flow through NiFi? I have a use case where I am writing the parquet files into data lake (in azure) and also need to write them on a network storage device. The intent here is keep the parquet in flowfile, so that it can flow into any sink later on. If we don't have Convert*ToParquet (or similar processors), we would require a hop (an i/o endpoint) that may not give good performance. ---
[jira] [Commented] (MINIFICPP-649) Fix some compiler warnings
[ https://issues.apache.org/jira/browse/MINIFICPP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656147#comment-16656147 ] ASF GitHub Bot commented on MINIFICPP-649: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/423 > Fix some compiler warnings > -- > > Key: MINIFICPP-649 > URL: https://issues.apache.org/jira/browse/MINIFICPP-649 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Some warnings might be related to errors, these worth checking and fixing. > For eg.: > _/Users/aboda/work/shadow/minifi/nifi-minifi-cpp/LibExample/monitor_directory.c:43:3: > warning: implicit declaration of function 'pthread_mutex_lock' is invalid in > C99 [-Wimplicit-function-declaration]_ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #423: MINIFICPP-649 - Fix some compiler warning...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/423 ---
[jira] [Commented] (NIFI-5446) Specify the file encoding option to UTF-8 in pom.xml
[
https://issues.apache.org/jira/browse/NIFI-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656072#comment-16656072
]
ASF GitHub Bot commented on NIFI-5446:
--
Github user tasanuma commented on the issue:
https://github.com/apache/nifi/pull/2910
Thanks for reviewing and merging it, @ijokarumawak!
> Specify the file encoding option to UTF-8 in pom.xml
>
>
> Key: NIFI-5446
> URL: https://issues.apache.org/jira/browse/NIFI-5446
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> {{TestJacksonCSVRecordReader.testUTF8}} failed in my environment (CentOS 7).
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.nifi.csv.TestJacksonCSVRecordReader
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.272
> s <<< FAILURE! - in org.apache.nifi.csv.TestJacksonCSVRecordReader
> [ERROR] testUTF8(org.apache.nifi.csv.TestJacksonCSVRecordReader) Time
> elapsed: 1.007 s <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[???]> but was:<[???]>
> at
> org.apache.nifi.csv.TestJacksonCSVRecordReader.testUTF8(TestJacksonCSVRecordReader.java:81)
> {noformat}
> After a bit of research, the problem seems to be same as
> [here|https://stackoverflow.com/questions/17656475/maven-source-encoding-in-utf-8-not-working].
> I specified the file-encoding option of UTF-8 in the command line arguments
> and the unit test succeeded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2910: NIFI-5446: Specify the file encoding option to UTF-8 in po...
Github user tasanuma commented on the issue: https://github.com/apache/nifi/pull/2910 Thanks for reviewing and merging it, @ijokarumawak! ---
[jira] [Commented] (NIFI-5642) QueryCassandra processor : output FlowFiles as soon fetch_size is reached
[ https://issues.apache.org/jira/browse/NIFI-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655839#comment-16655839 ] André Gomes Lamas Otero commented on NIFI-5642: --- Hi Folks! Is anyone available to review my PR ? Thanks! > QueryCassandra processor : output FlowFiles as soon fetch_size is reached > - > > Key: NIFI-5642 > URL: https://issues.apache.org/jira/browse/NIFI-5642 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.7.1 >Reporter: André Gomes Lamas Otero >Priority: Major > > When I'm using QueryCassandra alongside with fetch_size parameter I expected > that as soon my reader reaches the fetch_size the processor outputs some data > to be processed by the next processor, but QueryCassandra reads all the data, > then output the flow files. > I'll start to work on a patch for this situation, I'll appreciate any > suggestion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655640#comment-16655640 ] ASF GitHub Bot commented on NIFI-4805: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 @joewitt It's not done, more of a rough draft, but take a look at my latest commit and let me know if this is more in line with what you were thinking. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3078: NIFI-4805 Allow Delayed Transfer
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 @joewitt It's not done, more of a rough draft, but take a look at my latest commit and let me know if this is more in line with what you were thinking. ---
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[
https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655584#comment-16655584
]
ASF GitHub Bot commented on MINIFICPP-647:
--
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226386769
--- Diff: libminifi/src/capi/api.cpp ---
@@ -57,38 +57,28 @@ class DirectoryConfiguration {
}
};
-nifi_port *create_port(const char *port) {
- if (nullptr == port)
-return nullptr;
- nifi_port *p = new nifi_port();
- p->port_id = new char[strlen(port) + 1];
- memset(p->port_id, 0x00, strlen(port) + 1);
- strncpy(p->port_id, port, strlen(port));
- return p;
-}
-
-int free_port(nifi_port *port) {
- if (port == nullptr)
-return -1;
- delete[] port->port_id;
- delete port;
- return 0;
-}
-
/**
* Creates a NiFi Instance from the url and output port.
* @param url http URL for NiFi instance
* @param port Remote output port.
+ * @Deprecated for 0.6.0 in favor of
--- End diff --
Well rereading the deprecation note, it makes more sense than my first take
on things. I guess I anticipate this maybe being problematic in the near
future when both would reach the 1.0 mark. I know there is connectedness
between the both the capi and the agent itself but would almost prefer just
saying that it was done for 0.02. I could see the API reaching a certain level
of maturity where the agent would continue to evolve but the API itself is
fairly static.
As a general thought, we could probably use some exploration of minifi
cpp's versioning/support guarantees especially as this API gets built out.
> Deprecate functions in API that pass char* in favor of those more strongly
> typed
>
>
> Key: MINIFICPP-647
> URL: https://issues.apache.org/jira/browse/MINIFICPP-647
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226386769
--- Diff: libminifi/src/capi/api.cpp ---
@@ -57,38 +57,28 @@ class DirectoryConfiguration {
}
};
-nifi_port *create_port(const char *port) {
- if (nullptr == port)
-return nullptr;
- nifi_port *p = new nifi_port();
- p->port_id = new char[strlen(port) + 1];
- memset(p->port_id, 0x00, strlen(port) + 1);
- strncpy(p->port_id, port, strlen(port));
- return p;
-}
-
-int free_port(nifi_port *port) {
- if (port == nullptr)
-return -1;
- delete[] port->port_id;
- delete port;
- return 0;
-}
-
/**
* Creates a NiFi Instance from the url and output port.
* @param url http URL for NiFi instance
* @param port Remote output port.
+ * @Deprecated for 0.6.0 in favor of
--- End diff --
Well rereading the deprecation note, it makes more sense than my first take
on things. I guess I anticipate this maybe being problematic in the near
future when both would reach the 1.0 mark. I know there is connectedness
between the both the capi and the agent itself but would almost prefer just
saying that it was done for 0.02. I could see the API reaching a certain level
of maturity where the agent would continue to evolve but the API itself is
fairly static.
As a general thought, we could probably use some exploration of minifi
cpp's versioning/support guarantees especially as this API gets built out.
---
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[
https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655572#comment-16655572
]
ASF GitHub Bot commented on MINIFICPP-647:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226384252
--- Diff: libminifi/src/capi/api.cpp ---
@@ -57,38 +57,28 @@ class DirectoryConfiguration {
}
};
-nifi_port *create_port(const char *port) {
- if (nullptr == port)
-return nullptr;
- nifi_port *p = new nifi_port();
- p->port_id = new char[strlen(port) + 1];
- memset(p->port_id, 0x00, strlen(port) + 1);
- strncpy(p->port_id, port, strlen(port));
- return p;
-}
-
-int free_port(nifi_port *port) {
- if (port == nullptr)
-return -1;
- delete[] port->port_id;
- delete port;
- return 0;
-}
-
/**
* Creates a NiFi Instance from the url and output port.
* @param url http URL for NiFi instance
* @param port Remote output port.
+ * @Deprecated for 0.6.0 in favor of
--- End diff --
Oh that's a very good point.
> Deprecate functions in API that pass char* in favor of those more strongly
> typed
>
>
> Key: MINIFICPP-647
> URL: https://issues.apache.org/jira/browse/MINIFICPP-647
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226384252
--- Diff: libminifi/src/capi/api.cpp ---
@@ -57,38 +57,28 @@ class DirectoryConfiguration {
}
};
-nifi_port *create_port(const char *port) {
- if (nullptr == port)
-return nullptr;
- nifi_port *p = new nifi_port();
- p->port_id = new char[strlen(port) + 1];
- memset(p->port_id, 0x00, strlen(port) + 1);
- strncpy(p->port_id, port, strlen(port));
- return p;
-}
-
-int free_port(nifi_port *port) {
- if (port == nullptr)
-return -1;
- delete[] port->port_id;
- delete port;
- return 0;
-}
-
/**
* Creates a NiFi Instance from the url and output port.
* @param url http URL for NiFi instance
* @param port Remote output port.
+ * @Deprecated for 0.6.0 in favor of
--- End diff --
Oh that's a very good point.
---
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[
https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655567#comment-16655567
]
ASF GitHub Bot commented on MINIFICPP-647:
--
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226382717
--- Diff: libminifi/src/capi/api.cpp ---
@@ -57,38 +57,28 @@ class DirectoryConfiguration {
}
};
-nifi_port *create_port(const char *port) {
- if (nullptr == port)
-return nullptr;
- nifi_port *p = new nifi_port();
- p->port_id = new char[strlen(port) + 1];
- memset(p->port_id, 0x00, strlen(port) + 1);
- strncpy(p->port_id, port, strlen(port));
- return p;
-}
-
-int free_port(nifi_port *port) {
- if (port == nullptr)
-return -1;
- delete[] port->port_id;
- delete port;
- return 0;
-}
-
/**
* Creates a NiFi Instance from the url and output port.
* @param url http URL for NiFi instance
* @param port Remote output port.
+ * @Deprecated for 0.6.0 in favor of
--- End diff --
Sorry to nitpick, but shall we refer to minifi releases or API version,
could see this getting a little confusing at some point? How could we best
document these version relationships?
> Deprecate functions in API that pass char* in favor of those more strongly
> typed
>
>
> Key: MINIFICPP-647
> URL: https://issues.apache.org/jira/browse/MINIFICPP-647
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226382717
--- Diff: libminifi/src/capi/api.cpp ---
@@ -57,38 +57,28 @@ class DirectoryConfiguration {
}
};
-nifi_port *create_port(const char *port) {
- if (nullptr == port)
-return nullptr;
- nifi_port *p = new nifi_port();
- p->port_id = new char[strlen(port) + 1];
- memset(p->port_id, 0x00, strlen(port) + 1);
- strncpy(p->port_id, port, strlen(port));
- return p;
-}
-
-int free_port(nifi_port *port) {
- if (port == nullptr)
-return -1;
- delete[] port->port_id;
- delete port;
- return 0;
-}
-
/**
* Creates a NiFi Instance from the url and output port.
* @param url http URL for NiFi instance
* @param port Remote output port.
+ * @Deprecated for 0.6.0 in favor of
--- End diff --
Sorry to nitpick, but shall we refer to minifi releases or API version,
could see this getting a little confusing at some point? How could we best
document these version relationships?
---
[jira] [Commented] (NIFIREG-195) Add GitFlowPersistenceProvider video to Registry home page
[ https://issues.apache.org/jira/browse/NIFIREG-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1665#comment-1665 ] ASF GitHub Bot commented on NIFIREG-195: Github user asfgit closed the pull request at: https://github.com/apache/nifi-site/pull/29 > Add GitFlowPersistenceProvider video to Registry home page > -- > > Key: NIFIREG-195 > URL: https://issues.apache.org/jira/browse/NIFIREG-195 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Andrew Lim >Priority: Minor > > A new video can be added to the currently listed ones: > https://youtu.be/kK7eVppg9Aw -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFIREG-195) Add GitFlowPersistenceProvider video to Registry home page
[ https://issues.apache.org/jira/browse/NIFIREG-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved NIFIREG-195. - Resolution: Done > Add GitFlowPersistenceProvider video to Registry home page > -- > > Key: NIFIREG-195 > URL: https://issues.apache.org/jira/browse/NIFIREG-195 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > > A new video can be added to the currently listed ones: > https://youtu.be/kK7eVppg9Aw -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFIREG-195) Add GitFlowPersistenceProvider video to Registry home page
[ https://issues.apache.org/jira/browse/NIFIREG-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned NIFIREG-195: --- Assignee: Andrew Lim > Add GitFlowPersistenceProvider video to Registry home page > -- > > Key: NIFIREG-195 > URL: https://issues.apache.org/jira/browse/NIFIREG-195 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > > A new video can be added to the currently listed ones: > https://youtu.be/kK7eVppg9Aw -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-site pull request #29: NIFIREG-195 Add GitFlowPersistenceProvider video...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-site/pull/29 ---
[jira] [Updated] (NIFI-5723) PutDatabaseRecord strips underbar character from column names
[
https://issues.apache.org/jira/browse/NIFI-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Wicks updated NIFI-5723:
--
Summary: PutDatabaseRecord strips underbar character from column names
(was: PutDatabaseRecord stripps underbar character from column names)
> PutDatabaseRecord strips underbar character from column names
> -
>
> Key: NIFI-5723
> URL: https://issues.apache.org/jira/browse/NIFI-5723
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.7.1
> Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
> Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK
> Runtime Environment (build 1.8.0_181-b13)
>Reporter: Uwe Geercken
>Priority: Major
>
> In *PutDatabaseRecord.java* there is following code:
> private static String normalizeColumnName(final String colName, final boolean
> translateColumnNames) {
> return colName == null ? null : (translateColumnNames ?
> colName.toUpperCase().replace("_", "") : colName);
> }
> If translateColumnNames is set to true, then the underbar character is
> stripped from the column names.
> And it is also here:
> final ColumnDescription desc =
> tableSchema.getColumns().get(normalizeColumnName(fieldName,
> settings.translateFieldNames));
> if (desc == null && !settings.ignoreUnmappedFields) {
> throw new SQLDataException("Cannot map field '" +
> fieldName + "' to any column in the database");
> }
> I do not see a reason why this is done. An underbar is a common delimiter in
> database column names.
> Specifically, I have data comming from an Oracle database which is all
> uppercase and with the underbar character as the delimiter in the column
> names and due to the current code, I can not update my MySQL database using
> the same column names.
> Trying to solve the issue in other ways does not really work as well.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (NIFI-5723) PutDatabaseRecord strips underbar character from column names
[
https://issues.apache.org/jira/browse/NIFI-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Wicks updated NIFI-5723:
--
Comment: was deleted
(was: If memory serves, this was because Avro does not support underscore's in
field names. So if you have underscore's in your field names then the export of
your data in the Avro format would also fail.
By allowing you to replace underscores it gives users a way to extract the
data. In NiFi 1.8 you should see some new processors for working with SQL that
allow you to export the data using Record Writers, giving you more control over
this process. But if you use Avro, this is still necessary.)
> PutDatabaseRecord strips underbar character from column names
> -
>
> Key: NIFI-5723
> URL: https://issues.apache.org/jira/browse/NIFI-5723
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.7.1
> Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
> Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK
> Runtime Environment (build 1.8.0_181-b13)
>Reporter: Uwe Geercken
>Priority: Major
>
> In *PutDatabaseRecord.java* there is following code:
> private static String normalizeColumnName(final String colName, final boolean
> translateColumnNames) {
> return colName == null ? null : (translateColumnNames ?
> colName.toUpperCase().replace("_", "") : colName);
> }
> If translateColumnNames is set to true, then the underbar character is
> stripped from the column names.
> And it is also here:
> final ColumnDescription desc =
> tableSchema.getColumns().get(normalizeColumnName(fieldName,
> settings.translateFieldNames));
> if (desc == null && !settings.ignoreUnmappedFields) {
> throw new SQLDataException("Cannot map field '" +
> fieldName + "' to any column in the database");
> }
> I do not see a reason why this is done. An underbar is a common delimiter in
> database column names.
> Specifically, I have data comming from an Oracle database which is all
> uppercase and with the underbar character as the delimiter in the column
> names and due to the current code, I can not update my MySQL database using
> the same column names.
> Trying to solve the issue in other ways does not really work as well.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5723) PutDatabaseRecord stripps underbar character from column names
[
https://issues.apache.org/jira/browse/NIFI-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655525#comment-16655525
]
Peter Wicks commented on NIFI-5723:
---
If memory serves, this was because Avro does not support underscore's in field
names. So if you have underscore's in your field names then the export of your
data in the Avro format would also fail.
By allowing you to replace underscores it gives users a way to extract the
data. In NiFi 1.8 you should see some new processors for working with SQL that
allow you to export the data using Record Writers, giving you more control over
this process. But if you use Avro, this is still necessary.
> PutDatabaseRecord stripps underbar character from column names
> --
>
> Key: NIFI-5723
> URL: https://issues.apache.org/jira/browse/NIFI-5723
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.7.1
> Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
> Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK
> Runtime Environment (build 1.8.0_181-b13)
>Reporter: Uwe Geercken
>Priority: Major
>
> In *PutDatabaseRecord.java* there is following code:
> private static String normalizeColumnName(final String colName, final boolean
> translateColumnNames) {
> return colName == null ? null : (translateColumnNames ?
> colName.toUpperCase().replace("_", "") : colName);
> }
> If translateColumnNames is set to true, then the underbar character is
> stripped from the column names.
> And it is also here:
> final ColumnDescription desc =
> tableSchema.getColumns().get(normalizeColumnName(fieldName,
> settings.translateFieldNames));
> if (desc == null && !settings.ignoreUnmappedFields) {
> throw new SQLDataException("Cannot map field '" +
> fieldName + "' to any column in the database");
> }
> I do not see a reason why this is done. An underbar is a common delimiter in
> database column names.
> Specifically, I have data comming from an Oracle database which is all
> uppercase and with the underbar character as the delimiter in the column
> names and due to the current code, I can not update my MySQL database using
> the same column names.
> Trying to solve the issue in other ways does not really work as well.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655475#comment-16655475
]
ASF GitHub Bot commented on NIFI-5714:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3086
tried to squash everything and force push, but same result - tried to look
in the code if something is different and should be initialized differently but
didn't find anything obvious :(
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0,
> 1.7.0, 1.7.1
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 tried to squash everything and force push, but same result - tried to look in the code if something is different and should be initialized differently but didn't find anything obvious :( ---
[jira] [Commented] (NIFI-5706) Processor ConvertAvroToParquet
[ https://issues.apache.org/jira/browse/NIFI-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655455#comment-16655455 ] ASF GitHub Bot commented on NIFI-5706: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3079 What is the use case for having Parquet files flow through NiFi? I wrote ConvertAvroToORC and every time I saw it used, it was immediately followed by PutHDFS. So I followed the PutParquet lead and wrote a PutORC for the Hive 3 NAR. We can't currently do anything with a Parquet file (or ORC for that matter) in NiFi, so just curious as to how you envision it being used. Also I wonder if a ParquetRecordWriter might be a better idea? It would do the same thing as a processor but the record processors can read in something in any format, not just Avro. This was another impetus for having PutORC instead of ConvertAvroToORC. > Processor ConvertAvroToParquet > --- > > Key: NIFI-5706 > URL: https://issues.apache.org/jira/browse/NIFI-5706 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Mohit >Priority: Major > Labels: pull-request-available > > *Why*? > PutParquet support is limited to HDFS. > PutParquet bypasses the _flowfile_ implementation and writes the file > directly to sink. > We need a processor for parquet that works like _ConvertAvroToOrc_. > *What*? > _ConvertAvroToParquet_ will convert the incoming avro flowfile to a parquet > flowfile. Unlike PutParquet, which writes to the hdfs file system, processor > ConvertAvroToParquet would write into the flowfile, which can be pipelined to > put into other sinks, like _local_, _S3, Azure data lake_ etc. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3079: NIFI-5706 : Added ConvertAvroToParquet processor
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3079 What is the use case for having Parquet files flow through NiFi? I wrote ConvertAvroToORC and every time I saw it used, it was immediately followed by PutHDFS. So I followed the PutParquet lead and wrote a PutORC for the Hive 3 NAR. We can't currently do anything with a Parquet file (or ORC for that matter) in NiFi, so just curious as to how you envision it being used. Also I wonder if a ParquetRecordWriter might be a better idea? It would do the same thing as a processor but the record processors can read in something in any format, not just Avro. This was another impetus for having PutORC instead of ConvertAvroToORC. ---
[jira] [Commented] (NIFI-5683) Add record writer capability to ListenBeats
[ https://issues.apache.org/jira/browse/NIFI-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655452#comment-16655452 ] Matt Burgess commented on NIFI-5683: I don't see a PR/patch for it, can you supply the link? > Add record writer capability to ListenBeats > --- > > Key: NIFI-5683 > URL: https://issues.apache.org/jira/browse/NIFI-5683 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > Make ListenBeats able to write beats output as record sets using a configured > writer, with a limit for the number of records per flowfile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343199
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+.description("If true then when a FlowFile is routed to 'over
limit' a message will be logged at the level 'warn'.")
+.allowableValues("true", "false")
+.defaultValue("true")
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226341991
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+.description("If true then when a FlowFile is routed to 'over
limit' a message will be logged at the level 'warn'.")
+.allowableValues("true", "false")
+.defaultValue("true")
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655407#comment-16655407
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226341579
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
--- End diff --
What about changing the scope to `FLOWFILE_ATTRIBUTES` so it can be based
on a per `FlowFile` basis?
> A processor to facilitate retrying FlowFiles
>
>
> Key: NIFI-3792
> URL: https://issues.apache.org/jira/browse/NIFI-3792
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> When dealing with processor failures in production, potentially related to
> network issues, the current best practice is to retry the flowfile a couple
> times before declaring it a failure[1]. This currently requires multiple
> processors and penalizing
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226339913
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
--- End diff --
I really like `Retry` as the processor name. I think it would be more
intuitive for most users.
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226344163
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+.description("If true then when a FlowFile is routed to 'over
limit' a message will be logged at the level 'warn'.")
+.allowableValues("true", "false")
+.defaultValue("true")
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343311
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+.description("If true then when a FlowFile is routed to 'over
limit' a message will be logged at the level 'warn'.")
+.allowableValues("true", "false")
+.defaultValue("true")
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655409#comment-16655409
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226347867
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.display
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655402#comment-16655402
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340692
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
--- End diff --
All of your `.name(" ")` need to be changed. It's good that you have
separate `name` and `displayName`, but it should be more like, `retry-limit`.
> A processor to facilitate retrying FlowFiles
>
>
> Key: NIFI-3792
> URL: https://issues.apache.org/jira/browse/NIFI-3792
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> When dealing with processor failures in production, potentially related to
> network issues, the current best practice is to retry the flowfile a couple
> times before declaring it a failure[1]. This currently requires multiple
> processors and penalizing the flowfile isn't possible. Also if the flow is
> not fast enough, back-pressure can cause a livelocked state which requires
> manual intervention.
> A new processor should be added to not only
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655412#comment-16655412
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343311
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.display
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655411#comment-16655411
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226344163
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.display
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655401#comment-16655401
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340682
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
--- End diff --
Suggestion: `retry-penalize-flowfile`
> A processor to facilitate retrying FlowFiles
>
>
> Key: NIFI-3792
> URL: https://issues.apache.org/jira/browse/NIFI-3792
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
>
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340682
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
--- End diff --
Suggestion: `retry-penalize-flowfile`
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226341579
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
--- End diff --
What about changing the scope to `FLOWFILE_ATTRIBUTES` so it can be based
on a per `FlowFile` basis?
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340692
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
--- End diff --
All of your `.name(" ")` need to be changed. It's good that you have
separate `name` and `displayName`, but it should be more like, `retry-limit`.
---
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655408#comment-16655408
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343199
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.display
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655400#comment-16655400
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226339913
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
--- End diff --
I really like `Retry` as the processor name. I think it would be more
intuitive for most users.
> A processor to facilitate retrying FlowFiles
>
>
> Key: NIFI-3792
> URL: https://issues.apache.org/jira/browse/NIFI-3792
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> When dealing with processor failures in production, potentially related to
> network issues, the current best practice is to retry the flowfile a couple
> times before declaring it a failure[1]. This currently requires multiple
> processors and penalizing the flowfile isn't possible. Also if the flow is
> not fast enough, back-pressure can cause a livelocked state which requires
> manual intervention.
> A new processor should be added to not only encourage best practices but also
> offer configuration options to deal with un-optimal situations.
> [1]
> https://community.hortonworks.com/questions/77336/nifi-best-practices-for-error-handling.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226345973
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
--- End diff --
It came up in discussions on the Jira for the Delay processor that having a
fixed retry count attribute could be tricky. How do you handle Retry attribute
contamination? If this gets set in an upstream flow, and then the same FlowFile
goes through a new Retry loop later on in the Flow, won't this attribute cause
the FlowFile to behave as though it has already been retried?
The workaround for this we came up with was to append the Processor UUID to
the attribute name. Thoughts?
---
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655404#comment-16655404
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226341991
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.display
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655406#comment-16655406
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226345973
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
--- End diff --
It came up in discussions on the Jira for the Delay processor that having a
fixed retry count attribute could be tricky. How do you handle Retry attribute
contamination? If this gets set in an upstream flow, and then the same FlowFile
goes through a new Retry loop later on in the Flow, won't this attribute cause
the FlowFile to behave as though it has already been retried?
The workaround for this we came up with was to append the Processor UUID to
the attribute name. Thoughts?
> A processor to facilitate retrying FlowFiles
>
>
> Key: NIFI-3792
> URL: https://issues.apache.org/jira/browse/NIFI-3792
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> When dealing with processor failures in production, potentially related to
> network issues, the current best practice is to retry the flowfile a couple
> times before declaring it a failure[1]. This currently requires multiple
> processors and penalizing
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655405#comment-16655405
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343019
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.display
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655410#comment-16655410
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226347295
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.display
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[
https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655403#comment-16655403
]
ASF GitHub Bot commented on NIFI-3792:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340820
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
--- End diff --
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340820
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
--- End diff --
Suggestion: `retry-warn-overlimit`
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226347867
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+.description("If true then when a FlowFile is routed to 'over
limit' a message will be logged at the level 'warn'.")
+.allowableValues("true", "false")
+.defaultValue("true")
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226347295
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+.description("If true then when a FlowFile is routed to 'over
limit' a message will be logged at the level 'warn'.")
+.allowableValues("true", "false")
+.defaultValue("true")
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343019
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+.description("If true then when a FlowFile is routed to 'over
limit' a message will be logged at the level 'warn'.")
+.allowableValues("true", "false")
+.defaultValue("true")
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655394#comment-16655394
]
ASF GitHub Bot commented on NIFI-5714:
--
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/3086
Doesn't look like the tests were run due to a bad artifact. Maybe rebase
the PR (squashing just the commits where you tried individual things?) and
force push, hopefully it'll go through this time :( Sorry this is such a pain.
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0,
> 1.7.0, 1.7.1
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3086 Doesn't look like the tests were run due to a bad artifact. Maybe rebase the PR (squashing just the commits where you tried individual things?) and force push, hopefully it'll go through this time :( Sorry this is such a pain. ---
[jira] [Commented] (MINIFICPP-642) Shared lib building causes issues in Docker image
[ https://issues.apache.org/jira/browse/MINIFICPP-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655352#comment-16655352 ] Aldrin Piri commented on MINIFICPP-642: --- Resolved via f3febb3da8f8805a0d486284ade05b93dfd6f714 > Shared lib building causes issues in Docker image > - > > Key: MINIFICPP-642 > URL: https://issues.apache.org/jira/browse/MINIFICPP-642 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > With the adjustment to the CMakeLists in support of Python there have proven > to be issues with the shared libs in the docker environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-642) Shared lib building causes issues in Docker image
[ https://issues.apache.org/jira/browse/MINIFICPP-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-642: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Shared lib building causes issues in Docker image > - > > Key: MINIFICPP-642 > URL: https://issues.apache.org/jira/browse/MINIFICPP-642 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > With the adjustment to the CMakeLists in support of Python there have proven > to be issues with the shared libs in the docker environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-643) Disable windows builds
[ https://issues.apache.org/jira/browse/MINIFICPP-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-643: -- Resolution: Done Status: Resolved (was: Patch Available) > Disable windows builds > -- > > Key: MINIFICPP-643 > URL: https://issues.apache.org/jira/browse/MINIFICPP-643 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Blocker > Labels: windows > Fix For: 0.6.0 > > > Not sure if these are needed. Having builds in windows and appveyor would > give the impression that windows is to be supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #419: Minificpp 643
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/419 ---
[jira] [Commented] (MINIFICPP-642) Shared lib building causes issues in Docker image
[ https://issues.apache.org/jira/browse/MINIFICPP-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655333#comment-16655333 ] ASF GitHub Bot commented on MINIFICPP-642: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/418 Closing in favor of #419 > Shared lib building causes issues in Docker image > - > > Key: MINIFICPP-642 > URL: https://issues.apache.org/jira/browse/MINIFICPP-642 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > With the adjustment to the CMakeLists in support of Python there have proven > to be issues with the shared libs in the docker environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-642) Shared lib building causes issues in Docker image
[ https://issues.apache.org/jira/browse/MINIFICPP-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655334#comment-16655334 ] ASF GitHub Bot commented on MINIFICPP-642: -- Github user apiri closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/418 > Shared lib building causes issues in Docker image > - > > Key: MINIFICPP-642 > URL: https://issues.apache.org/jira/browse/MINIFICPP-642 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > With the adjustment to the CMakeLists in support of Python there have proven > to be issues with the shared libs in the docker environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #418: MINIFICPP-642 Disable shared libs for Doc...
Github user apiri closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/418 ---
[GitHub] nifi-minifi-cpp issue #418: MINIFICPP-642 Disable shared libs for Docker ima...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/418 Closing in favor of #419 ---
[jira] [Created] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
vish uma created NIFI-5724: -- Summary: Make the autocommit value in the PutSQL processor configurable Key: NIFI-5724 URL: https://issues.apache.org/jira/browse/NIFI-5724 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: vish uma The PutSQL processor currently always sets the autocommit value on the database session to false before the SQL statement is run and resets it back to the original value after. i am not sure if the autocommit value is hardcoded to false for a reason, if it is, please let me know. This is causing an issue with the snowflake DB where abruptly disconnected sessions do not release the locks they have taken. i would like to make this autocommit value configurable. I can submit a patch for this if there is no objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5721) ConsumeMQTT processor can exhaust system threads
[ https://issues.apache.org/jira/browse/NIFI-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende reassigned NIFI-5721: - Assignee: Bryan Bende > ConsumeMQTT processor can exhaust system threads > > > Key: NIFI-5721 > URL: https://issues.apache.org/jira/browse/NIFI-5721 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Aldrin Piri >Assignee: Bryan Bende >Priority: Major > Fix For: 1.7.1 > > Attachments: ConsumeMQTT_OOME.xml, nifi-app-oome.log > > > Originally reported on the mailing list as a MiNiFi issue, > https://lists.apache.org/thread.html/d64afc392d8c503997634ed4e9285ba4e89ccc83002e99c66d259c89@%3Cusers.nifi.apache.org%3E. > After some initial evaluation it appears there is a bug being triggered in > the mqtt library that is not being handled by NiFi and causing a resource > leak. > Original thread here: > https://lists.apache.org/thread.html/d64afc392d8c503997634ed4e9285ba4e89ccc83002e99c66d259c89@%3Cusers.nifi.apache.org%3E > Additional details to follow as I evaluate how to trigger it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[
https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655277#comment-16655277
]
ASF GitHub Bot commented on MINIFICPP-647:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226321459
--- Diff: libminifi/test/capi/CAPITests.cpp ---
@@ -33,8 +32,14 @@
#include
#include
+static nifi_instance *create_instance_obj(const char *name =
"random_instance") {
+ nifi_port port;
+ port.port_id = "12345";
+ return create_instance("random_instance", &port);
--- End diff --
My bad, just didn't notice this being in the test code.
> Deprecate functions in API that pass char* in favor of those more strongly
> typed
>
>
> Key: MINIFICPP-647
> URL: https://issues.apache.org/jira/browse/MINIFICPP-647
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226321459
--- Diff: libminifi/test/capi/CAPITests.cpp ---
@@ -33,8 +32,14 @@
#include
#include
+static nifi_instance *create_instance_obj(const char *name =
"random_instance") {
+ nifi_port port;
+ port.port_id = "12345";
+ return create_instance("random_instance", &port);
--- End diff --
My bad, just didn't notice this being in the test code.
---
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[ https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655272#comment-16655272 ] ASF GitHub Bot commented on MINIFICPP-647: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226320573 --- Diff: python/minifi/__init__.py --- @@ -66,9 +66,9 @@ def setBase(self, proc): def get(self, session): ff = self._minifi.get(self._instance.get_instance(),self._flow, session) if ff: --- End diff -- Yeah i tend to avoid ternary operators in general since my experience with production code has always led maintainers at some point to result in error. Looks great -- but more errors than positives in my experience I'm afraid. > Deprecate functions in API that pass char* in favor of those more strongly > typed > > > Key: MINIFICPP-647 > URL: https://issues.apache.org/jira/browse/MINIFICPP-647 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226320573 --- Diff: python/minifi/__init__.py --- @@ -66,9 +66,9 @@ def setBase(self, proc): def get(self, session): ff = self._minifi.get(self._instance.get_instance(),self._flow, session) if ff: --- End diff -- Yeah i tend to avoid ternary operators in general since my experience with production code has always led maintainers at some point to result in error. Looks great -- but more errors than positives in my experience I'm afraid. ---
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[
https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655269#comment-16655269
]
ASF GitHub Bot commented on MINIFICPP-647:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226320291
--- Diff: libminifi/test/capi/CAPITests.cpp ---
@@ -33,8 +32,14 @@
#include
#include
+static nifi_instance *create_instance_obj(const char *name =
"random_instance") {
+ nifi_port port;
+ port.port_id = "12345";
+ return create_instance("random_instance", &port);
--- End diff --
Are you asking how can they change the port in the instance?
> Deprecate functions in API that pass char* in favor of those more strongly
> typed
>
>
> Key: MINIFICPP-647
> URL: https://issues.apache.org/jira/browse/MINIFICPP-647
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226320291
--- Diff: libminifi/test/capi/CAPITests.cpp ---
@@ -33,8 +32,14 @@
#include
#include
+static nifi_instance *create_instance_obj(const char *name =
"random_instance") {
+ nifi_port port;
+ port.port_id = "12345";
+ return create_instance("random_instance", &port);
--- End diff --
Are you asking how can they change the port in the instance?
---
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655232#comment-16655232
]
ASF GitHub Bot commented on NIFI-5629:
--
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226305582
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -270,6 +280,9 @@ private FileFilter createFileFilter(final
ProcessContext context) {
return new FileFilter() {
@Override
public boolean accept(final File file) {
+if (file.isDirectory()) {
+return false;
--- End diff --
If you look at the performListing, the filter is applied after the walk,
with no feedback mechanism, so the additional filter just prevents Directories
being returned as entities for GetFile to process.
> GetFile becomes slow listing vast directories
> -
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Adam
>Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226305582
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -270,6 +280,9 @@ private FileFilter createFileFilter(final
ProcessContext context) {
return new FileFilter() {
@Override
public boolean accept(final File file) {
+if (file.isDirectory()) {
+return false;
--- End diff --
If you look at the performListing, the filter is applied after the walk,
with no feedback mechanism, so the additional filter just prevents Directories
being returned as entities for GetFile to process.
---
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/3086
The krb5.conf from the TestRangerNiFiAuthorizer looks like this:
```
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_kdc = false
dns_lookup_realm = false
[realms]
EXAMPLE.COM = {
kdc = kerberos.example.com
admin_server = kerberos.example.com
}
```
And doesn't have the setting of the `java.security.krb5.realm` or
`java.security.krb5.kdc` (I assume because they are unnecessary based on the
dns_lookup_* properties?). Might be worth a try...
---
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655231#comment-16655231
]
ASF GitHub Bot commented on NIFI-5714:
--
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/3086
The krb5.conf from the TestRangerNiFiAuthorizer looks like this:
```
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_kdc = false
dns_lookup_realm = false
[realms]
EXAMPLE.COM = {
kdc = kerberos.example.com
admin_server = kerberos.example.com
}
```
And doesn't have the setting of the `java.security.krb5.realm` or
`java.security.krb5.kdc` (I assume because they are unnecessary based on the
dns_lookup_* properties?). Might be worth a try...
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0,
> 1.7.0, 1.7.1
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2910: NIFI-5446: Specify the file encoding option to UTF-...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2910 ---
[jira] [Commented] (NIFI-5446) Specify the file encoding option to UTF-8 in pom.xml
[
https://issues.apache.org/jira/browse/NIFI-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655221#comment-16655221
]
ASF GitHub Bot commented on NIFI-5446:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2910
> Specify the file encoding option to UTF-8 in pom.xml
>
>
> Key: NIFI-5446
> URL: https://issues.apache.org/jira/browse/NIFI-5446
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> {{TestJacksonCSVRecordReader.testUTF8}} failed in my environment (CentOS 7).
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.nifi.csv.TestJacksonCSVRecordReader
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.272
> s <<< FAILURE! - in org.apache.nifi.csv.TestJacksonCSVRecordReader
> [ERROR] testUTF8(org.apache.nifi.csv.TestJacksonCSVRecordReader) Time
> elapsed: 1.007 s <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[???]> but was:<[???]>
> at
> org.apache.nifi.csv.TestJacksonCSVRecordReader.testUTF8(TestJacksonCSVRecordReader.java:81)
> {noformat}
> After a bit of research, the problem seems to be same as
> [here|https://stackoverflow.com/questions/17656475/maven-source-encoding-in-utf-8-not-working].
> I specified the file-encoding option of UTF-8 in the command line arguments
> and the unit test succeeded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #419: Minificpp 643
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/419 reviewing ---
[jira] [Updated] (NIFI-5382) ConvertRecord 1.7.0 - Creates emtpy Flow File
[ https://issues.apache.org/jira/browse/NIFI-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-5382: -- Issue Type: Improvement (was: New Feature) > ConvertRecord 1.7.0 - Creates emtpy Flow File > - > > Key: NIFI-5382 > URL: https://issues.apache.org/jira/browse/NIFI-5382 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Stefan Goldener >Assignee: Peter Wicks >Priority: Minor > Fix For: 1.8.0 > > Attachments: ConvertRecordBUG.xml > > > The Nifi ConvertRecord Processor does create an empty flow file in case the > input is a CSV with Header but no content. If no content is available the > flow file should be ignored or at least there should be an option to ignore > empty files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655198#comment-16655198
]
ASF GitHub Bot commented on NIFI-5629:
--
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226293537
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -308,32 +321,45 @@ public boolean accept(final File file) {
};
}
-private Set performListing(final File directory, final
FileFilter filter, final boolean recurseSubdirectories) {
-Path p = directory.toPath();
-if (!Files.isWritable(p) || !Files.isReadable(p)) {
-throw new IllegalStateException("Directory '" + directory + "'
does not have sufficient permissions (i.e., not writable and readable)");
-}
-final Set queue = new HashSet<>();
-if (!directory.exists()) {
-return queue;
-}
-
-final File[] children = directory.listFiles();
-if (children == null) {
-return queue;
-}
-
-for (final File child : children) {
-if (child.isDirectory()) {
+private Set performListing(final File directory, final
FileFilter filter, final boolean recurseSubdirectories, final int batchSize) {
+try {
+if (directoryStream == null || !this.fileIterator.hasNext()) {
+final Path p = directory.toPath();
+if (!Files.isReadable(p) || !Files.isWritable(p)) {
+throw new IllegalStateException("Directory '" +
directory + "' does not have sufficient permissions (i.e., not writable and
readable)");
+}
+
+if (!directory.exists()) {
+return Collections.emptySet();
+}
+
+Stream listStream;
if (recurseSubdirectories) {
-queue.addAll(performListing(child, filter,
recurseSubdirectories));
+listStream = Files.walk(p);
--- End diff --
@markap14 hadn't we switched to using the streaming api previously and
backed this out? If so what was the problem? Or maybe that was ListFile?
> GetFile becomes slow listing vast directories
> -
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Adam
>Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226293537
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -308,32 +321,45 @@ public boolean accept(final File file) {
};
}
-private Set performListing(final File directory, final
FileFilter filter, final boolean recurseSubdirectories) {
-Path p = directory.toPath();
-if (!Files.isWritable(p) || !Files.isReadable(p)) {
-throw new IllegalStateException("Directory '" + directory + "'
does not have sufficient permissions (i.e., not writable and readable)");
-}
-final Set queue = new HashSet<>();
-if (!directory.exists()) {
-return queue;
-}
-
-final File[] children = directory.listFiles();
-if (children == null) {
-return queue;
-}
-
-for (final File child : children) {
-if (child.isDirectory()) {
+private Set performListing(final File directory, final
FileFilter filter, final boolean recurseSubdirectories, final int batchSize) {
+try {
+if (directoryStream == null || !this.fileIterator.hasNext()) {
+final Path p = directory.toPath();
+if (!Files.isReadable(p) || !Files.isWritable(p)) {
+throw new IllegalStateException("Directory '" +
directory + "' does not have sufficient permissions (i.e., not writable and
readable)");
+}
+
+if (!directory.exists()) {
+return Collections.emptySet();
+}
+
+Stream listStream;
if (recurseSubdirectories) {
-queue.addAll(performListing(child, filter,
recurseSubdirectories));
+listStream = Files.walk(p);
--- End diff --
@markap14 hadn't we switched to using the streaming api previously and
backed this out? If so what was the problem? Or maybe that was ListFile?
---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226292672
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -270,6 +280,9 @@ private FileFilter createFileFilter(final
ProcessContext context) {
return new FileFilter() {
@Override
public boolean accept(final File file) {
+if (file.isDirectory()) {
+return false;
--- End diff --
does filtering work differently with the streams? I ask because we want to
be able to support the same depth/recursive searches and this filter seems like
it would stop at dir level.
---
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655193#comment-16655193
]
ASF GitHub Bot commented on NIFI-5629:
--
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226292672
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -270,6 +280,9 @@ private FileFilter createFileFilter(final
ProcessContext context) {
return new FileFilter() {
@Override
public boolean accept(final File file) {
+if (file.isDirectory()) {
+return false;
--- End diff --
does filtering work differently with the streams? I ask because we want to
be able to support the same depth/recursive searches and this filter seems like
it would stop at dir level.
> GetFile becomes slow listing vast directories
> -
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Adam
>Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-649) Fix some compiler warnings
[ https://issues.apache.org/jira/browse/MINIFICPP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655100#comment-16655100 ] ASF GitHub Bot commented on MINIFICPP-649: -- GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/423 MINIFICPP-649 - Fix some compiler warnings Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-649 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/423.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #423 commit f1de66109a001f75785c20779c9430133d4a025b Author: Arpad Boda Date: 2018-10-18T11:50:27Z MINIFICPP-649 - Fix some compiler warnings > Fix some compiler warnings > -- > > Key: MINIFICPP-649 > URL: https://issues.apache.org/jira/browse/MINIFICPP-649 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Some warnings might be related to errors, these worth checking and fixing. > For eg.: > _/Users/aboda/work/shadow/minifi/nifi-minifi-cpp/LibExample/monitor_directory.c:43:3: > warning: implicit declaration of function 'pthread_mutex_lock' is invalid in > C99 [-Wimplicit-function-declaration]_ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #423: MINIFICPP-649 - Fix some compiler warning...
GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/423 MINIFICPP-649 - Fix some compiler warnings Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-649 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/423.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #423 commit f1de66109a001f75785c20779c9430133d4a025b Author: Arpad Boda Date: 2018-10-18T11:50:27Z MINIFICPP-649 - Fix some compiler warnings ---
[jira] [Created] (MINIFICPP-649) Fix some compiler warnings
Arpad Boda created MINIFICPP-649: Summary: Fix some compiler warnings Key: MINIFICPP-649 URL: https://issues.apache.org/jira/browse/MINIFICPP-649 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Arpad Boda Assignee: Arpad Boda Fix For: 0.6.0 Some warnings might be related to errors, these worth checking and fixing. For eg.: _/Users/aboda/work/shadow/minifi/nifi-minifi-cpp/LibExample/monitor_directory.c:43:3: warning: implicit declaration of function 'pthread_mutex_lock' is invalid in C99 [-Wimplicit-function-declaration]_ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655092#comment-16655092 ] ASF GitHub Bot commented on NIFI-3792: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/3090 I think this general subject, and processor in particular can use some usage documentation, maybe an additional documentation page would be a good addition for this? > A processor to facilitate retrying FlowFiles > > > Key: NIFI-3792 > URL: https://issues.apache.org/jira/browse/NIFI-3792 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > When dealing with processor failures in production, potentially related to > network issues, the current best practice is to retry the flowfile a couple > times before declaring it a failure[1]. This currently requires multiple > processors and penalizing the flowfile isn't possible. Also if the flow is > not fast enough, back-pressure can cause a livelocked state which requires > manual intervention. > A new processor should be added to not only encourage best practices but also > offer configuration options to deal with un-optimal situations. > [1] > https://community.hortonworks.com/questions/77336/nifi-best-practices-for-error-handling.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3090: NIFI-3792 Adding a RetryCount processor to faciliate retry...
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/3090 I think this general subject, and processor in particular can use some usage documentation, maybe an additional documentation page would be a good addition for this? ---
[jira] [Created] (NIFI-5723) PutDatabaseRecord stripps underbar character from column names
Uwe Geercken created NIFI-5723:
--
Summary: PutDatabaseRecord stripps underbar character from column
names
Key: NIFI-5723
URL: https://issues.apache.org/jira/browse/NIFI-5723
Project: Apache NiFi
Issue Type: Bug
Components: Core UI
Affects Versions: 1.7.1
Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK Runtime
Environment (build 1.8.0_181-b13)
Reporter: Uwe Geercken
In *PutDatabaseRecord.java* there is following code:
private static String normalizeColumnName(final String colName, final boolean
translateColumnNames) {
return colName == null ? null : (translateColumnNames ?
colName.toUpperCase().replace("_", "") : colName);
}
If translateColumnNames is set to true, then the underbar character is stripped
from the column names.
And it is also here:
final ColumnDescription desc =
tableSchema.getColumns().get(normalizeColumnName(fieldName,
settings.translateFieldNames));
if (desc == null && !settings.ignoreUnmappedFields) {
throw new SQLDataException("Cannot map field '" + fieldName
+ "' to any column in the database");
}
I do not see a reason why this is done. An underbar is a common delimiter in
database column names.
Specifically, I have data comming from an Oracle database which is all
uppercase and with the underbar character as the delimiter in the column names
and due to the current code, I can not update my MySQL database using the same
column names.
Trying to solve the issue in other ways does not really work as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[
https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655013#comment-16655013
]
Olivier Tarnus commented on NIFI-5029:
--
I'm actually facing the same error message, but in slightly different context:
copying flows between two registries.
I've a DEV and PROD registries and wrote a simple script using NiPyApi to
synchronize flow versions from DEV to PROD. This works perfectly until we add
embedded process groups to version control:
We have one main process group A, that embed multiple process groups B,C,D...
all process groups versions being stored in the same bucket. Any NiFi instance
connected to the DEV registry can successfully import the flows. Then we can
successfully copy the flows to the PROD registry, but we hit the above error
when importing the flow initially.
I tried to first first add A to version control with all embedded process
groups not versioned: we can then successfully import the process after a copy.
Then I started version control on embedded process groups, successfully synced
registries, but it does fail with the same error when trying to import the new
version after the copy.
This is not 100% the case described here (no copy and 2 buckets), but I think
this is related to same area of code. Let me know if I should create a separate
improvement requests.
> CLI - Handling of embedded versioned process groups during export/import
>
>
> Key: NIFI-5029
> URL: https://issues.apache.org/jira/browse/NIFI-5029
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Flow Versioning, SDLC, Tools and Build
>Reporter: Pierre Villard
>Priority: Major
>
> I'm in a situation where, in my dev environment, I have a versioned process
> group A in bucket bA containing an embedded process group B from bucket bB.
> Both A and B are versioned in the NiFi Registry of my dev environment.
> I'm using the CLI to export both A and B from my dev Registry, and then
> importing A and B in my prod Registry. The issue is that in the json
> representing A, there is a reference to B containing the url of the dev
> Registry, the bucket ID and the workflow ID representing B.
> To import B in my prod registry, I first created a bucket bB and a workflow
> B. New UUIDs have been generated for both in the prod registry. Then I did
> the import of the JSON representing B.
> Now I manually update the JSON representing A to set the UUIDs related to B
> with the new values of my prod registry. Then I created a bucket and a
> workflow for A in my prod registry, and did the import.
> Problem occurs when I try to import the process group A in my prod NiFi.
> It'll fail with an error looking like this:
> {code:java}
> #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080
> > Using a positional back-reference for 'A'
> ERROR: Error executing command 'pg-change-version' : Error updating version
> control information: The Flow Registry with ID
> 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket
> 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow
> 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code}
> I guess there are multiple ways of handling it. It could be on the CLI side
> or on the Registry side (could be related to NIFIREG-148).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #422: Minificpp 644 - C API: add support to reg...
GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/422 Minificpp 644 - C API: add support to register terminate handler (and fix build of examples) Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-644 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #422 commit a31241fa6836de23f1abaec6773f0685984c8be7 Author: Arpad Boda Date: 2018-10-09T15:56:38Z MINIFICPP-638 - C API: add unit tests commit de7569416e8a1b20a29ea27d19edb7a10a1e20ed Author: Arpad Boda Date: 2018-10-17T14:50:49Z MINIFICPP-644 - C API: add support to register terminate handler ---
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[
https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654953#comment-16654953
]
ASF GitHub Bot commented on MINIFICPP-647:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226237825
--- Diff: libminifi/test/capi/CAPITests.cpp ---
@@ -33,8 +32,14 @@
#include
#include
+static nifi_instance *create_instance_obj(const char *name =
"random_instance") {
+ nifi_port port;
+ port.port_id = "12345";
+ return create_instance("random_instance", &port);
--- End diff --
How can you set port after this?
> Deprecate functions in API that pass char* in favor of those more strongly
> typed
>
>
> Key: MINIFICPP-647
> URL: https://issues.apache.org/jira/browse/MINIFICPP-647
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[ https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654954#comment-16654954 ] ASF GitHub Bot commented on MINIFICPP-647: -- Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226238300 --- Diff: python/minifi/__init__.py --- @@ -66,9 +66,9 @@ def setBase(self, proc): def get(self, session): ff = self._minifi.get(self._instance.get_instance(),self._flow, session) if ff: --- End diff -- Python also supports ternary: ``` return FlowFile(self._minifi, ff) if ff else None ``` > Deprecate functions in API that pass char* in favor of those more strongly > typed > > > Key: MINIFICPP-647 > URL: https://issues.apache.org/jira/browse/MINIFICPP-647 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226238300 --- Diff: python/minifi/__init__.py --- @@ -66,9 +66,9 @@ def setBase(self, proc): def get(self, session): ff = self._minifi.get(self._instance.get_instance(),self._flow, session) if ff: --- End diff -- Python also supports ternary: ``` return FlowFile(self._minifi, ff) if ff else None ``` ---
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/420#discussion_r226237825
--- Diff: libminifi/test/capi/CAPITests.cpp ---
@@ -33,8 +32,14 @@
#include
#include
+static nifi_instance *create_instance_obj(const char *name =
"random_instance") {
+ nifi_port port;
+ port.port_id = "12345";
+ return create_instance("random_instance", &port);
--- End diff --
How can you set port after this?
---
[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...
GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/421 Minificpp 641 - C API: add support to register failure callback Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-641 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/421.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #421 commit b3c292c2bb0f326d2d39c6620ce4cc7d73f5a393 Author: Arpad Boda Date: 2018-10-09T15:56:38Z MINIFICPP-638 - C API: add unit tests commit 00376b84fea6497d62c995575f1377439e678480 Author: Arpad Boda Date: 2018-10-16T14:33:09Z MINIFICPP-641 - C API: add support to register failure callback ---
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654779#comment-16654779
]
ASF GitHub Bot commented on NIFI-5714:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3086
Tried few things... no luck :(
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0,
> 1.7.0, 1.7.1
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 Tried few things... no luck :( ---
