[GitHub] nifi issue #3203: NIFI-5871 ignore UUID attribute when copying flow file att...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3203 @MikeThomsen I think the test is showing that this isn't a parent/child scenario, but showing that it's the same FlowFile from start to finish. If there were a way to inject content into `MockFlowFile`, then we could build one, pass that in to `enqueue` and the test would actually work. ---
[GitHub] nifi issue #3203: NIFI-5871 ignore UUID attribute when copying flow file att...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3203 @SavtechSolutions Yes, this is a bug in the tests I think. I can see they manually provide a custom `UUID` instead of letting NiFi generate it. ---
[GitHub] nifi pull request #3195: NIFI-5862 MockRecordParser Has Bad Logic for failAf...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3195 NIFI-5862 MockRecordParser Has Bad Logic for failAfterN `MockRecordParser` has a function that allows it to throw an exception after a certain number of records have been read. This feature is not working at all, and instead the reader fails immediately without reading any records. The two unit tests that use this (TestSplitRecord.testReadFailure and TestConvertRecord.testReadFailure) both work because they route to Failure on any error in the read, but you can see the error in the console. Also, I'm working on a custom processor using this test class for testing. My processor interfaces with an external system and sends batches of data. In some cases a read failure could cause a partial load to a remote system, and we need to be able to check for this. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-5862 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3195.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3195 commit 50a34ffd638622da72c6667a4364e99055e3d505 Author: Peter Wicks Date: 2018-12-03T17:44:01Z NIFI-5862 MockRecordParser Has Bad Logic for failAfterN ---
[GitHub] nifi pull request #3188: NIFI-5829 Create Lookup Controller Services for Rec...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3188#discussion_r237530330
--- Diff:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/record/script/ScriptedRecordSetWriter.java
---
@@ -60,12 +61,16 @@ public void onEnabled(final ConfigurationContext
context) {
super.onEnabled(context);
}
-
@Override
public RecordSetWriter createWriter(ComponentLog logger, RecordSchema
schema, OutputStream out) throws SchemaNotFoundException, IOException {
+return createWriter(new HashMap<>(), logger, schema, out);
--- End diff --
Thanks, Updated.
---
[GitHub] nifi issue #3185: NIFI-5846: Redirect URL is incorrect after logout
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3185 +1 LG. ---
[GitHub] nifi issue #3185: NIFI-5846: Redirect URL is incorrect after logout
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3185 Building and Testing. I only have a Kerberos test environment, but I think it will work the same for all of the user/pass logins. ---
[GitHub] nifi pull request #3187: NIFI-5847 Added GovCloud-East region to enum list.
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3187#discussion_r237310352
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/regions/AWSRegions.java
---
@@ -18,7 +18,8 @@
public enum AWSRegions {
-GovCloud("us-gov-west-1", "AWS GovCloud (US)"),
+GovCloud("us-gov-west-1", "AWS GovCloud West (US)"),
--- End diff --
Just the display name is all I'm worried about, everything else looks fine.
---
[GitHub] nifi pull request #3185: NIFI-5846: Redirect URL is incorrect after logout
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3185#discussion_r237256616
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java
---
@@ -50,7 +50,7 @@ public void doFilter(ServletRequest request,
ServletResponse response, FilterCha
final ServletContext apiContext =
servletContext.getContext("/nifi-api");
apiContext.getRequestDispatcher("/access/knox/logout").forward(request,
response);
} else {
-((HttpServletResponse) response).sendRedirect("../login");
+((HttpServletResponse) response).sendRedirect("../nifi/login");
--- End diff --
Can you test, `sendRedirect("login");`? I think the reason it breaks is
because we are already in the `nifi` directory, but the `..` makes us leave it.
---
[GitHub] nifi pull request #3187: NIFI-5847 Added GovCloud-East region to enum list.
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3187#discussion_r237254575
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/regions/AWSRegions.java
---
@@ -18,7 +18,8 @@
public enum AWSRegions {
-GovCloud("us-gov-west-1", "AWS GovCloud (US)"),
+GovCloud("us-gov-west-1", "AWS GovCloud West (US)"),
--- End diff --
I'm going to correct myself. Some docs say (US). Some say, `(US-West)` and
`(US-East)`.
https://docs.aws.amazon.com/govcloud-us/index.html#lang/en_us
So maybe that would be better?
---
[GitHub] nifi pull request #3187: NIFI-5847 Added GovCloud-East region to enum list.
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3187#discussion_r237253874
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/regions/AWSRegions.java
---
@@ -18,7 +18,8 @@
public enum AWSRegions {
-GovCloud("us-gov-west-1", "AWS GovCloud (US)"),
+GovCloud("us-gov-west-1", "AWS GovCloud West (US)"),
--- End diff --
In the past, the display names have always been kept as what AWS shows.
AWS docs still show:
- AWS GovCloud (US-East)
- AWS GovCloud (US)
Maybe it would be better to revert the change to `AWS GovCloud West (US)`?
Then it will match the docs.
---
[GitHub] nifi issue #3188: NIFI-5829 Create Lookup Controller Services for RecordSetW...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3188 @mattyb149 My biggest concern with this PR was how many processors I had to touch for the RecordSetWriter, and how many I should touch. I decided updating all of the core/standard processors probably made sense, but did not update anything outside of that (Kafka, etc...) ---
[GitHub] nifi pull request #3188: NIFI-5829 Create Lookup Controller Services for Rec...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3188#discussion_r237252372
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/lookup/ReaderLookup.java
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Tags({"lookup", "parse", "record", "row", "reader"})
+@SeeAlso({RecordSetWriterLookup.class})
+@CapabilityDescription("Provides a RecordReaderFactory that can be used to
dynamically select another RecordReaderFactory. This service " +
+"requires an attribute named 'recordreader.name' to be passed in
when asking for a record record, and will throw an exception " +
+"if the attribute is missing. The value of 'recordreader.name'
will be used to select the RecordReaderFactory that has been " +
+"registered with that name. This will allow multiple
RecordReaderFactory's to be defined and registered, and then selected " +
+"dynamically at runtime by tagging flow files with the appropriate
'recordreader.name' attribute.")
+@DynamicProperty(name = "The ", value = "RecordWriterFactory property
value", expressionLanguageScope = ExpressionLanguageScope.NONE,
--- End diff --
Thanks. Fixed.
---
[GitHub] nifi pull request #3188: NIFI-5829 Create Lookup Controller Services for Rec...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3188 NIFI-5829 Create Lookup Controller Services for RecordSetWriter and R⦠â¦ecordReader The Reader Lookup is a fairly small change, because the reader factory was already getting the FlowFile attributes passed in. The Writer Lookup on the other hand... I had to update the createWriter method to accept attributes, and then update a lot processors to pass in attributes. `getSchema` already accepted the attributes, interestingly enough. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-5829 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3188.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3188 commit d9d659682b49505362d2ae847f5d8cac67926af9 Author: Peter Wicks Date: 2018-11-17T04:46:07Z NIFI-5829 Create Lookup Controller Services for RecordSetWriter and RecordReader ---
[GitHub] nifi pull request #3107: NIFI-5744: Put exception message to attribute while...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3107#discussion_r234310982
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteSQLRecord.java
---
@@ -350,6 +357,37 @@ public void invokeOnTriggerRecords(final Integer
queryTimeout, final String quer
assertEquals(durationTime, fetchTime + executionTime);
}
+@SuppressWarnings("unchecked")
+@Test
+public void testWithSqlExceptionErrorProcessingResultSet() throws
Exception {
+DBCPService dbcp = mock(DBCPService.class);
+Connection conn = mock(Connection.class);
+when(dbcp.getConnection(any(Map.class))).thenReturn(conn);
+when(dbcp.getIdentifier()).thenReturn("mockdbcp");
+PreparedStatement statement = mock(PreparedStatement.class);
+when(conn.prepareStatement(anyString())).thenReturn(statement);
+when(statement.execute()).thenReturn(true);
+ResultSet rs = mock(ResultSet.class);
+when(statement.getResultSet()).thenReturn(rs);
+// Throw an exception the first time you access the ResultSet,
this is after the flow file to hold the results has been created.
+when(rs.getMetaData()).thenThrow(new SQLException("test execute
statement failed"));
+
--- End diff --
I ran the tests, but this one failed because the required `RecordWriter` is
missing. I think you need:
```
MockRecordWriter recordWriter = new MockRecordWriter(null, true, -1);
runner.addControllerService("writer", recordWriter);
runner.setProperty(ExecuteSQLRecord.RECORD_WRITER_FACTORY,
"writer");
runner.enableControllerService(recordWriter);
```
Here is the error text:
> java.lang.AssertionError: Processor has 1 validation failures:
> 'Record Writer' is invalid because Record Writer is required
---
[GitHub] nifi issue #3107: NIFI-5744: Put exception message to attribute while Execut...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3107 Hey, @yjhyjhyjh0, can you rebase against master and push a new squashed commit? I think this is ready to merge if we resolve this new code conflict. ---
[GitHub] nifi issue #3075: NIFI-5604: Added property to allow empty FlowFile when no ...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3075 +1 LGTM. I tested this out with several different scenarios. After our chat on Slack, I squashed this down to a single commit under your name to make things cleaner. ---
[GitHub] nifi issue #3156: NIFI-5780 Add pre and post statements to ExecuteSQL and Ex...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3156 +1 Looks good. Built and tested. I ran into one issue that I might submit a seperate enhancement for (allowing `;` in the SQL string if they are surrounded by `'` or `"`). ---
[GitHub] nifi issue #3156: NIFI-5780 Add pre and post statements to ExecuteSQL and Ex...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3156 Sorry for the delay, been having issues with a bug that just got fixed in NIFI-5822. Will wrap up testing. ---
[GitHub] nifi issue #3131: NIFI-3229 When a queue contains only Penalized FlowFile's ...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3131 @markap14 I built unit tests, but I'm having trouble running them at scale. I temporarily checked back in the original method so I could run side-by-side speed comparisons on the same `Connectable`. But if I exceed about 100k tests my unit tests seem to go out to lunch, even if I increase heap so they don't run out. These are checked in right now to run 1 million iterations, but that has not succeeded for me... This is true of the unmodified method if run by itself also (at least on my poor little computer). ---
[GitHub] nifi pull request #3075: NIFI-5604: Added property to allow empty FlowFile w...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233524977
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -450,7 +450,33 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
-session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
+FlowFile emptyFlowFile = (fileToProcess == null) ?
session.create() : session.create(fileToProcess);
+Map attributesToAdd = new HashMap<>();
+
+attributesToAdd.put("generatetablefetch.tableName",
tableName);
+if (columnNames != null) {
+
attributesToAdd.put("generatetablefetch.columnNames", columnNames);
+}
+whereClause = maxValueClauses.isEmpty() ? "1=1" :
StringUtils.join(maxValueClauses, " AND ");
+if (StringUtils.isNotBlank(whereClause)) {
+
attributesToAdd.put("generatetablefetch.whereClause", whereClause);
+}
+final String maxColumnNames =
StringUtils.join(maxValueColumnNameList, ", ");
+if (StringUtils.isNotBlank(maxColumnNames)) {
+
attributesToAdd.put("generatetablefetch.maxColumnNames", maxColumnNames);
+}
+attributesToAdd.put("generatetablefetch.limit", null);
--- End diff --
There is no existing test in `master`, but I think in normal circumstances
`limit` actually gets written as `"null"`.
```
if ((i == numberOfFetches - 1) && useColumnValsForPaging &&
(maxValueClauses.isEmpty() || customWhereClause != null)) {
maxValueClauses.add(columnForPartitioning + "
<= " + maxValueForPartitioning);
limit = null;
}
```
This value is then written using `String.valueOf(limit)`, which is going to
translate the `null` to `"null"`.
---
[GitHub] nifi pull request #3075: NIFI-5604: Added property to allow empty FlowFile w...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233524164
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -450,7 +450,33 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
-session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
+FlowFile emptyFlowFile = (fileToProcess == null) ?
session.create() : session.create(fileToProcess);
+Map attributesToAdd = new HashMap<>();
+
+attributesToAdd.put("generatetablefetch.tableName",
tableName);
+if (columnNames != null) {
+
attributesToAdd.put("generatetablefetch.columnNames", columnNames);
+}
+whereClause = maxValueClauses.isEmpty() ? "1=1" :
StringUtils.join(maxValueClauses, " AND ");
+if (StringUtils.isNotBlank(whereClause)) {
--- End diff --
I know it was already like this, but I'm not sure it's even possible for
`whereClause` to be blank based on the logic here. Same for existing code down
below for a normal run.
---
[GitHub] nifi pull request #3075: NIFI-5604: Added property to allow empty FlowFile w...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233132498
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -434,48 +448,53 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
numberOfFetches = (partitionSize == 0) ? 1 : (rowCount
/ partitionSize) + (rowCount % partitionSize == 0 ? 0 : 1);
}
-// Generate SQL statements to read "pages" of data
-Long limit = partitionSize == 0 ? null : (long)
partitionSize;
-final String fragmentIdentifier =
UUID.randomUUID().toString();
-for (long i = 0; i < numberOfFetches; i++) {
-// Add a right bounding for the partitioning column if
necessary (only on last partition, meaning we don't need the limit)
-if ((i == numberOfFetches - 1) &&
useColumnValsForPaging && (maxValueClauses.isEmpty() || customWhereClause !=
null)) {
-maxValueClauses.add(columnForPartitioning + " <= "
+ maxValueForPartitioning);
-limit = null;
-}
+// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
+if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
+session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
--- End diff --
@mattyb149 It's not just the `fragment` attributes, I think you should add
all the standard attributes. Right now this Flowfile get's routed to success
with no attributes at all.
I would suggest refactoring this to include all the standard attributes
like `tablename`, `offset` etc... just like a real FlowFile would have. This
will make it much easier for downstream processors to use this file like
normal. I think the existing logic would cause `offset` to be `null`, which
would make sense for this scenario too.
---
[GitHub] nifi issue #3075: NIFI-5604: Added property to allow empty FlowFile when no ...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3075 Reviewing ---
[GitHub] nifi issue #3167: NIFI-5812: Marked Database processors as 'PrimaryNodeOnly'
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3167 @mattyb149 I'm in agreement, but I feel like the spirit of this ticket is still an interesting direction to look at. How do you prevent QDB or GTF from running against the same table on two nodes or two threads at the same time? Multiple threads sounds reasonable, through some kind of lock in the processor, but across multiple nodes? Would be interesting to come up with a solution. ---
[GitHub] nifi issue #3131: NIFI-3229 When a queue contains only Penalized FlowFile's ...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3131 @markap14 Sounds reasonable, I'll work on it. ---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean Your correct, patch releases are for very special circumstances. I don't believe you have any option except to build this yourself until 1.9.0 gets released; and since 1.8.0 just came out, that is going to be a bit. To keep things stable, you can checkout the 1.8.0 tag and apply your change as a cherry pick so you don't accidentally include any unexpected changes, but still an internal build. ---
[GitHub] nifi issue #3100: NIFI-5718: Implemented LineDemarcator and removed NLKBuffe...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3100 +1 LGTM Ran all of the unit tests, looks good. Built it and ran it locally, had no issues. I also ran a unit test that never got put in from the ticket that started this whole journey, https://issues.apache.org/jira/browse/NIFI-5689?focusedCommentId=16652020=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16652020, no issues. ---
[GitHub] nifi issue #3131: NIFI-3229 When a queue contains only Penalized FlowFile's ...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3131 @markap14 I was worried about the same thing, which is why the `if` statement is structured as it is. First, we do the standard check on `isActiveQueueEmpty`. This happens in the code now as you mentioned, and right now if this passes we create a writelock update the queue and call the processor. All my change does is add one additional check, but only if the queue is not empty. So as far as I can tell, I'm locking one extra time for a queue that is already going to get locked, but not locking any queues that would not already get locked. Also, because I'm updating the queue during my check, when the processor does get called the lock should not last as long as it would otherwise, as there is less work to do. So overall lock time should be affected only minimally. Thoughts? ---
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r232317090
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -59,6 +61,31 @@
+ "Note that no flow file input (attributes, e.g.) is
available for use in Expression Language constructs for these properties.")
public class DBCPConnectionPool extends AbstractControllerService
implements DBCPService {
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MIN_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MIN_IDLE = "0";
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MAX_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_IDLE = "8";
+/**
+ * Copied from private variable {@link
BasicDataSource.maxConnLifetimeMillis} in Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_CONN_LIFETIME = "-1";
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_TIME_BETWEEN_EVICTION_RUNS_MILLIS} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_EVICTION_RUN_PERIOD =
String.valueOf(-1L);
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_MIN_EVICTABLE_IDLE_TIME_MILLIS} in Commons-DBCP
2.5.0
+ */
+private static final String DEFAULT_MIN_EVICTABLE_IDLE_TIME =
String.valueOf(180L) + " millis";
--- End diff --
Testing on my local instance is looking good so far. One request, can you
change this to `30 min`? A bit easier to read as a default value that way :)
---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean Disregard. Git was misbehaving. ---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean I am getting compile errors. I had to add an import, `import org.apache.nifi.components.PropertyValue;`, otherwise your method `extractMillisWithInfinite` would not build. This is building OK for you? Maybe something got lost in the squash? ---
[GitHub] nifi issue #3100: NIFI-5718: Implemented LineDemarcator and removed NLKBuffe...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3100 @markap14, I agree that the class would be useful. Can you include a unit test that references it? As long as some piece of the PR legitimately references the class then I'm completely happy keeping it in and merging it, even with the `@Ignore` attribute, now that it's been explained to me. ---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232148571
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
Sounds good. Can you squash your next commit?
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232072895
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -94,6 +106,16 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+public static final PropertyDescriptor SQL_POST_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-post-query")
+.displayName("SQL Post-Query")
+.description("SQL post-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
"A semicolon-delimited list of queries executed after the main SQL query is
executed. Results/outputs from these queries will be suppressed if there are no
errors."
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232072719
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
@colindean I understand why it was done this way. This functionality is
being ported from the HIVE processor where the description begins: `HiveQL
pre-query to execute...`. The same for the `Note` comment below.
That doesn't mean it couldn't be clarified, but it is currently internally
consistent.
@yjhyjhyjh0 How about, "A semicolon-delimited list of queries executed
executed before the main SQL query is executed. Results/outputs from these
queries will be suppressed if there are no errors."
---
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r232065708
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
---
@@ -93,6 +94,155 @@ public void testMaxWait() throws
InitializationException {
runner.assertValid(service);
}
+/**
+ * Checks validity of idle limit and time settings including a default
+ */
+@Test
+public void testIdleConnectionsSettings() throws
InitializationException {
+final TestRunner runner =
TestRunners.newTestRunner(TestProcessor.class);
+final DBCPConnectionPool service = new DBCPConnectionPool();
+runner.addControllerService("test-good1", service);
+
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// set embedded Derby database connection url
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
+runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD,
"testerp");
+runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
+runner.setProperty(service, DBCPConnectionPool.MAX_WAIT_TIME,
"-1");
+runner.setProperty(service, DBCPConnectionPool.MAX_IDLE, "2");
+runner.setProperty(service, DBCPConnectionPool.MAX_CONN_LIFETIME,
"1 secs");
+runner.setProperty(service,
DBCPConnectionPool.EVICTION_RUN_PERIOD, "1 secs");
+runner.setProperty(service,
DBCPConnectionPool.MIN_EVICTABLE_IDLE_TIME, "1 secs");
+runner.setProperty(service,
DBCPConnectionPool.SOFT_MIN_EVICTABLE_IDLE_TIME, "1 secs");
+
+runner.enableControllerService(service);
+runner.assertValid(service);
+}
+
+@Test
+public void testMinIdleCannotBeNegative() throws
InitializationException {
+final TestRunner runner =
TestRunners.newTestRunner(TestProcessor.class);
+final DBCPConnectionPool service = new DBCPConnectionPool();
+runner.addControllerService("test-good1", service);
+
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// set embedded Derby database connection url
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
+runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD,
"testerp");
+runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
+runner.setProperty(service, DBCPConnectionPool.MAX_WAIT_TIME,
"-1");
+runner.setProperty(service, DBCPConnectionPool.MIN_IDLE, "-1");
+
+runner.assertNotValid(service);
+}
+
+/**
+ * Checks to ensure that settings have been passed down into the DBCP
+ */
+@Test
+public void testIdleSettingsAreSet() throws InitializationException,
SQLException {
+final TestRunner runner =
TestRunners.newTestRunner(TestProcessor.class);
+final DBCPConnectionPool service = new DBCPConnectionPool();
+runner.addControllerService("test-good1", service);
+
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// set embedded Derby database connection url
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
+runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD,
"testerp");
+runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
+runner.setProperty(service, DBCPConnectionPool.MAX_WAIT_TIME,
"-1");
+runner.setProperty(service, DBCPConnectionPool.MAX_IDLE, "6");
+runner.setProperty(service, DBCPConnectionPool.MIN_IDLE, "4");
+runner.setPropert
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231759360
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL pre-query")
--- End diff --
They are capitalized in the HIVE processors, so this would match them.
---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 My thought was you could look for the idle count and see if it was 0, 8, etc... based on the config, and not worry about testing the timeouts for now. ---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean I know Matt responded right after I did before, what are your thoughts on working on enabling unit tests by exposing the idle/active connection counts? ---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231596320
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL pre-query")
--- End diff --
Can you make this `SQL Pre-Query`? Same for `SQL Post-Query`.
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231603266
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteSQLRecord.java
---
@@ -350,6 +352,141 @@ public void invokeOnTriggerRecords(final Integer
queryTimeout, final String quer
assertEquals(durationTime, fetchTime + executionTime);
}
+@Test
+public void testPreQuery() throws Exception {
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// load test data to database
+final Connection con = ((DBCPService)
runner.getControllerService("dbcp")).getConnection();
+Statement stmt = con.createStatement();
+
+try {
+stmt.execute("drop table TEST_NULL_INT");
+} catch (final SQLException sqle) {
+}
+
+stmt.execute("create table TEST_NULL_INT (id integer not null,
val1 integer, val2 integer, constraint my_pk primary key (id))");
+
+runner.setIncomingConnection(true);
+runner.setProperty(ExecuteSQL.SQL_PRE_QUERY, "insert into
TEST_NULL_INT values(1,2,3);insert into TEST_NULL_INT values(4,5,6)");
--- End diff --
I know these tests were easy to write with insert/delete, but they don't
really show why this feature is needed.
The idea is that we need to configure the DBCP connection in some way, but
that putting two ExecuteSQL processors together might cause us to use different
connections from the connection pool.
Could you try changing them to set Derby session properties?Maybe something
from here? https://db.apache.org/derby/docs/10.1/ref/rrefsetdbpropproc.html
This one looked good, it tells Derby to capture runtime statistics for the
current connection, and then turns them back off after, so a legitimate use
case.
Pre: `CALL SYSCS_UTIL.SYSCS_SET_RUNTIMESTATISTICS(1)`
Post: `CALL SYSCS_UTIL.SYSCS_SET_RUNTIMESTATISTICS(0)`
---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean I don't have a good answer yet, hoping I can get some input from other developers. But I was thinking about unit tests, and what you could do to help make this code change unit testable. What if you exposed the number of active and idle connections in the connection pool as properties on the DBCPConnectinoPool? These are available by calling `getNumActive()` and `getNumIdle()`. Or you could call `listAllObjects()` and get back the full pool on the `dataSource` object. With these numbers it would be possible to test at least the min/max connection settings, and maybe more. ---
[GitHub] nifi pull request #3100: NIFI-5718: Implemented LineDemarcator and removed N...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3100#discussion_r231534519
--- Diff:
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+public class RepeatingInputStream extends InputStream {
--- End diff --
@markap14 Thoughts?
---
[GitHub] nifi pull request #3128: NIFI-5788: Introduce batch size limit in PutDatabas...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3128#discussion_r231153599
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java
---
@@ -669,11 +685,20 @@ private void executeDML(ProcessContext context,
ProcessSession session, FlowFile
}
}
ps.addBatch();
+if (++currentBatchSize == batchSize) {
--- End diff --
True, I missed that override before, but I see it now. So definitely less
valuable, the only thing it would provide would be troubleshooting guidance,
"your bad data is roughly in this part of the file". Probably not worth it.
Thanks!
---
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r231152202
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -164,6 +161,71 @@ public ValidationResult validate(final String subject,
final String input, final
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor MIN_IDLE = new
PropertyDescriptor.Builder()
+.name("Minimum Idle Connections")
+.description("The minimum number of connections that can
remain idle in the pool, without extra ones being " +
+"created, or zero to create none.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor MAX_IDLE = new
PropertyDescriptor.Builder()
+.name("Max Idle Connections")
+.description("The maximum number of connections that can
remain idle in the pool, without extra ones being " +
+"released, or negative for no limit.")
+.defaultValue("8")
--- End diff --
@mattyb149 If you have a second, I'd appreciate your thoughts on this.
---
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r230927408
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -164,6 +161,71 @@ public ValidationResult validate(final String subject,
final String input, final
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor MIN_IDLE = new
PropertyDescriptor.Builder()
+.name("Minimum Idle Connections")
--- End diff --
I saw your comment about `.name`. I know the existing properties in
DBCPConnectionPool do not use `displayName`, but that is only because they are
from before the change in standards.
Can you move the new `name` property values to `displayName`, and set
`name` to something like `dbcp-min-idle-conns`?
---
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r230921657
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -164,6 +161,71 @@ public ValidationResult validate(final String subject,
final String input, final
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor MIN_IDLE = new
PropertyDescriptor.Builder()
+.name("Minimum Idle Connections")
+.description("The minimum number of connections that can
remain idle in the pool, without extra ones being " +
+"created, or zero to create none.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor MAX_IDLE = new
PropertyDescriptor.Builder()
+.name("Max Idle Connections")
+.description("The maximum number of connections that can
remain idle in the pool, without extra ones being " +
+"released, or negative for no limit.")
+.defaultValue("8")
--- End diff --
Setting this to `8` feels so weird, even though it is legitimately the
default value in the DBCP library.
---
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r230920291
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -164,6 +161,71 @@ public ValidationResult validate(final String subject,
final String input, final
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor MIN_IDLE = new
PropertyDescriptor.Builder()
+.name("Minimum Idle Connections")
+.description("The minimum number of connections that can
remain idle in the pool, without extra ones being " +
+"created, or zero to create none.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.sensitive(false)
--- End diff --
You don't need `sensitive(false)`. I think you'r safe to remove it from
these new properties.
---
[GitHub] nifi pull request #3128: NIFI-5788: Introduce batch size limit in PutDatabas...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3128#discussion_r230917511
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java
---
@@ -669,11 +685,20 @@ private void executeDML(ProcessContext context,
ProcessSession session, FlowFile
}
}
ps.addBatch();
+if (++currentBatchSize == batchSize) {
--- End diff --
Would it be beneficial to capture `currentBatchSize*batchIndex`, with
`batchIndex` being incremented only after a successful call to `executeBatch()`
as an attribute? My thinking is, if you have a failure, and only part of a
batch was loaded, you could store how many rows were loaded successfully as an
attribute?
---
[GitHub] nifi pull request #3128: NIFI-5788: Introduce batch size limit in PutDatabas...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3128#discussion_r230916140
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java
---
@@ -265,6 +265,17 @@
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+static final PropertyDescriptor BATCH_SIZE = new
PropertyDescriptor.Builder()
+.name("put-db-record-batch-size")
+.displayName("Bulk Size")
+.description("Specifies batch size for INSERT and UPDATE
statements. This parameter has no effect for other statements specified in
'Statement Type'."
++ " Non-positive value has the effect of infinite bulk
size.")
+.defaultValue("-1")
--- End diff --
I agree that `0` should be the default, and would replicate the current
behavior of the processor, "All records in one batch".
---
[GitHub] nifi pull request #3131: NIFI-3229 When a queue contains only Penalized Flow...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3131 NIFI-3229 When a queue contains only Penalized FlowFile's the next pr⦠â¦ocessor Tasks/Time statistics becomes extremely large ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-3229 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3131.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3131 commit 3a42c7b671972001ed912ef8e907d5b8658554e9 Author: patricker Date: 2018-11-05T18:33:11Z NIFI-3229 When a queue contains only Penalized FlowFile's the next processor Tasks/Time statistics becomes extremely large ---
[GitHub] nifi issue #3107: NIFI-5744: Put exception message to attribute while Execut...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3107 @mattyb149 Not sure if you've seen my latest reply to the email chain, but it looks like this is already a standard pattern used in ~12 other processors. Would love to see the discussion come to a conclusion in the email chain though. ---
[GitHub] nifi issue #3074: NIFI-5601: Add fragment.* attributes to GenerateTableFetch
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3074 +1 LG - Reviewed, ran tests, built and ran the changes locally against a larger table. ---
[GitHub] nifi pull request #3100: NIFI-5718: Implemented LineDemarcator and removed N...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3100#discussion_r229750511
--- Diff:
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+public class RepeatingInputStream extends InputStream {
--- End diff --
It would probably be best to remove it from this specific PR and include it
in a future one when needed.
Generally, aren't we trying to avoid more `@ignore` tests? Otherwise I'd
say just include your previous tests and call it good.
---
[GitHub] nifi pull request #3100: NIFI-5718: Implemented LineDemarcator and removed N...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3100#discussion_r229500969
--- Diff:
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+public class RepeatingInputStream extends InputStream {
--- End diff --
Where is this class being used? I can't find any reference to it.
---
[GitHub] nifi pull request #3107: NIFI-5744: Put exception message to attribute while...
Github user patricker commented on a diff in the pull request: https://github.com/apache/nifi/pull/3107#discussion_r228384912 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java --- @@ -83,6 +83,8 @@ @WritesAttribute(attribute = "executesql.query.fetchtime", description = "Duration of the result set fetch time in milliseconds"), @WritesAttribute(attribute = "executesql.resultset.index", description = "Assuming multiple result sets are returned, " + "the zero based index of this result set."), +@WritesAttribute(attribute = "executesql.error.message", description = "If processing an incoming flow file causes " ++ "an Exception, the flow file is routed to failure and this attribute is set to the exception message."), --- End diff -- Can you capitalize "flow file" to "Flow File" in both processors? ---
[GitHub] nifi issue #3068: NIFI-5693 add support for multiple attachments in PutEmail...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3068 No rush, but am ready to review it whenever your ready. ---
[GitHub] nifi issue #3078: NIFI-4805 Allow Delayed Transfer
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 @joewitt It's not done, more of a rough draft, but take a look at my latest commit and let me know if this is more in line with what you were thinking. ---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343199
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226341991
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226339913
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
--- End diff --
I really like `Retry` as the processor name. I think it would be more
intuitive for most users.
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226344163
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343311
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340692
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
--- End diff --
All of your `.name(" ")` need to be changed. It's good that you have
separate `name` and `displayName`, but it should be more like, `retry-limit`.
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340682
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
--- End diff --
Suggestion: `retry-penalize-flowfile`
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226341579
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
--- End diff --
What about changing the scope to `FLOWFILE_ATTRIBUTES` so it can be based
on a per `FlowFile` basis?
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226345973
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
--- End diff --
It came up in discussions on the Jira for the Delay processor that having a
fixed retry count attribute could be tricky. How do you handle Retry attribute
contamination? If this gets set in an upstream flow, and then the same FlowFile
goes through a new Retry loop later on in the Flow, won't this attribute cause
the FlowFile to behave as though it has already been retried?
The workaround for this we came up with was to append the Processor UUID to
the attribute name. Thoughts?
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226340820
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
--- End diff --
Suggestion: `retry-warn-overlimit`
---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226347867
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226347295
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3090#discussion_r226343019
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RetryCount.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.nifi.processors.standard.RetryCount.RETRY_COUNT_ATTRIBUTE_KEY;
+import static
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_REQUIRED;
+import static
org.apache.nifi.expression.ExpressionLanguageScope.VARIABLE_REGISTRY;
+
+/**
+ * Created by jpercivall on 3/17/17.
+ */
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(INPUT_REQUIRED)
+@Tags({"penalize", "retry", "penalize"})
+@CapabilityDescription("Used to facilitate retrying a FlowFile a set
number of times before considering the FlowFile 'failed'. Can also be used as a
utility to penalize FlowFiles.")
+@WritesAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+@ReadsAttribute(attribute = RETRY_COUNT_ATTRIBUTE_KEY, description = "The
number of times this FlowFile has been routed to 'Retry'.")
+public class RetryCount extends AbstractProcessor {
+
+public static final String RETRY_COUNT_ATTRIBUTE_KEY =
"retryCountProcessor.timesRetried";
+
+public static final PropertyDescriptor PROP_RETRY_LIMIT = new
PropertyDescriptor.Builder()
+.name("Retry limit")
+.displayName("Retry limit")
+.description("The number of times to retry the FlowFile before
considering it 'failed' and routing it to 'over limit'.")
+.expressionLanguageSupported(VARIABLE_REGISTRY)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.defaultValue("3")
+.build();
+
+public static final PropertyDescriptor PROP_PENALIZE_FLOWFILE = new
PropertyDescriptor.Builder()
+.name("Penalize FlowFile")
+.displayName("Penalize FlowFile")
+.description("If true then the FlowFiles routed to 'retry'
will be penalized.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor PROP_WARN_ON_OVER_LIMIT = new
PropertyDescriptor.Builder()
+.name("Warn on 'over limit'")
+.displayName("Warn on 'over limit'")
+
[GitHub] nifi pull request #3091: NIFI-5722 Expose Penalty Remaining Duration
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3091 NIFI-5722 Expose Penalty Remaining Duration ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-5722 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3091.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3091 commit 064d51ceedb9fb2056b9faeb0efa25bd10098ba8 Author: patricker Date: 2018-10-18T02:40:02Z NIFI-5722 Expose Penalty Remaining Duration ---
[GitHub] nifi issue #3078: NIFI-4805 Allow Delayed Transfer
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 I'm confused by your message @joewitt. I understand, "Yes, make it a delay processor". But then you mention `penalize` a lot of times. My thought was if it's a delay processor then it wouldn't ever call Penalize, but just not pass FlowFile's through to the next queue until the appropriate amount of time had passed. ---
[GitHub] nifi issue #3078: NIFI-4805 Allow Delayed Transfer
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 @joewitt Originally I was going to write a processor that looked at how long files had been queued and only moved them forward after the appropriate time had elapsed, a true Delay processor. But when I found this existing JIRA ticket, and since two other developers had discussed it quite a bit, I went with building a Penalize processor. The use case is absolutely causing a delay in the Flow. Would you prefer I go back to something more like my original approach and only let FlowFile's through based on their Queued times? This way would be more configurable, and we could use an optional attribute/EL to decide how long they should be forced to delay. ---
[GitHub] nifi pull request #3078: NIFI-4805 Allow Delayed Transfer
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3078#discussion_r226012968
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java
---
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"penalty", "penalize", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Penalizes a FlowFile.")
+@WritesAttributes({
+@WritesAttribute(attribute = "penalization.count.{processor
uuid}", description = "How many times this processor has penalized this
FlowFile.")
+})
+
+public class PenalizeFlowFile extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("Successfully penalized FlowFile").build();
+
+private List properties;
+private Set relationships;
+
+@Override
+protected void init(final ProcessorInitializationContext context) {
+final Set relationships = new HashSet<>();
+relationships.add(REL_SUCCESS);
+this.relationships = Collections.unmodifiableSet(relationships);
+}
+@Override
+public Set getRelationships() {
+return relationships;
+}
+
+@Override
+public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+// Track how many times a FlowFile passes through this processor
to better support the Retry use case
+final String retryAttrName = "penalization.count." +
this.getIdentifier();
+final String initialCount = flowFile.getAttribute(retryAttrName);
+long cnt = 0;
+if(initialCount != null) {
+cnt = Long.parseLong(initialCount);
+}
+
+cnt++;
+
+flowFile = session.putAttribute(flowFile, retryAttrName,
Long.toString(cnt));
--- End diff --
I agree. I've updated the PR. All of those other features related to retry
should be handled under NIFI-3792. It sounds like there is a processor setup,
he's planning to PR it soon.
---
[GitHub] nifi issue #3085: NIFI-5711 NLKBufferedReader appears extend and copy portio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3085 @markap14 Thanks Mark. One thing I didn't test, and I don't know if it's covered by the current test suite, is very long lines. I didn't think about it until this morning, but with the way I re-wrote this, if a file has a single very long line I don't think it will fail when the length of the line exceeds the buffer; which is what the docs say should happen. This is because I just call `read()`, and read handles keeping the buffer filled... ---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225997511
--- Diff:
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
---
@@ -1422,10 +1424,15 @@ public String
getDefaultBackPressureDataSizeThreshold() {
public static NiFiProperties createBasicNiFiProperties(final String
propertiesFilePath, final Map additionalProperties) {
final Map addProps = (additionalProperties ==
null) ? Collections.EMPTY_MAP : additionalProperties;
final Properties properties = new Properties();
-final String nfPropertiesFilePath = (propertiesFilePath == null)
+String nfPropertiesFilePath = (propertiesFilePath == null)
? System.getProperty(NiFiProperties.PROPERTIES_FILE_PATH)
: propertiesFilePath;
if (nfPropertiesFilePath != null) {
+try {
--- End diff --
This sounds like it could be a legitimate bug, but it would need to be
submitted separately as it's not related to fixing `GetFile`. I run on Windows,
but I don't have spaces in the path, maybe switch to a new path with no spaces
for now?
---
[GitHub] nifi pull request #3085: NIFI-5711 NLKBufferedReader appears extend and copy...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3085 NIFI-5711 NLKBufferedReader appears extend and copy portions of the J⦠â¦DK BufferedReader Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-5711 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3085.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3085 commit d0d9f0172839e62cf624fe1801f8f532d42a783d Author: patricker Date: 2018-10-16T21:56:55Z NIFI-5711 NLKBufferedReader appears extend and copy portions of the JDK BufferedReader ---
[GitHub] nifi issue #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3033 @adyoun2 Looks like you are having some issues with your Git, I've been there many times... But I'm sure we can get it working. Make sure that after you rebase your Github fork you push the changes back to Github. Right now Github thinks you made a lot of extra changes to your branch that you didn't (thus the ~500 changed files). Also, when you squash, don't squash your changes in with another users, just squash your own changes. ---
[GitHub] nifi pull request #3078: NIFI-4805 Allow Delayed Transfer
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3078 NIFI-4805 Allow Delayed Transfer Went looking for this processor, found a work around, and found that others had already discussed building one throughout 2018 (see ticket/email thread from today, October 15th). Happy to make changes, feedback welcome. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-4805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3078.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3078 commit badfe8651651e5cefe1d2447458ff43be7ba0cdc Author: patricker Date: 2018-10-16T02:55:20Z NIFI-4805 Allow Delayed Transfer ---
[GitHub] nifi pull request #3077: NIFI-5704 TestStandardProcessSession testBatchQueue...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3077 NIFI-5704 TestStandardProcessSession testBatchQueuedHaveSameQueuedTim⦠â¦e is a brittle test ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-5704 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3077.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3077 commit 6c7e38ebcedd5de518ca27c1a356ef5c06bc39b3 Author: Peter Wicks Date: 2018-10-16T02:34:10Z NIFI-5704 TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test ---
[GitHub] nifi issue #2022: NIFI-4200 - Initial commit for a ControlNiFi processor
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2022 @pvillard31 I was reviewing this today, but wanted to make sure you still wanted to move forward. No reason not to that I can see, it's just been dormant a long time. ---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r224948420
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
---
@@ -151,6 +151,90 @@ public void testFilePickedUp() throws IOException {
assertEquals(absTargetPathStr, absolutePath);
}
+@Test
+public void testFilePickedUpArrivalSinceListing() throws IOException {
+final File directory = new File("target/test/data/in");
+deleteDirectory(directory);
+assertTrue("Unable to create test data directory " +
directory.getAbsolutePath(), directory.exists() || directory.mkdirs());
+
+final TestRunner runner = TestRunners.newTestRunner(new GetFile());
+runner.setProperty(GetFile.DIRECTORY, directory.getAbsolutePath());
+runner.setProperty(GetFile.KEEP_SOURCE_FILE, "false");
+
+final File inFile = new File("src/test/resources/hello.txt");
+{
+ final Path inPath = inFile.toPath();
+ final File destFile = new File(directory, inFile.getName());
+ final Path targetPath = destFile.toPath();
+ Files.copy(inPath, targetPath);
+}
+{
+ final Path inPath = inFile.toPath();
+ final File destFile = new File(directory, inFile.getName() +
"1");
+ final Path targetPath = destFile.toPath();
+ Files.copy(inPath, targetPath);
+
+ runner.run(1);
+}
+final Path inPath = inFile.toPath();
+final File destFile = new File(directory, inFile.getName() + "2");
+final Path targetPath = destFile.toPath();
+final Path absTargetPath = targetPath.toAbsolutePath();
+final String absTargetPathStr = absTargetPath.getParent() + "/";
+Files.copy(inPath, targetPath);
+
+runner.run(1);
+runner.run(1);
+
+runner.assertAllFlowFilesTransferred(GetFile.REL_SUCCESS, 3);
+final List successFiles =
runner.getFlowFilesForRelationship(GetFile.REL_SUCCESS);
+successFiles.get(0).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+successFiles.get(1).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+successFiles.get(2).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+
+final String path = successFiles.get(2).getAttribute("path");
+assertEquals("/", path);
+final String absolutePath =
successFiles.get(2).getAttribute(CoreAttributes.ABSOLUTE_PATH.key());
+assertEquals(absTargetPathStr, absolutePath);
+}
+
+@Test
--- End diff --
On further testing of this test case, weird results. Does this test work
for you? I get a mix of Actual counts. I ran this test on the full code and get
Actual 9996. I took out the limit, and got an actual of . Not really sure
what's going on.
---
[GitHub] nifi issue #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3033 @adyoun2 Also, to retrigger the build, I found online that you can close and re-open your PR (no need for a new PR, just re-open). That _should_ re-trigger the checks. ---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r224946482
--- Diff:
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -458,6 +458,9 @@ public MockFlowFile importFrom(final Path path, final
boolean keepSourceFile, Fl
newFlowFile.setData(baos.toByteArray());
newFlowFile = putAttribute(newFlowFile,
CoreAttributes.FILENAME.key(), path.getFileName().toString());
+if (!keepSourceFile) {
--- End diff --
Was this included on accident? Does not seem relevant to your ticket.
---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r224946495
--- Diff:
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
---
@@ -1422,10 +1424,15 @@ public String
getDefaultBackPressureDataSizeThreshold() {
public static NiFiProperties createBasicNiFiProperties(final String
propertiesFilePath, final Map additionalProperties) {
final Map addProps = (additionalProperties ==
null) ? Collections.EMPTY_MAP : additionalProperties;
final Properties properties = new Properties();
-final String nfPropertiesFilePath = (propertiesFilePath == null)
+String nfPropertiesFilePath = (propertiesFilePath == null)
? System.getProperty(NiFiProperties.PROPERTIES_FILE_PATH)
: propertiesFilePath;
if (nfPropertiesFilePath != null) {
+try {
--- End diff --
Was this included on accident? Does not seem relevant to your ticket.
---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r224946474
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/test/java/org/apache/nifi/documentation/DocGeneratorTest.java
---
@@ -72,7 +74,12 @@ public void testProcessorLoadsNarResources() throws
IOException, ClassNotFoundEx
}
private NiFiProperties loadSpecifiedProperties(final String
propertiesFile, final String key, final String value) {
-String file =
DocGeneratorTest.class.getResource(propertiesFile).getFile();
+String file = propertiesFile;
--- End diff --
Was this included on accident? Does not seem relevant to your ticket.
---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r224947166
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
---
@@ -151,6 +151,90 @@ public void testFilePickedUp() throws IOException {
assertEquals(absTargetPathStr, absolutePath);
}
+@Test
+public void testFilePickedUpArrivalSinceListing() throws IOException {
+final File directory = new File("target/test/data/in");
+deleteDirectory(directory);
+assertTrue("Unable to create test data directory " +
directory.getAbsolutePath(), directory.exists() || directory.mkdirs());
+
+final TestRunner runner = TestRunners.newTestRunner(new GetFile());
+runner.setProperty(GetFile.DIRECTORY, directory.getAbsolutePath());
+runner.setProperty(GetFile.KEEP_SOURCE_FILE, "false");
+
+final File inFile = new File("src/test/resources/hello.txt");
+{
+ final Path inPath = inFile.toPath();
+ final File destFile = new File(directory, inFile.getName());
+ final Path targetPath = destFile.toPath();
+ Files.copy(inPath, targetPath);
+}
+{
+ final Path inPath = inFile.toPath();
+ final File destFile = new File(directory, inFile.getName() +
"1");
+ final Path targetPath = destFile.toPath();
+ Files.copy(inPath, targetPath);
+
+ runner.run(1);
+}
+final Path inPath = inFile.toPath();
+final File destFile = new File(directory, inFile.getName() + "2");
+final Path targetPath = destFile.toPath();
+final Path absTargetPath = targetPath.toAbsolutePath();
+final String absTargetPathStr = absTargetPath.getParent() + "/";
+Files.copy(inPath, targetPath);
+
+runner.run(1);
+runner.run(1);
+
+runner.assertAllFlowFilesTransferred(GetFile.REL_SUCCESS, 3);
+final List successFiles =
runner.getFlowFilesForRelationship(GetFile.REL_SUCCESS);
+successFiles.get(0).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+successFiles.get(1).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+successFiles.get(2).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+
+final String path = successFiles.get(2).getAttribute("path");
+assertEquals("/", path);
+final String absolutePath =
successFiles.get(2).getAttribute(CoreAttributes.ABSOLUTE_PATH.key());
+assertEquals(absTargetPathStr, absolutePath);
+}
+
+@Test
--- End diff --
I still have more testing to do, but so far this test looks good at
demonstrating the issue. When i run this with the unmodified GetFile it runs
out steam around 9993 files.
---
[GitHub] nifi issue #3068: NIFI-5693 add support for multiple attachments in PutEmail...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3068 I like your #2 approach, as shown in this PR. I haven't tested it yet, but can you provide a better description on the property? Right now there is no way for a user to know that TAR is the expected merge format (instead of, for example, the NiFi FlowFile packing format) ---
[GitHub] nifi issue #2846: NIFI-5381 Add GetSFTP and PutSFTP Unit Tests
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2846 @joewitt Thanks Joe, I've removed the NOTICE file change. ---
[GitHub] nifi issue #2231: NIFI-4521 MS SQL CDC Processor
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2231 @mattyb149 Ready when you are. I have plans to add in new functionality to support 'SQL Server Change Tracking', which is a simpler form of change tracking. Would really like to see this PR merged in prior to making any future changes. ---
[GitHub] nifi pull request #2231: NIFI-4521 MS SQL CDC Processor
Github user patricker commented on a diff in the pull request: https://github.com/apache/nifi/pull/2231#discussion_r205896795 --- Diff: nifi-assembly/pom.xml --- @@ -637,6 +637,12 @@ language governing permissions and limitations under the License. --> 1.7.0-SNAPSHOT nar + +org.apache.nifi +nifi-cdc-mssql-nar +1.7.0-SNAPSHOT --- End diff -- Rebased and version numbers updated. I had some files that I had failed to update from 1.6.0 to 1.7.0, so it wasn't even building correctly as it was... Builds and seems OK now. ---
[GitHub] nifi issue #2899: NIFI-4535 Only update Page Title to root flow name when us...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2899 @mcgilman I played around with permissions, and I think this one covers all the cases you mentioned. It will loop up the breadcrumb tree, which you can do even if you have no read access, and then checks at the root level if read access exists. ---
[GitHub] nifi pull request #2899: NIFI-4535 Only update Page Title to root flow name ...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2899#discussion_r203389775
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-canvas.js
---
@@ -167,12 +167,14 @@
nfNgBridge.injector.get('breadcrumbsCtrl').resetScrollPosition();
// set page title to the name of the root processor group
-var rootBreadcrumb = breadcrumb;
-while(rootBreadcrumb.parentBreadcrumb != null) {
-rootBreadcrumb = rootBreadcrumb.parentBreadcrumb
-}
+if (breadcrumb.permissions.canRead) {
+var rootBreadcrumb = breadcrumb;
+while(rootBreadcrumb.parentBreadcrumb != null) {
+rootBreadcrumb = rootBreadcrumb.parentBreadcrumb
+}
-document.title = rootBreadcrumb.breadcrumb.name;
+document.title = rootBreadcrumb.breadcrumb.name;
--- End diff --
I really appreciate your deep understanding of these more corner case
security issues!
I'm thinking on it, will hope to have it updated Friday.
---
[GitHub] nifi pull request #2899: NIFI-4535 Only update Page Title to root flow name ...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/2899 NIFI-4535 Only update Page Title to root flow name when user has perm⦠â¦ission. Updated to handle scenario where user does not have read access. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-4535 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2899.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2899 commit 0bb229b3e0d26fdef2fc3f6287f3e228de6b21d2 Author: patricker Date: 2018-07-16T19:03:05Z NIFI-4535 Only update Page Title to root flow name when user has permission. ---
[GitHub] nifi issue #2893: NIFI-5329 Made a client service an optional source of conn...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2893 @MikeThomsen, you typo'd the JIRA ticket number associated with this PR. (NIFI-5329 vs NIFI-5239). ---
[GitHub] nifi issue #2429: Allow delayed transfer
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2429 @alfonz19 are you still working on this? ---
[GitHub] nifi pull request #2868: NIFI-5312 QueryDatabaseTable updates state when an ...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2868#discussion_r201466437
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
---
@@ -543,5 +550,10 @@ public void processRow(ResultSet resultSet) throws
IOException {
throw new IOException(e);
}
}
+
+@Override
+public void rollbackStateChanges() {
+this.newColMap = this.originalState;
--- End diff --
Unit test is passing now (thanks again for that). Tried to avoid the word
`save` in the updated code, so as not to imply that state is being saved at
that point in time.
---
[GitHub] nifi issue #2874: NIFI-4279 PutDatabaseRecord and ConvertJSONToSQL stream ha...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2874 @lfrancke I considered adding a link to the article or ticket, but it's not something that I see done frequently in NiFi; if you feel it makes sense to do so in this scenario I gladly will. Thanks for the additional research, good to see other people resolved the same issue 13 years ago in another project :) ---
[GitHub] nifi pull request #2868: NIFI-5312 QueryDatabaseTable updates state when an ...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2868#discussion_r201461138
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
---
@@ -543,5 +550,10 @@ public void processRow(ResultSet resultSet) throws
IOException {
throw new IOException(e);
}
}
+
+@Override
+public void rollbackStateChanges() {
+this.newColMap = this.originalState;
--- End diff --
Yeah... forgot that state wasn't being retrieved from the Max Value
tracker. Thanks for the unit test, this demonstrates the error well. Will
include it in my next update.
---
