[jira] [Commented] (NIFI-908) Add Support for SSL in JMS Processors
[ https://issues.apache.org/jira/browse/NIFI-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973713#comment-14973713 ] Tony Kurc commented on NIFI-908: got patch to apply. removed the reordering of the imports. made slight to the URI handling and exceptions thrown. tested with activemq with ssl:// uri, tcp:// uri, failover uri with two ssl servers, two tcp servers, and (one tcp server and one ssl server). > Add Support for SSL in JMS Processors > - > > Key: NIFI-908 > URL: https://issues.apache.org/jira/browse/NIFI-908 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Luke Williamson >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix, jms, newbie > Fix For: 0.4.0 > > Attachments: nifi-908.patch.zip, nifi-908.patch.zip, > nifi-908.patch.zip > > > Add support for SSL in the JMS processors. > ActiveMQ is the only provider and in order to do this it just needs to use > the SSL connection factory instead of the normal one when the connection is > using the SSL protocol. > This should also support SSL in failover mode. So the url's for SSL > connections would be something like > ssl://127.0.0.1:65189 > failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-908) Add Support for SSL in JMS Processors
[ https://issues.apache.org/jira/browse/NIFI-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973710#comment-14973710 ] ASF subversion and git services commented on NIFI-908: -- Commit 26edab3185008f1d34647ff3c11ba8b87815de02 in nifi's branch refs/heads/master from [~luke3145] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=26edab3 ] NIFI-908 Added support for SSL in JMS connections. - Added SSL context to JMS producer and consumer processors - Tony Kurc Amended patch to check SSL need by scheme and exception consistency Reviewed by Tony Kurc (tk...@apache.org) > Add Support for SSL in JMS Processors > - > > Key: NIFI-908 > URL: https://issues.apache.org/jira/browse/NIFI-908 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Luke Williamson >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix, jms, newbie > Fix For: 0.4.0 > > Attachments: nifi-908.patch.zip, nifi-908.patch.zip, > nifi-908.patch.zip > > > Add support for SSL in the JMS processors. > ActiveMQ is the only provider and in order to do this it just needs to use > the SSL connection factory instead of the normal one when the connection is > using the SSL protocol. > This should also support SSL in failover mode. So the url's for SSL > connections would be something like > ssl://127.0.0.1:65189 > failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
nifi git commit: NIFI-908 Added support for SSL in JMS connections. - Added SSL context to JMS producer and consumer processors - Tony Kurc Amended patch to check SSL need by scheme and exception cons
Repository: nifi
Updated Branches:
refs/heads/master 8d2f9bc64 -> 26edab318
NIFI-908 Added support for SSL in JMS connections.
- Added SSL context to JMS producer and consumer processors
- Tony Kurc Amended patch to check SSL need by scheme and exception consistency
Reviewed by Tony Kurc (tk...@apache.org)
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/26edab31
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/26edab31
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/26edab31
Branch: refs/heads/master
Commit: 26edab3185008f1d34647ff3c11ba8b87815de02
Parents: 8d2f9bc
Author: Luke Williamson
Authored: Mon Oct 26 00:29:05 2015 -0400
Committer: Tony Kurc
Committed: Mon Oct 26 00:29:05 2015 -0400
--
.../nifi/processors/standard/JmsConsumer.java | 2 +
.../apache/nifi/processors/standard/PutJMS.java | 2 +
.../processors/standard/util/JmsFactory.java| 79 ++--
.../processors/standard/util/JmsProperties.java | 8 ++
4 files changed, 86 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/26edab31/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
index b53d62f..461d381 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
@@ -25,6 +25,7 @@ import static
org.apache.nifi.processors.standard.util.JmsProperties.JMS_PROPS_T
import static
org.apache.nifi.processors.standard.util.JmsProperties.JMS_PROVIDER;
import static
org.apache.nifi.processors.standard.util.JmsProperties.MESSAGE_SELECTOR;
import static org.apache.nifi.processors.standard.util.JmsProperties.PASSWORD;
+import static
org.apache.nifi.processors.standard.util.JmsProperties.SSL_CONTEXT_SERVICE;
import static org.apache.nifi.processors.standard.util.JmsProperties.TIMEOUT;
import static org.apache.nifi.processors.standard.util.JmsProperties.URL;
import static org.apache.nifi.processors.standard.util.JmsProperties.USERNAME;
@@ -87,6 +88,7 @@ public abstract class JmsConsumer extends AbstractProcessor {
descriptors.add(BATCH_SIZE);
descriptors.add(USERNAME);
descriptors.add(PASSWORD);
+descriptors.add(SSL_CONTEXT_SERVICE);
descriptors.add(ACKNOWLEDGEMENT_MODE);
descriptors.add(MESSAGE_SELECTOR);
descriptors.add(JMS_PROPS_TO_ATTRIBUTES);
http://git-wip-us.apache.org/repos/asf/nifi/blob/26edab31/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutJMS.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutJMS.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutJMS.java
index dff5a6b..b8902a9 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutJMS.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutJMS.java
@@ -47,6 +47,7 @@ import static
org.apache.nifi.processors.standard.util.JmsProperties.REPLY_TO_QU
import static org.apache.nifi.processors.standard.util.JmsProperties.TIMEOUT;
import static org.apache.nifi.processors.standard.util.JmsProperties.URL;
import static org.apache.nifi.processors.standard.util.JmsProperties.USERNAME;
+import static
org.apache.nifi.processors.standard.util.JmsProperties.SSL_CONTEXT_SERVICE;
import java.io.IOException;
import java.io.InputStream;
@@ -122,6 +123,7 @@ public class PutJMS extends AbstractProcessor {
descriptors.add(BATCH_SIZE);
descriptors.add(USERNAME);
descriptors.add(PASSWORD);
+descriptors.add(SSL_CONTEXT_SERVICE);
descriptors.add(MESSAGE_TYPE);
descriptors.add(MESSAGE_PRIORITY);
descriptors.add(REPLY_TO_QUEUE);
http://git-wip-us.apache.org/repos/asf/nifi/blob/26edab31/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JmsFactory.java
--
[jira] [Commented] (NIFI-973) Create a Getting Started Guide
[ https://issues.apache.org/jira/browse/NIFI-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973670#comment-14973670 ] Aldrin Piri commented on NIFI-973: -- Eegads, sorry for missing the follow up patch. Read through your new patch again and it looks like a solid start and the areas that needed to be fixed up were. Nice work and this is a great foundation to build on, +1 > Create a Getting Started Guide > -- > > Key: NIFI-973 > URL: https://issues.apache.org/jira/browse/NIFI-973 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Mark Payne > Fix For: 0.4.0 > > Attachments: 0001-NIFI-973-Created-a-Getting-Started-Guide.patch > > > Currently, we have an overview document that is very high level and is more > of an architectural document, and we also have a user guide that is very > verbose and extensive and several other guides that focus on specific aspects > of nifi in great detail. > We do not have anything that is more brief introduction to the features of > NiFi that a new user would be interested in reading through. The User Guide > is a bit longer than many users want to read and is more of a reference guide. > I'd like to see a guide targeted toward those who are not extremely familiar > with NiFi but don't want to read a 70+ page user guide. > Recommended topics: > - Downloading and starting nifi > - Building a simple flow > - What Processors are available > - Working with Attributes > - Working with Templates > - Data Provenance > - Monitoring nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-820) Provide improved provenance search functions for a component ID
[ https://issues.apache.org/jira/browse/NIFI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973577#comment-14973577 ] Aldrin Piri commented on NIFI-820: -- Will keep any eye out for an update. Regardless, awesome stuff. > Provide improved provenance search functions for a component ID > --- > > Key: NIFI-820 > URL: https://issues.apache.org/jira/browse/NIFI-820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Aldrin Piri >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-820.patch, provenance context entry.png, > provenance search auto suggest.png > > > In terms of making provenance more accessible, it would be nice to have a > provenance search via the processor context menu which auto populates the > provenance with the processor ID. Typically when I want evaluate how a > processor is behaving, I have to select configure, copy ID, hit provenance, > paste the component ID. > In a related fashion, providing the functionality of the search of main > workflow in the component ID field of the provenance dialog to discover > component ID could also prove to be useful. > Not sure which of these makes sense, is feasible, or is actually an improved > user experience, but each feels like it could expedite the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-820) Provide improved provenance search functions for a component ID
[ https://issues.apache.org/jira/browse/NIFI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973573#comment-14973573 ] Matt Gilman commented on NIFI-820: -- Agreed. Especially now that users can drop the flow files manually it may make sense for them to be th source of an event. I'm thinking the process group thing may have been a oversight in our conversation. Unfortunately, the term component can be overloaded at times. I'll update the patch accordingly when/if necessary. > Provide improved provenance search functions for a component ID > --- > > Key: NIFI-820 > URL: https://issues.apache.org/jira/browse/NIFI-820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Aldrin Piri >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-820.patch, provenance context entry.png, > provenance search auto suggest.png > > > In terms of making provenance more accessible, it would be nice to have a > provenance search via the processor context menu which auto populates the > provenance with the processor ID. Typically when I want evaluate how a > processor is behaving, I have to select configure, copy ID, hit provenance, > paste the component ID. > In a related fashion, providing the functionality of the search of main > workflow in the component ID field of the provenance dialog to discover > component ID could also prove to be useful. > Not sure which of these makes sense, is feasible, or is actually an improved > user experience, but each feels like it could expedite the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-10) Add new Provenance Event types
[ https://issues.apache.org/jira/browse/NIFI-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973567#comment-14973567 ] Aldrin Piri commented on NIFI-10: - Patch did not apply for current master (8d2f9bc64b03702f4db53de6ac741ce102413bfa) > Add new Provenance Event types > -- > > Key: NIFI-10 > URL: https://issues.apache.org/jira/browse/NIFI-10 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Joseph Witt >Assignee: Mark Payne >Priority: Minor > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-10-Added-FETCH-and-DOWNLOAD-Provenance-Events-u.patch > > > Provenance download event should support the notion of proxy chains as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-820) Provide improved provenance search functions for a component ID
[ https://issues.apache.org/jira/browse/NIFI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973561#comment-14973561 ] Aldrin Piri commented on NIFI-820: -- Okay, at first thought I would think connections could generate provenance events before process groups. The group level doesn't seem as intuitive for me as they aren't actually providing any other function besides an abstraction of the processors and/or connections. Would be interested to hear additional details though. Thanks! > Provide improved provenance search functions for a component ID > --- > > Key: NIFI-820 > URL: https://issues.apache.org/jira/browse/NIFI-820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Aldrin Piri >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-820.patch, provenance context entry.png, > provenance search auto suggest.png > > > In terms of making provenance more accessible, it would be nice to have a > provenance search via the processor context menu which auto populates the > provenance with the processor ID. Typically when I want evaluate how a > processor is behaving, I have to select configure, copy ID, hit provenance, > paste the component ID. > In a related fashion, providing the functionality of the search of main > workflow in the component ID field of the provenance dialog to discover > component ID could also prove to be useful. > Not sure which of these makes sense, is feasible, or is actually an improved > user experience, but each feels like it could expedite the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-820) Provide improved provenance search functions for a component ID
[ https://issues.apache.org/jira/browse/NIFI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973559#comment-14973559 ] Matt Gilman commented on NIFI-820: -- Per a conversation with [~markap14] all components could be a source of provenance events. Basically anything that's not a connection. I think the gist may have been around who generates events for expiration/dropping/sending?/etc. Or maybe the process group was just an oversight. Either way, it will be a trivial adjustment. Mark? > Provide improved provenance search functions for a component ID > --- > > Key: NIFI-820 > URL: https://issues.apache.org/jira/browse/NIFI-820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Aldrin Piri >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-820.patch, provenance context entry.png, > provenance search auto suggest.png > > > In terms of making provenance more accessible, it would be nice to have a > provenance search via the processor context menu which auto populates the > provenance with the processor ID. Typically when I want evaluate how a > processor is behaving, I have to select configure, copy ID, hit provenance, > paste the component ID. > In a related fashion, providing the functionality of the search of main > workflow in the component ID field of the provenance dialog to discover > component ID could also prove to be useful. > Not sure which of these makes sense, is feasible, or is actually an improved > user experience, but each feels like it could expedite the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-820) Provide improved provenance search functions for a component ID
[ https://issues.apache.org/jira/browse/NIFI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973552#comment-14973552 ] Aldrin Piri commented on NIFI-820: -- 0 Build, contrib, and code all looks good. I was able to test it in the UI and all seemed good, however, the provenance menu was also provided for processing groups, which of do not have provenance events. If there was some search that would span all nested components, I could see the value, but this should likely be removed from the context menu in the interim. Otherwise, looks great and is a tremendous time saver. > Provide improved provenance search functions for a component ID > --- > > Key: NIFI-820 > URL: https://issues.apache.org/jira/browse/NIFI-820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Aldrin Piri >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-820.patch, provenance context entry.png, > provenance search auto suggest.png > > > In terms of making provenance more accessible, it would be nice to have a > provenance search via the processor context menu which auto populates the > provenance with the processor ID. Typically when I want evaluate how a > processor is behaving, I have to select configure, copy ID, hit provenance, > paste the component ID. > In a related fashion, providing the functionality of the search of main > workflow in the component ID field of the provenance dialog to discover > component ID could also prove to be useful. > Not sure which of these makes sense, is feasible, or is actually an improved > user experience, but each feels like it could expedite the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-820) Provide improved provenance search functions for a component ID
[ https://issues.apache.org/jira/browse/NIFI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973548#comment-14973548 ] Aldrin Piri commented on NIFI-820: -- Reviewing > Provide improved provenance search functions for a component ID > --- > > Key: NIFI-820 > URL: https://issues.apache.org/jira/browse/NIFI-820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Aldrin Piri >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-820.patch, provenance context entry.png, > provenance search auto suggest.png > > > In terms of making provenance more accessible, it would be nice to have a > provenance search via the processor context menu which auto populates the > provenance with the processor ID. Typically when I want evaluate how a > processor is behaving, I have to select configure, copy ID, hit provenance, > paste the component ID. > In a related fashion, providing the functionality of the search of main > workflow in the component ID field of the provenance dialog to discover > component ID could also prove to be useful. > Not sure which of these makes sense, is feasible, or is actually an improved > user experience, but each feels like it could expedite the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-908) Add Support for SSL in JMS Processors
[ https://issues.apache.org/jira/browse/NIFI-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973480#comment-14973480 ] Tony Kurc commented on NIFI-908: for anyone else building, I had to apply patch at f2c4f2d > Add Support for SSL in JMS Processors > - > > Key: NIFI-908 > URL: https://issues.apache.org/jira/browse/NIFI-908 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Luke Williamson >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix, jms, newbie > Fix For: 0.4.0 > > Attachments: nifi-908.patch.zip, nifi-908.patch.zip, > nifi-908.patch.zip > > > Add support for SSL in the JMS processors. > ActiveMQ is the only provider and in order to do this it just needs to use > the SSL connection factory instead of the normal one when the connection is > using the SSL protocol. > This should also support SSL in failover mode. So the url's for SSL > connections would be something like > ssl://127.0.0.1:65189 > failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-820) Provide improved provenance search functions for a component ID
[ https://issues.apache.org/jira/browse/NIFI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-820: - Attachment: 0001-NIFI-820.patch > Provide improved provenance search functions for a component ID > --- > > Key: NIFI-820 > URL: https://issues.apache.org/jira/browse/NIFI-820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Aldrin Piri >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-820.patch, provenance context entry.png, > provenance search auto suggest.png > > > In terms of making provenance more accessible, it would be nice to have a > provenance search via the processor context menu which auto populates the > provenance with the processor ID. Typically when I want evaluate how a > processor is behaving, I have to select configure, copy ID, hit provenance, > paste the component ID. > In a related fashion, providing the functionality of the search of main > workflow in the component ID field of the provenance dialog to discover > component ID could also prove to be useful. > Not sure which of these makes sense, is feasible, or is actually an improved > user experience, but each feels like it could expedite the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-10) Add new Provenance Event types
[ https://issues.apache.org/jira/browse/NIFI-10?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-10: --- Attachment: 0001-NIFI-10-Added-FETCH-and-DOWNLOAD-Provenance-Events-u.patch > Add new Provenance Event types > -- > > Key: NIFI-10 > URL: https://issues.apache.org/jira/browse/NIFI-10 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Joseph Witt >Priority: Minor > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-10-Added-FETCH-and-DOWNLOAD-Provenance-Events-u.patch > > > Provenance download event should support the notion of proxy chains as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-908) Add Support for SSL in JMS Processors
[ https://issues.apache.org/jira/browse/NIFI-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973304#comment-14973304 ] Tony Kurc commented on NIFI-908: Started review and testing > Add Support for SSL in JMS Processors > - > > Key: NIFI-908 > URL: https://issues.apache.org/jira/browse/NIFI-908 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Luke Williamson >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix, jms, newbie > Fix For: 0.4.0 > > Attachments: nifi-908.patch.zip, nifi-908.patch.zip, > nifi-908.patch.zip > > > Add support for SSL in the JMS processors. > ActiveMQ is the only provider and in order to do this it just needs to use > the SSL connection factory instead of the normal one when the connection is > using the SSL protocol. > This should also support SSL in failover mode. So the url's for SSL > connections would be something like > ssl://127.0.0.1:65189 > failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-908) Add Support for SSL in JMS Processors
[ https://issues.apache.org/jira/browse/NIFI-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Kurc updated NIFI-908: --- Labels: easyfix jms newbie (was: easyfix newbie) > Add Support for SSL in JMS Processors > - > > Key: NIFI-908 > URL: https://issues.apache.org/jira/browse/NIFI-908 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Luke Williamson >Priority: Minor > Labels: easyfix, jms, newbie > Fix For: 0.4.0 > > Attachments: nifi-908.patch.zip, nifi-908.patch.zip, > nifi-908.patch.zip > > > Add support for SSL in the JMS processors. > ActiveMQ is the only provider and in order to do this it just needs to use > the SSL connection factory instead of the normal one when the connection is > using the SSL protocol. > This should also support SSL in failover mode. So the url's for SSL > connections would be something like > ssl://127.0.0.1:65189 > failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-908) Add Support for SSL in JMS Processors
[ https://issues.apache.org/jira/browse/NIFI-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Kurc reassigned NIFI-908: -- Assignee: Tony Kurc > Add Support for SSL in JMS Processors > - > > Key: NIFI-908 > URL: https://issues.apache.org/jira/browse/NIFI-908 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Luke Williamson >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix, jms, newbie > Fix For: 0.4.0 > > Attachments: nifi-908.patch.zip, nifi-908.patch.zip, > nifi-908.patch.zip > > > Add support for SSL in the JMS processors. > ActiveMQ is the only provider and in order to do this it just needs to use > the SSL connection factory instead of the normal one when the connection is > using the SSL protocol. > This should also support SSL in failover mode. So the url's for SSL > connections would be something like > ssl://127.0.0.1:65189 > failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1064) Create a blog for how to pull files, decompress, prepare, drive into HDFS
Joseph Witt created NIFI-1064:
-
Summary: Create a blog for how to pull files, decompress, prepare,
drive into HDFS
Key: NIFI-1064
URL: https://issues.apache.org/jira/browse/NIFI-1064
Project: Apache NiFi
Issue Type: Task
Components: Documentation & Website
Reporter: Joseph Witt
markpetro...@gmail.com
So, I stumbled onto Nifi at a Laurel, MD Spark meetup and was pretty
excited about using it. I'm running HDP and need to construct an ETL
like flow and would like to try to start, as a new user to Nifi, using
a "best practice" approach. Wondering if some of you more seasoned
users might provide some thoughts on my problem?
1. 160 zip files/day show up on an NFS share in various sub
directories and their filenames contain the mmddHHMMSS of when the
stats where generated.
2. Each zip file contains 4 or more large CSV files
3. I need just one of those CSVs from each zip file each day and they
all add up to about 10GB uncompressed
4. I need to extract that one file from each zip, strip off the first
line (the headers), and store it in HDFS compressed again using gzip
or snappy
5. I cannot delete the NFS file after the copy to HDFS because others
need access to it for some time
So, where I am having a hard time visualizing doing this in Nifi is
with the first step. I need to scan the NFS files after 8 AM every day
(when I know all files for the previous 24 hours will be present),
find that set of files for that day using the yyymmdd part of file
names, then perform the extract of the one file I need and process it
into HDFS.
I could imagine a processor that runs once every 24 hours on a cron
schedule. I could imaging running an ExecuteProcess processor against
a bash script to get the list of all the files that match the
mmdd. Then I get stuck. How to take this list of 160 file paths
and start the job of processing each one of them in parallel to run
the ETL flow?
Thanks in advance for any ideas
-
Lee Laim
2:27 AM (9 hours ago)
to users
Mark,
I'm far from seasoned but I'll take a swing at it to check my understanding (or
lack thereof).
I'd break the task into 2 parts:
Identify and move files to a staging location, then process the zip files from
the staging location.
Flow1:
Run a cron driven GenerateFlowFile Processor to start the process every 24
hours after 8AM ->
ExecuteStreamCommand to run your bash script to stream the the 160 files of
interest into ->
SplitText processor to generate a new flow file for each zip filename. This can
be routed into a
DistributeLoad processor which can will distribute the flowfiles to
ExtractText processors to extract the text out of the flowfile (extract
contents: filename and path), then pass to
UpdateAttribute of the flow file to be able to access the filename and path via
Nifi expression language. Pass flow file to
ExecuteStreamProcess(cp /${path_attribute}/${filename} /location2/${filename} )
this will copy the zipfile to a another directory(location2), to keep files at
the source for other users.
Flow2:
GetFile from location2 -> Unpack Contents -> RouteOnAttribute (to
select CSV of interest, discard the rest) -> (ExecuteStreamProcess (sed '1d')
to remove header -> CompressContent -> PutHDFS
Hope this helps, and I hope this isn't too far off.
Thanks,
Lee
Mark Payne via nifi.apache.org
9:45 AM (1 hour ago)
to users
Hey Mark,
Thanks for sharing your use case with us in pretty good details so that we can
understand
what you're trying to do here.
There are actually a few processors coming in the next release that I think
should help here.
First, there's the FetchFile processor that you noticed in NIFI-631. Hopefully
the ListFile will
make its way in there as well because it's much easier that way :) In either
case, you can right-click
on the Processor and click Configure. If you go to the Scheduling tab, you can
change the Scheduling
Strategy to CRON-Driven and set the schedule to run whenever you'd like.
As-is, the GetFile is expected
to remove the file from the current location, as the idea was that NiFi would
sort of assume
ownership of the file. It turns out that in the Open Source world, that's often
not desirable, so
we are moving more toward the List/Fetch pattern as described in that ticket.
Once you pull the files into NiFI, though, UnpackContent should unzip the
files, each into its
own FlowFile. You could then use a RouteOnAttribute to pull out just the file
that you care about,
based on its filename. You can then allow the others to be routed to Unmatched
and auto-terminate
them from the flow.
Stripping off the first line could probably be done using the ReplaceText, but
in the next version
of NiFi, we will have a RouteText processor that should make working with CSV's
far easier. You could,
for instance, route any line that begin
[jira] [Commented] (NIFI-1056) ExecuteSQL fails when trying to pull a NUMERIC value from Oracle database
[ https://issues.apache.org/jira/browse/NIFI-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973301#comment-14973301 ] ASF subversion and git services commented on NIFI-1056: --- Commit 8d2f9bc64b03702f4db53de6ac741ce102413bfa in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8d2f9bc ] NIFI-1056: Do not treat BigDecimal, BigInteger the same way we treat Number's when serializing to Avro because Avro Java Library doesn't support them Reviewed and Unit Test Amended by Tony Kurc (tk...@apache.org) > ExecuteSQL fails when trying to pull a NUMERIC value from Oracle database > - > > Key: NIFI-1056 > URL: https://issues.apache.org/jira/browse/NIFI-1056 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.3.0 >Reporter: Mark Payne >Assignee: Mark Payne > Labels: oracle, sql > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-1056-Do-not-treat-BigDecimal-BigInteger-the-sam.patch > > > We received the following error message from the mailing list: > 2015-10-20 15:59:59,859 ERROR [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] failed to process due to > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence; rolling back session: > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,860 ERROR [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] failed to process session > due to org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence: > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,860 WARN [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] Processor > Administratively Yielded for 1 sec due to processing failure > 2015-10-20 15:59:59,860 WARN [Timer-Driven Process Thread-9] > o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] due to uncaught > Exception: org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,864 WARN [Timer-Driven Process Thread-9] > o.a.n.c.t.ContinuallyRunProcessorTask > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296) > ~[na:na] > at > org.apache.nifi.processors.standard.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:87) > ~[na:na] > at > org.apache.nifi.processors.standard.ExecuteSQL$1.process(ExecuteSQL.java:142) > ~[na:na] > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.processors.standard.ExecuteSQL.onTrigger(ExecuteSQL.java:136) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127) > [nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49) > [nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) > [nifi-framework-core-0.3.0.jar:0.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_79] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [na:1.7.0_79] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [na:1.7.0_79] > at > java
nifi git commit: NIFI-1056: Do not treat BigDecimal, BigInteger the same way we treat Number's when serializing to Avro because Avro Java Library doesn't support them
Repository: nifi
Updated Branches:
refs/heads/master 385bfbb2c -> 8d2f9bc64
NIFI-1056: Do not treat BigDecimal, BigInteger the same way we treat Number's
when serializing to Avro because Avro Java Library doesn't support them
Reviewed and Unit Test Amended by Tony Kurc (tk...@apache.org)
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/8d2f9bc6
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/8d2f9bc6
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/8d2f9bc6
Branch: refs/heads/master
Commit: 8d2f9bc64b03702f4db53de6ac741ce102413bfa
Parents: 385bfbb
Author: Mark Payne
Authored: Sun Oct 25 11:31:30 2015 -0400
Committer: Tony Kurc
Committed: Sun Oct 25 11:31:30 2015 -0400
--
.../processors/standard/util/JdbcCommon.java| 6 +++
.../standard/util/TestJdbcCommon.java | 45
2 files changed, 51 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/8d2f9bc6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
index 59eaf30..ac95c8b 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
@@ -46,6 +46,8 @@ import static java.sql.Types.VARCHAR;
import java.io.IOException;
import java.io.OutputStream;
+import java.math.BigDecimal;
+import java.math.BigInteger;
import java.nio.ByteBuffer;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
@@ -99,6 +101,10 @@ public class JdbcCommon {
// org.apache.avro.AvroRuntimeException: Unknown datum
type java.lang.Byte
rec.put(i - 1, ((Byte) value).intValue());
+} else if (value instanceof BigDecimal || value instanceof
BigInteger) {
+// Avro can't handle BigDecimal and BigInteger as
numbers - it will throw an AvroRuntimeException such as: "Unknown datum type:
java.math.BigDecimal: 38"
+rec.put(i - 1, value.toString());
+
} else if (value instanceof Number || value instanceof
Boolean) {
rec.put(i - 1, value);
http://git-wip-us.apache.org/repos/asf/nifi/blob/8d2f9bc6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
index 9c9532f..b8fcfed 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
@@ -25,6 +25,7 @@ import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Field;
+import java.math.BigDecimal;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
@@ -34,6 +35,7 @@ import java.sql.Statement;
import java.sql.Types;
import java.util.HashSet;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileStream;
@@ -44,6 +46,8 @@ import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;
import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
public class TestJdbcCommon {
@@ -180,6 +184,47 @@ public class TestJdbcCommon {
}
+@Test
+public void testConvertToAvroStreamForBigDecimal() throws SQLException,
IOException {
+final ResultSetMetaData metadata =
Mockito.mock(ResultSetMetaData.class);
+Mockito.when(metadata.getColumnCount()).thenReturn(1);
+Mockito.when(metadata.getColumnType(1)).thenReturn(Types.NUMERIC);
+Mockito.when(metadata.getColumnName(1)).thenReturn("Chairman");
+
[jira] [Commented] (NIFI-631) Create ListFile and FetchFile processors
[ https://issues.apache.org/jira/browse/NIFI-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973299#comment-14973299 ] Mark Payne commented on NIFI-631: - [~jskora] - for what it is worth, the ListSFTP / FetchSFTP processors have now been merged to master. This includes the AbstractListProcessor, which should make the ListFile processor far easier to implement and test. > Create ListFile and FetchFile processors > > > Key: NIFI-631 > URL: https://issues.apache.org/jira/browse/NIFI-631 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne >Assignee: Joe Skora > Attachments: > 0001-NIFI-631-Initial-implementation-of-FetchFile-process.patch > > > This pair of Processors will provide several benefits over the existing > GetFile processor: > 1. Currently, GetFile will continually pull the same files if the "Keep > Source File" property is set to true. There is no way to pull the file and > leave it in the directory without continually pulling the same file. We could > implement state here, but it would either be a huge amount of state to > remember everything pulled or it would have to always pull the oldest file > first so that we can maintain just the Last Modified Date of the last file > pulled plus all files with the same Last Modified Date that have already been > pulled. > 2. If pulling from a network attached storage such as NFS, this would allow a > single processor to run ListFiles and then distribute those FlowFiles to the > cluster so that the cluster can share the work of pulling the data. > 3. There are use cases when we may want to pull a specific file (for example, > in conjunction with ProcessHttpRequest/ProcessHttpResponse) rather than just > pull all files in a directory. GetFile does not support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973293#comment-14973293 ] ASF subversion and git services commented on NIFI-673: -- Commit b0322d9ffe8d117aae4faf7dd3e2881a28940f96 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b0322d9 ] NIFI-673: Added Completion Strategy to FetchSFTP > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Added-sftp.listing.user-attribute-to-FlowFi.patch, > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch, > 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973295#comment-14973295 ] ASF subversion and git services commented on NIFI-673: -- Commit 385bfbb2c635f74aca318617175cd57a8e26ecda in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=385bfbb ] NIFI-673: Rebased from master; Added InputRequirement annotation, as it is now merged into master > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Added-sftp.listing.user-attribute-to-FlowFi.patch, > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch, > 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1056) ExecuteSQL fails when trying to pull a NUMERIC value from Oracle database
[ https://issues.apache.org/jira/browse/NIFI-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973296#comment-14973296 ] Tony Kurc commented on NIFI-1056: - review completed. making a slight amendment to the unit test to verify the output > ExecuteSQL fails when trying to pull a NUMERIC value from Oracle database > - > > Key: NIFI-1056 > URL: https://issues.apache.org/jira/browse/NIFI-1056 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.3.0 >Reporter: Mark Payne >Assignee: Mark Payne > Labels: oracle, sql > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-1056-Do-not-treat-BigDecimal-BigInteger-the-sam.patch > > > We received the following error message from the mailing list: > 2015-10-20 15:59:59,859 ERROR [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] failed to process due to > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence; rolling back session: > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,860 ERROR [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] failed to process session > due to org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence: > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,860 WARN [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] Processor > Administratively Yielded for 1 sec due to processing failure > 2015-10-20 15:59:59,860 WARN [Timer-Driven Process Thread-9] > o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] due to uncaught > Exception: org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,864 WARN [Timer-Driven Process Thread-9] > o.a.n.c.t.ContinuallyRunProcessorTask > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296) > ~[na:na] > at > org.apache.nifi.processors.standard.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:87) > ~[na:na] > at > org.apache.nifi.processors.standard.ExecuteSQL$1.process(ExecuteSQL.java:142) > ~[na:na] > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.processors.standard.ExecuteSQL.onTrigger(ExecuteSQL.java:136) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127) > [nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49) > [nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) > [nifi-framework-core-0.3.0.jar:0.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_79] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [na:1.7.0_79] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [na:1.7.0_79] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [na:1.7.0_79] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_79] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_79] > at
[jira] [Commented] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973292#comment-14973292 ] ASF subversion and git services commented on NIFI-673: -- Commit d1d57931bf996a230ab7941cb6c1524286c97606 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d1d5793 ] NIFI-673: Initial implementation of ListSFTP, FetchSFTP > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Added-sftp.listing.user-attribute-to-FlowFi.patch, > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch, > 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973294#comment-14973294 ] ASF subversion and git services commented on NIFI-673: -- Commit 4e38288062f33ecafdfd3b4bc453bdffe54e752e in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=4e38288 ] NIFI-673: Added sftp.listing.user attribute to FlowFiles created by ListSFTP; ensure that FetchSFTP indicates that the username supports Expression Language > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Added-sftp.listing.user-attribute-to-FlowFi.patch, > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch, > 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[5/5] nifi git commit: NIFI-673: Rebased from master; Added InputRequirement annotation, as it is now merged into master
NIFI-673: Rebased from master; Added InputRequirement annotation, as it is now
merged into master
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/385bfbb2
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/385bfbb2
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/385bfbb2
Branch: refs/heads/master
Commit: 385bfbb2c635f74aca318617175cd57a8e26ecda
Parents: 4e38288
Author: Mark Payne
Authored: Sun Oct 25 11:14:11 2015 -0400
Committer: Mark Payne
Committed: Sun Oct 25 11:14:11 2015 -0400
--
.../main/java/org/apache/nifi/processors/standard/FetchSFTP.java | 3 +++
.../main/java/org/apache/nifi/processors/standard/ListSFTP.java | 3 +++
2 files changed, 6 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/385bfbb2/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
index b95d864..8379987 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
@@ -22,6 +22,8 @@ import java.util.Collection;
import java.util.Collections;
import java.util.List;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.behavior.WritesAttributes;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
@@ -35,6 +37,7 @@ import org.apache.nifi.processors.standard.util.FileTransfer;
import org.apache.nifi.processors.standard.util.SFTPTransfer;
// Note that we do not use @SupportsBatching annotation. This processor cannot
support batching because it must ensure that session commits happen before
remote files are deleted.
+@InputRequirement(Requirement.INPUT_REQUIRED)
@Tags({"sftp", "get", "retrieve", "files", "fetch", "remote", "ingest",
"source", "input"})
@CapabilityDescription("Fetches the content of a file from a remote SFTP
server and overwrites the contents of an incoming FlowFile with the content of
the remote file.")
@SeeAlso({GetSFTP.class, PutSFTP.class, GetFTP.class, PutFTP.class})
http://git-wip-us.apache.org/repos/asf/nifi/blob/385bfbb2/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListSFTP.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListSFTP.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListSFTP.java
index 7226263..609b693 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListSFTP.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListSFTP.java
@@ -20,6 +20,8 @@ package org.apache.nifi.processors.standard;
import java.util.ArrayList;
import java.util.List;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.TriggerSerially;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.behavior.WritesAttributes;
@@ -32,6 +34,7 @@ import org.apache.nifi.processors.standard.util.FileTransfer;
import org.apache.nifi.processors.standard.util.SFTPTransfer;
@TriggerSerially
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
@Tags({"list", "sftp", "remote", "ingest", "source", "input", "files"})
@CapabilityDescription("Performs a listing of the files residing on an SFTP
server. For each file that is found on the remote server, a new FlowFile will
be created with the filename attribute "
+ "set to the name of the file on the remote server. This can then be used
in conjunction with FetchSFTP in order to fetch those files.")
[1/5] nifi git commit: NIFI-673: Added sftp.listing.user attribute to FlowFiles created by ListSFTP; ensure that FetchSFTP indicates that the username supports Expression Language
Repository: nifi
Updated Branches:
refs/heads/master 8a8006085 -> 385bfbb2c
NIFI-673: Added sftp.listing.user attribute to FlowFiles created by ListSFTP;
ensure that FetchSFTP indicates that the username supports Expression Language
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/4e382880
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/4e382880
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/4e382880
Branch: refs/heads/master
Commit: 4e38288062f33ecafdfd3b4bc453bdffe54e752e
Parents: b0322d9
Author: Mark Payne
Authored: Fri Oct 23 16:13:52 2015 -0400
Committer: Mark Payne
Committed: Sun Oct 25 11:13:02 2015 -0400
--
.../apache/nifi/processors/standard/FetchFileTransfer.java | 7 +++
.../java/org/apache/nifi/processors/standard/FetchSFTP.java | 2 +-
.../apache/nifi/processors/standard/ListFileTransfer.java| 8
.../java/org/apache/nifi/processors/standard/ListSFTP.java | 3 ++-
4 files changed, 18 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/4e382880/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFileTransfer.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFileTransfer.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFileTransfer.java
index ab0be78..a405afb 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFileTransfer.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFileTransfer.java
@@ -77,6 +77,13 @@ public abstract class FetchFileTransfer extends
AbstractProcessor {
.expressionLanguageSupported(true)
.required(true)
.build();
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("Username")
+.description("Username")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.required(true)
+.build();
public static final PropertyDescriptor REMOTE_FILENAME = new
PropertyDescriptor.Builder()
.name("Remote File")
.description("The fully qualified filename on the remote system")
http://git-wip-us.apache.org/repos/asf/nifi/blob/4e382880/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
index ad81c83..b95d864 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchSFTP.java
@@ -53,7 +53,7 @@ public class FetchSFTP extends FetchFileTransfer {
final List properties = new ArrayList<>();
properties.add(HOSTNAME);
properties.add(port);
-properties.add(SFTPTransfer.USERNAME);
+properties.add(USERNAME);
properties.add(SFTPTransfer.PASSWORD);
properties.add(SFTPTransfer.PRIVATE_KEY_PATH);
properties.add(SFTPTransfer.PRIVATE_KEY_PASSPHRASE);
http://git-wip-us.apache.org/repos/asf/nifi/blob/4e382880/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFileTransfer.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFileTransfer.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFileTransfer.java
index 1176fd0..b6c8c28 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFileTransfer.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFileTransfer.java
@@ -45,6 +45,13 @@ public abstract class ListFileTransfer extends
AbstractListProcessor {
.expressionLanguag
[3/5] nifi git commit: NIFI-673: Initial implementation of ListSFTP, FetchSFTP
http://git-wip-us.apache.org/repos/asf/nifi/blob/d1d57931/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileTransfer.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileTransfer.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileTransfer.java
index f0061b8..fe277df 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileTransfer.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileTransfer.java
@@ -23,6 +23,7 @@ import java.io.InputStream;
import java.util.List;
import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.processor.util.StandardValidators;
@@ -34,6 +35,8 @@ public interface FileTransfer extends Closeable {
InputStream getInputStream(String remoteFileName) throws IOException;
+InputStream getInputStream(String remoteFileName, FlowFile flowFile)
throws IOException;
+
void flush() throws IOException;
FileInfo getRemoteFileInfo(FlowFile flowFile, String path, String
remoteFileName) throws IOException;
@@ -51,127 +54,127 @@ public interface FileTransfer extends Closeable {
void ensureDirectoryExists(FlowFile flowFile, File remoteDirectory) throws
IOException;
public static final PropertyDescriptor HOSTNAME = new
PropertyDescriptor.Builder()
-.name("Hostname")
-.description("The fully qualified hostname or IP address of the
remote system")
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.required(true)
-.expressionLanguageSupported(true)
-.build();
+.name("Hostname")
+.description("The fully qualified hostname or IP address of the remote
system")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.required(true)
+.expressionLanguageSupported(true)
+.build();
public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
-.name("Username")
-.description("Username")
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.required(true)
-.build();
+.name("Username")
+.description("Username")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.required(true)
+.build();
public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
-.name("Password")
-.description("Password for the user account")
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.required(false)
-.sensitive(true)
-.build();
+.name("Password")
+.description("Password for the user account")
+.addValidator(Validator.VALID)
+.required(false)
+.sensitive(true)
+.build();
public static final PropertyDescriptor DATA_TIMEOUT = new
PropertyDescriptor.Builder()
-.name("Data Timeout")
-.description("Amount of time to wait before timing out while
transferring data")
-.required(true)
-.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
-.defaultValue("30 sec")
-.build();
+.name("Data Timeout")
+.description("When transferring a file between the local and remote
system, this value specifies how long is allowed to elapse without any data
being transferred between systems")
+.required(true)
+.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.defaultValue("30 sec")
+.build();
public static final PropertyDescriptor CONNECTION_TIMEOUT = new
PropertyDescriptor.Builder()
-.name("Connection Timeout")
-.description("Amount of time to wait before timing out while
creating a connection")
-.required(true)
-.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
-.defaultValue("30 sec")
-.build();
+.name("Connection Timeout")
+.description("Amount of time to wait before timing out while creating
a connection")
+.required(true)
+.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.defaultValue("30 sec")
+.build();
public static final PropertyDescriptor REMOTE_PATH = new
PropertyDescriptor.Builder()
-.name("Remote Path")
-.description("The path on the remote system from which to pull or
push files")
-.required(false)
-
[4/5] nifi git commit: NIFI-673: Initial implementation of ListSFTP, FetchSFTP
NIFI-673: Initial implementation of ListSFTP, FetchSFTP Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/d1d57931 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/d1d57931 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/d1d57931 Branch: refs/heads/master Commit: d1d57931bf996a230ab7941cb6c1524286c97606 Parents: 8a80060 Author: Mark Payne Authored: Sun Oct 4 15:48:28 2015 -0400 Committer: Mark Payne Committed: Sun Oct 25 11:13:02 2015 -0400 -- .../standard/AbstractListProcessor.java | 505 +++ .../processors/standard/FetchFileTransfer.java | 296 +++ .../nifi/processors/standard/FetchSFTP.java | 89 .../processors/standard/ListFileTransfer.java | 103 .../nifi/processors/standard/ListSFTP.java | 81 +++ .../processors/standard/util/EntityListing.java | 71 +++ .../processors/standard/util/FTPTransfer.java | 135 ++--- .../nifi/processors/standard/util/FileInfo.java | 18 +- .../processors/standard/util/FileTransfer.java | 335 ++-- .../standard/util/ListableEntity.java | 40 ++ .../util/PermissionDeniedException.java | 32 ++ .../processors/standard/util/SFTPTransfer.java | 174 --- .../org.apache.nifi.processor.Processor | 2 + .../standard/TestAbstractListProcessor.java | 221 .../standard/TestFetchFileTransfer.java | 186 +++ 15 files changed, 1988 insertions(+), 300 deletions(-) -- http://git-wip-us.apache.org/repos/asf/nifi/blob/d1d57931/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java -- diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java new file mode 100644 index 000..8a7fade --- /dev/null +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java @@ -0,0 +1,505 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStream; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.Date; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.Set; + +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.notification.OnPrimaryNodeStateChange; +import org.apache.nifi.annotation.notification.PrimaryNodeState; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.distributed.cache.client.exception.DeserializationException; +import org.apache.nifi.distributed.cache.client.exception.SerializationException; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processors.standard.util.EntityListing; +import org.apache.nifi.processors.standard.util.ListableEntity; +import org.codehaus.jackson.JsonGenerationException; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.JsonParseException; +import org.codehaus.jackson.map.JsonMappin
[2/5] nifi git commit: NIFI-673: Added Completion Strategy to FetchSFTP
NIFI-673: Added Completion Strategy to FetchSFTP Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/b0322d9f Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/b0322d9f Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/b0322d9f Branch: refs/heads/master Commit: b0322d9ffe8d117aae4faf7dd3e2881a28940f96 Parents: d1d5793 Author: Mark Payne Authored: Mon Oct 5 16:11:40 2015 -0400 Committer: Mark Payne Committed: Sun Oct 25 11:13:02 2015 -0400 -- .../standard/AbstractListProcessor.java | 38 +++--- .../processors/standard/FetchFileTransfer.java | 66 -- .../nifi/processors/standard/FetchSFTP.java | 15 ++- .../processors/standard/ListFileTransfer.java | 8 ++ .../nifi/processors/standard/ListSFTP.java | 7 +- .../processors/standard/util/EntityListing.java | 2 +- .../processors/standard/util/FTPTransfer.java | 26 ++-- .../processors/standard/util/FileTransfer.java | 2 + .../processors/standard/util/SFTPTransfer.java | 17 +++ .../standard/TestFetchFileTransfer.java | 131 +++ 10 files changed, 265 insertions(+), 47 deletions(-) -- http://git-wip-us.apache.org/repos/asf/nifi/blob/b0322d9f/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java -- diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java index 8a7fade..e592483 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java @@ -70,23 +70,25 @@ import org.codehaus.jackson.map.ObjectMapper; * * * In order to make use of this abstract class, the entities listed must meet the following criteria: - * - * - * Entity must have a timestamp associated with it. This timestamp is used to determine if entities are "new" or not. Any entity that is - * returned by the listing will be considered "new" if the timestamp is later than the latest timestamp pulled. - * - * - * Entity must have a unique identifier. This is used in conjunction with the timestamp in order to determine whether or not the entity is - * new. If the timestamp of an entity is before the latest timestamp pulled, then the entity is not considered new. If the timestamp is later - * than the last timestamp pulled, then the entity is considered new. If the timestamp is equal to the latest timestamp pulled, then the entity's - * identifier is compared to all of the entity identifiers that have that same timestamp in order to determine whether or not the entity has been - * seen already. - * - * - * Entity must have a user-readable name that can be used for logging purposes. - * * * + * + * + * Entity must have a timestamp associated with it. This timestamp is used to determine if entities are "new" or not. Any entity that is + * returned by the listing will be considered "new" if the timestamp is later than the latest timestamp pulled. + * + * + * Entity must have a unique identifier. This is used in conjunction with the timestamp in order to determine whether or not the entity is + * new. If the timestamp of an entity is before the latest timestamp pulled, then the entity is not considered new. If the timestamp is later + * than the last timestamp pulled, then the entity is considered new. If the timestamp is equal to the latest timestamp pulled, then the entity's + * identifier is compared to all of the entity identifiers that have that same timestamp in order to determine whether or not the entity has been + * seen already. + * + * + * Entity must have a user-readable name that can be used for logging purposes. + * + * + * * * This class persists state across restarts so that even if NiFi is restarted, duplicates will not be pulled from the remote system. This is performed using * two different mechanisms. First, state is stored locally. This allows the system to be restarted and begin processing where it left off. The state that is @@ -111,6 +113,7 @@ import org.codehaus.jackson.map.ObjectMapper; * * * Subclasses are responsible for the following: + * * * * @@ -134,7 +137,6 @@ import org.codehaus.jackson.map.ObjectMapper; * a boolean indicating whether or not a change in the va
[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection
[ https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973288#comment-14973288 ] ASF subversion and git services commented on NIFI-810: -- Commit ccfb57fe9ff43f11319dcb1625bfc78b1d88f56a in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ccfb57f ] NIFI-810: Addressed several checkstyle violations > Create Annotation that indicates that a Processor cannot be scheduled to run > without an incoming connection > --- > > Key: NIFI-810 > URL: https://issues.apache.org/jira/browse/NIFI-810 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > > Currently, if a Processor has no incoming connections but is started, it will > continually without ever accomplishing anything. We should have an > annotation, perhaps @RequiresInput, that indicates that the Processor should > not be scheduled to run unless it has an incoming connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[14/19] nifi git commit: NIFI-810: rebased from master
http://git-wip-us.apache.org/repos/asf/nifi/blob/b974445d/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
--
diff --cc
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
index 3c816d0,0c39eda..f69c510
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
@@@ -74,1241 -74,1242 +74,1241 @@@ import org.slf4j.LoggerFactory
*/
public class StandardProcessorNode extends ProcessorNode implements
Connectable {
- public static final String BULLETIN_OBSERVER_ID = "bulletin-observer";
-
- public static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS;
- public static final String DEFAULT_YIELD_PERIOD = "1 sec";
- public static final String DEFAULT_PENALIZATION_PERIOD = "30 sec";
- private final AtomicReference processGroup;
- private final Processor processor;
- private final AtomicReference identifier;
- private final Map destinations;
- private final Map> connections;
- private final AtomicReference>
undefinedRelationshipsToTerminate;
- private final AtomicReference> incomingConnectionsRef;
- private final ReentrantReadWriteLock rwLock;
- private final Lock readLock;
- private final Lock writeLock;
- private final AtomicBoolean isolated;
- private final AtomicBoolean lossTolerant;
- private final AtomicReference scheduledState;
- private final AtomicReference comments;
- private final AtomicReference name;
- private final AtomicReference position;
- private final AtomicReference annotationData;
- private final AtomicReference schedulingPeriod; // stored as
string so it's presented to user as they entered it
- private final AtomicReference yieldPeriod;
- private final AtomicReference penalizationPeriod;
- private final AtomicReference> style;
- private final AtomicInteger concurrentTaskCount;
- private final AtomicLong yieldExpiration;
- private final AtomicLong schedulingNanos;
- private final boolean triggerWhenEmpty;
- private final boolean sideEffectFree;
- private final boolean triggeredSerially;
- private final boolean triggerWhenAnyDestinationAvailable;
- private final boolean eventDrivenSupported;
- private final boolean batchSupported;
- private final Requirement inputRequirement;
- private final ValidationContextFactory validationContextFactory;
- private final ProcessScheduler processScheduler;
- private long runNanos = 0L;
-
- private SchedulingStrategy schedulingStrategy; // guarded by
read/write lock
-
- @SuppressWarnings("deprecation")
- public StandardProcessorNode(final Processor processor, final String
uuid, final ValidationContextFactory validationContextFactory,
- final ProcessScheduler scheduler, final
ControllerServiceProvider controllerServiceProvider) {
- super(processor, uuid, validationContextFactory,
controllerServiceProvider);
-
- this.processor = processor;
- identifier = new AtomicReference<>(uuid);
- destinations = new HashMap<>();
- connections = new HashMap<>();
- incomingConnectionsRef = new
AtomicReference>(new ArrayList());
- scheduledState = new AtomicReference<>(ScheduledState.STOPPED);
- rwLock = new ReentrantReadWriteLock(false);
- readLock = rwLock.readLock();
- writeLock = rwLock.writeLock();
- lossTolerant = new AtomicBoolean(false);
- final Set emptySetOfRelationships = new
HashSet<>();
- undefinedRelationshipsToTerminate = new
AtomicReference<>(emptySetOfRelationships);
- comments = new AtomicReference<>("");
- name = new
AtomicReference<>(processor.getClass().getSimpleName());
- schedulingPeriod = new AtomicReference<>("0 sec");
- schedulingNanos = new AtomicLong(MINIMUM_SCHEDULING_NANOS);
- yieldPeriod = new AtomicReference<>(DEFAULT_YIELD_PERIOD);
- yieldExpiration = new AtomicLong(0L);
- concurrentTaskCount = new AtomicInteger(1);
- position = new AtomicReference<>(new Position(0D, 0D));
- style = new AtomicReference<>(Collections.unmodifiableMap(new
HashMap()));
- this.processGroup = new AtomicReference<>();
- processScheduler = scheduler;
- annota
[12/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
index cbd0f88..0c39eda 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
@@ -16,14 +16,6 @@
*/
package org.apache.nifi.controller;
-import org.apache.nifi.annotation.behavior.EventDriven;
-import org.apache.nifi.annotation.behavior.SideEffectFree;
-import org.apache.nifi.annotation.behavior.SupportsBatching;
-import org.apache.nifi.annotation.behavior.TriggerSerially;
-import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
-import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
-import org.apache.nifi.annotation.documentation.CapabilityDescription;
-
import static java.util.Objects.requireNonNull;
import java.util.ArrayList;
@@ -43,6 +35,17 @@ import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.components.ValidationContext;
import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.connectable.Connectable;
@@ -61,8 +64,6 @@ import org.apache.nifi.processor.Processor;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.scheduling.SchedulingStrategy;
import org.apache.nifi.util.FormatUtils;
-import org.apache.commons.lang3.builder.EqualsBuilder;
-import org.apache.commons.lang3.builder.HashCodeBuilder;
import org.quartz.CronExpression;
import org.slf4j.LoggerFactory;
@@ -73,1185 +74,1242 @@ import org.slf4j.LoggerFactory;
*/
public class StandardProcessorNode extends ProcessorNode implements
Connectable {
-public static final String BULLETIN_OBSERVER_ID = "bulletin-observer";
-
-public static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS;
-public static final String DEFAULT_YIELD_PERIOD = "1 sec";
-public static final String DEFAULT_PENALIZATION_PERIOD = "30 sec";
-private final AtomicReference processGroup;
-private final Processor processor;
-private final AtomicReference identifier;
-private final Map destinations;
-private final Map> connections;
-private final AtomicReference>
undefinedRelationshipsToTerminate;
-private final AtomicReference> incomingConnectionsRef;
-private final ReentrantReadWriteLock rwLock;
-private final Lock readLock;
-private final Lock writeLock;
-private final AtomicBoolean isolated;
-private final AtomicBoolean lossTolerant;
-private final AtomicReference scheduledState;
-private final AtomicReference comments;
-private final AtomicReference name;
-private final AtomicReference position;
-private final AtomicReference annotationData;
-private final AtomicReference schedulingPeriod; // stored as
string so it's presented to user as they entered it
-private final AtomicReference yieldPeriod;
-private final AtomicReference penalizationPeriod;
-private final AtomicReference> style;
-private final AtomicInteger concurrentTaskCount;
-private final AtomicLong yieldExpiration;
-private final AtomicLong schedulingNanos;
-private final boolean triggerWhenEmpty;
-private final boolean sideEffectFree;
-private final boolean triggeredSerially;
-private final boolean triggerWhenAnyDestinationAvailable;
-private final boolean eventDrivenSupported;
-private final boolean batchSupported;
-private final ValidationContextFactory validationContextFactory;
-private final ProcessScheduler processScheduler;
-private long runNanos
[17/19] nifi git commit: NIFI-810: Reworded validation errors pertaining ot upstream connections
NIFI-810: Reworded validation errors pertaining ot upstream connections
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/8e2308b7
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/8e2308b7
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/8e2308b7
Branch: refs/heads/master
Commit: 8e2308b78de480dd7848ffe8efb485a5ee61c42a
Parents: ccfb57f
Author: Mark Payne
Authored: Wed Oct 7 17:53:04 2015 -0400
Committer: Mark Payne
Committed: Wed Oct 7 17:53:04 2015 -0400
--
.../org/apache/nifi/controller/StandardProcessorNode.java| 8
1 file changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/8e2308b7/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
index ad22c6d..2b0d413 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
@@ -1048,8 +1048,8 @@ public class StandardProcessorNode extends ProcessorNode
implements Connectable
final int incomingConnCount =
getIncomingConnections().size();
if (incomingConnCount != 0) {
results.add(new ValidationResult.Builder()
-.explanation("Processor does not accept Incoming
Connections but is currently configured with " + incomingConnCount + " Incoming
Connections")
-.subject("Incoming Connections")
+.explanation("Processor does not allow upstream
connections but currently has " + incomingConnCount)
+.subject("Upstream Connections")
.valid(false)
.build());
}
@@ -1058,8 +1058,8 @@ public class StandardProcessorNode extends ProcessorNode
implements Connectable
case INPUT_REQUIRED: {
if (getIncomingConnections().isEmpty()) {
results.add(new ValidationResult.Builder()
-.explanation("Processor required at least one
Incoming Connection in order to perform its function but currently has no
Incoming Connection")
-.subject("Incoming Connections")
+.explanation("Processor requires an upstream
connection but currently has none")
+.subject("Upstream Connections")
.valid(false)
.build());
}
[09/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
index 68155d1..98a56bf 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
@@ -16,33 +16,7 @@
*/
package org.apache.nifi.processors.standard;
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessorInitializationContext;
-import org.apache.nifi.processor.DataUnit;
-import org.apache.nifi.processor.ProcessSession;
-import org.apache.nifi.processor.Relationship;
import java.io.BufferedWriter;
-
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.components.Validator;
-import org.apache.nifi.expression.AttributeValueDecorator;
-import org.apache.nifi.flowfile.FlowFile;
-import org.apache.nifi.stream.io.StreamUtils;
-import org.apache.nifi.logging.ProcessorLog;
-import org.apache.nifi.annotation.documentation.CapabilityDescription;
-import org.apache.nifi.annotation.behavior.EventDriven;
-import org.apache.nifi.annotation.behavior.SideEffectFree;
-import org.apache.nifi.annotation.behavior.SupportsBatching;
-import org.apache.nifi.annotation.documentation.Tags;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.processor.io.OutputStreamCallback;
-import org.apache.nifi.processor.io.StreamCallback;
-import org.apache.nifi.processor.util.FlowFileFilters;
-import org.apache.nifi.processor.util.StandardValidators;
-import org.apache.nifi.processors.standard.util.NLKBufferedReader;
-import org.apache.nifi.util.StopWatch;
-
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
@@ -58,9 +32,37 @@ import java.util.concurrent.TimeUnit;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.expression.AttributeValueDecorator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.FlowFileFilters;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.util.NLKBufferedReader;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
@EventDriven
@SideEffectFree
@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
@Tags({"Text", "Regular Expression", "Update", "Change", "Replace", "Modify",
"Regex"})
@CapabilityDescription("Updates the content of a FlowFile by evaluating a
Regular Expression against it and replacing the section of "
+ "the content that matches the Regular Expression with some alternate
value.")
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java
index 04a9c56..f68ac6c 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standar
[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection
[ https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973287#comment-14973287 ] ASF subversion and git services commented on NIFI-810: -- Commit b974445ddd38ec7e84995225b86987e6af1af52c in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b974445 ] NIFI-810: rebased from master > Create Annotation that indicates that a Processor cannot be scheduled to run > without an incoming connection > --- > > Key: NIFI-810 > URL: https://issues.apache.org/jira/browse/NIFI-810 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > > Currently, if a Processor has no incoming connections but is started, it will > continually without ever accomplishing anything. We should have an > annotation, perhaps @RequiresInput, that indicates that the Processor should > not be scheduled to run unless it has an incoming connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection
[ https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973289#comment-14973289 ] ASF subversion and git services commented on NIFI-810: -- Commit 8e2308b78de480dd7848ffe8efb485a5ee61c42a in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8e2308b ] NIFI-810: Reworded validation errors pertaining ot upstream connections > Create Annotation that indicates that a Processor cannot be scheduled to run > without an incoming connection > --- > > Key: NIFI-810 > URL: https://issues.apache.org/jira/browse/NIFI-810 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > > Currently, if a Processor has no incoming connections but is started, it will > continually without ever accomplishing anything. We should have an > annotation, perhaps @RequiresInput, that indicates that the Processor should > not be scheduled to run unless it has an incoming connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[18/19] nifi git commit: NIFI-810: Merged master into branch
NIFI-810: Merged master into branch
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/0636f0e7
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/0636f0e7
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/0636f0e7
Branch: refs/heads/master
Commit: 0636f0e731cd28299edd3a6e9db90de5045ab662
Parents: 8e2308b d63cd6b
Author: Mark Payne
Authored: Sun Oct 25 11:02:40 2015 -0400
Committer: Mark Payne
Committed: Sun Oct 25 11:02:40 2015 -0400
--
.../src/main/asciidoc/administration-guide.adoc | 4 +-
.../src/main/java/MyProcessor.java | 11 +-
.../nifi/processors/avro/ConvertAvroToJSON.java | 67 -
.../processors/avro/TestConvertAvroToJSON.java | 47 ++-
.../processors/aws/AbstractAWSProcessor.java| 2 +-
.../nifi/processors/aws/s3/DeleteS3Object.java | 98 ++
.../org.apache.nifi.processor.Processor | 1 +
.../processors/aws/s3/TestDeleteS3Object.java | 141 +
.../nifi/controller/FlowUnmarshaller.java | 77 -
.../src/main/resources/FlowConfiguration.xsd| 2 +-
.../src/main/resources/bin/nifi.sh | 96 +++---
.../canvas/new-controller-service-dialog.jsp| 1 -
.../partials/canvas/new-processor-dialog.jsp| 1 -
.../canvas/new-reporting-task-dialog.jsp| 1 -
.../css/new-controller-service-dialog.css | 9 -
.../main/webapp/css/new-processor-dialog.css| 9 -
.../webapp/css/new-reporting-task-dialog.css| 9 -
.../webapp/js/nf/canvas/nf-canvas-toolbox.js| 60 ++--
.../src/main/webapp/js/nf/canvas/nf-settings.js | 140 +
.../processors/kite/AbstractKiteProcessor.java | 11 +-
.../nifi/processors/kite/ConvertCSVToAvro.java | 296 ++-
.../processors/kite/TestCSVToAvroProcessor.java | 39 +++
.../nifi-standard-prioritizers/pom.xml | 4 +
.../PriorityAttributePrioritizer.java | 7 +-
.../PriorityAttributePrioritizerTest.java | 17 +-
.../nifi-standard-processors/pom.xml| 9 +
.../nifi/processors/standard/ExecuteSQL.java| 9 +-
.../nifi/processors/standard/InvokeHTTP.java| 1 +
.../nifi/processors/standard/ListenHTTP.java| 105 ---
.../standard/PutDistributedMapCache.java| 252
.../servlets/ContentAcknowledgmentServlet.java | 3 +-
.../standard/servlets/ListenHTTPServlet.java| 8 +-
.../processors/standard/util/JdbcCommon.java| 70 -
.../org.apache.nifi.processor.Processor | 1 +
.../nifi/processors/standard/TestGetFile.java | 21 +-
.../standard/TestPutDistributedMapCache.java| 277 +
.../standard/util/TestJdbcCommon.java | 42 +++
.../standard/util/TestJdbcTypesDerby.java | 133 +
.../standard/util/TestJdbcTypesH2.java | 149 ++
pom.xml | 2 +-
40 files changed, 1725 insertions(+), 507 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/0636f0e7/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
--
diff --cc
nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
index b214427,f0ba71a..f0f1630
---
a/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
+++
b/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
@@@ -35,7 -36,8 +38,7 @@@ import org.apache.nifi.annotation.behav
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
--import org.apache.nifi.annotation.documentation.Tags;
+ import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.flowfile.attributes.CoreAttributes;
import org.apache.nifi.processor.AbstractProcessor;
@@@ -47,8 -50,7 +51,7 @@@ import org.apache.nifi.processor.io.Str
@SideEffectFree
@SupportsBatching
- @Tags({ "json", "avro", "binary" })
-@Tags({"json", "avro", "binary"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
@CapabilityDescription("Converts a Binary Avro record into a JSON object.
This processor provides a direct mapping of an Avro field to a JSON field, such
"
+ "that the resulting JSON will have the same hierarchical structure as
the Avro document. Note that the Avro schema information will be lost, as this "
+ "is not a translation from binary Avro to JSON formatted Avro. The
output JSON is encoded the
[06/19] nifi git commit: NIFI-810: - Adding basic support for preventing connection when appropriate. - Updating validation when [dis]connecting processors.
NIFI-810:
- Adding basic support for preventing connection when appropriate.
- Updating validation when [dis]connecting processors.
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/2215bc84
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/2215bc84
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/2215bc84
Branch: refs/heads/master
Commit: 2215bc848b7db395b2ca9ac7cc4dc10891393721
Parents: 034ee6d
Author: Matt Gilman
Authored: Fri Sep 25 17:46:58 2015 -0400
Committer: Matt Gilman
Committed: Fri Sep 25 17:46:58 2015 -0400
--
.../org/apache/nifi/web/api/dto/ProcessorDTO.java| 15 +++
.../org/apache/nifi/controller/TemplateManager.java | 1 +
.../java/org/apache/nifi/web/api/dto/DtoFactory.java | 1 +
.../src/main/webapp/js/nf/canvas/nf-actions.js | 4 +++-
.../src/main/webapp/js/nf/canvas/nf-canvas-utils.js | 13 ++---
.../js/nf/canvas/nf-connection-configuration.js | 8 ++--
6 files changed, 36 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/2215bc84/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
index c65c46a..866d77c 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
@@ -36,6 +36,7 @@ public class ProcessorDTO extends NiFiComponentDTO {
private String description;
private Boolean supportsParallelProcessing;
private Boolean supportsEventDriven;
+private String inputRequirement;
private ProcessorConfigDTO config;
@@ -121,6 +122,20 @@ public class ProcessorDTO extends NiFiComponentDTO {
}
/**
+ * @return the input requirement of this processor
+ */
+@ApiModelProperty(
+value = "The input requirement for this processor."
+)
+public String getInputRequirement() {
+return inputRequirement;
+}
+
+public void setInputRequirement(String inputRequirement) {
+this.inputRequirement = inputRequirement;
+}
+
+/**
* @return whether this processor supports event driven scheduling
*/
@ApiModelProperty(
http://git-wip-us.apache.org/repos/asf/nifi/blob/2215bc84/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
index 7b8e173..a332e05 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
@@ -320,6 +320,7 @@ public class TemplateManager {
// remove validation errors
processorDTO.setValidationErrors(null);
+processorDTO.setInputRequirement(null);
}
}
http://git-wip-us.apache.org/repos/asf/nifi/blob/2215bc84/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
index 76bce6f..16b114e 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
@@ -1402,6 +1402,7 @@ public final class DtoFactory {
dto.setPosition(createPositionDto(node.getPosition()));
dto.setStyle(nod
[02/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
index b825972..39dc725 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
@@ -28,11 +28,13 @@ import java.util.concurrent.TimeUnit;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.SeeAlso;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.distributed.cache.client.Deserializer;
import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
@@ -52,6 +54,7 @@ import org.apache.nifi.processor.util.StandardValidators;
@EventDriven
@SupportsBatching
@Tags({"hash", "dupe", "duplicate", "dedupe"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
@CapabilityDescription("Caches a value, computed from FlowFile attributes, for
each incoming FlowFile and determines if the cached value has already been
seen. "
+ "If so, routes the FlowFile to 'duplicate' with an attribute named
'original.identifier' that specifies the original FlowFile's"
+ "\"description\", which is specified in the
property. If the FlowFile is not determined to be a duplicate, the Processor "
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
index afff3c4..73ada84 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
@@ -32,9 +32,11 @@ import java.util.concurrent.atomic.AtomicReference;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.annotation.behavior.DynamicProperty;
import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
import org.apache.nifi.annotation.behavior.SideEffectFree;
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.behavior.DynamicRelationship;
import org.apache.nifi.annotation.documentation.Tags;
@@ -57,6 +59,7 @@ import org.apache.nifi.processor.util.StandardValidators;
@EventDriven
@SideEffectFree
@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
@TriggerWhenAnyDestinationAvailable
@Tags({"distribute", "load balance", "route", "round robin", "weighted"})
@CapabilityDescription("Distributes FlowFiles to downstream processors based
on a Distribution Strategy. If using the Round Robin "
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
index 7400821..021a94f 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-proce
[08/19] nifi git commit: NIFI-810: Updated the wording on validation errors due to upstream connections
NIFI-810: Updated the wording on validation errors due to upstream connections
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/5ecdb185
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/5ecdb185
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/5ecdb185
Branch: refs/heads/master
Commit: 5ecdb1858e94938cd426b12bc48d3725109c6e96
Parents: 13edcfd
Author: Mark Payne
Authored: Wed Oct 7 17:26:04 2015 -0400
Committer: Mark Payne
Committed: Wed Oct 7 17:26:14 2015 -0400
--
.../nifi/controller/StandardProcessorNode.java | 2475 +-
1 file changed, 1237 insertions(+), 1238 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/5ecdb185/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
index 0c39eda..3c816d0 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
@@ -74,1242 +74,1241 @@ import org.slf4j.LoggerFactory;
*/
public class StandardProcessorNode extends ProcessorNode implements
Connectable {
- public static final String BULLETIN_OBSERVER_ID = "bulletin-observer";
-
- public static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS;
- public static final String DEFAULT_YIELD_PERIOD = "1 sec";
- public static final String DEFAULT_PENALIZATION_PERIOD = "30 sec";
- private final AtomicReference processGroup;
- private final Processor processor;
- private final AtomicReference identifier;
- private final Map destinations;
- private final Map> connections;
- private final AtomicReference>
undefinedRelationshipsToTerminate;
- private final AtomicReference> incomingConnectionsRef;
- private final ReentrantReadWriteLock rwLock;
- private final Lock readLock;
- private final Lock writeLock;
- private final AtomicBoolean isolated;
- private final AtomicBoolean lossTolerant;
- private final AtomicReference scheduledState;
- private final AtomicReference comments;
- private final AtomicReference name;
- private final AtomicReference position;
- private final AtomicReference annotationData;
- private final AtomicReference schedulingPeriod; // stored as
string so it's presented to user as they entered it
- private final AtomicReference yieldPeriod;
- private final AtomicReference penalizationPeriod;
- private final AtomicReference> style;
- private final AtomicInteger concurrentTaskCount;
- private final AtomicLong yieldExpiration;
- private final AtomicLong schedulingNanos;
- private final boolean triggerWhenEmpty;
- private final boolean sideEffectFree;
- private final boolean triggeredSerially;
- private final boolean triggerWhenAnyDestinationAvailable;
- private final boolean eventDrivenSupported;
- private final boolean batchSupported;
- private final Requirement inputRequirement;
- private final ValidationContextFactory validationContextFactory;
- private final ProcessScheduler processScheduler;
- private long runNanos = 0L;
-
- private SchedulingStrategy schedulingStrategy; // guarded by
read/write lock
-
- @SuppressWarnings("deprecation")
- public StandardProcessorNode(final Processor processor, final String
uuid, final ValidationContextFactory validationContextFactory,
- final ProcessScheduler scheduler, final
ControllerServiceProvider controllerServiceProvider) {
- super(processor, uuid, validationContextFactory,
controllerServiceProvider);
-
- this.processor = processor;
- identifier = new AtomicReference<>(uuid);
- destinations = new HashMap<>();
- connections = new HashMap<>();
- incomingConnectionsRef = new
AtomicReference>(new ArrayList());
- scheduledState = new AtomicReference<>(ScheduledState.STOPPED);
- rwLock = new ReentrantReadWriteLock(false);
- readLock = rwLock.readLock();
- writeLock = rwLock.writeLock();
- lossTolerant = new
[03/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
--
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
index cbcc54d..385ac73 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
@@ -23,7 +23,8 @@ import java.util.List;
import java.util.Set;
import org.apache.hadoop.io.SequenceFile;
-import org.apache.hadoop.io.SequenceFile.CompressionType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.SideEffectFree;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.SeeAlso;
@@ -57,6 +58,7 @@ import
org.apache.nifi.processors.hadoop.util.SequenceFileWriter;
*
*/
@SideEffectFree
+@InputRequirement(Requirement.INPUT_REQUIRED)
@Tags({"hadoop", "sequence file", "create", "sequencefile"})
@CapabilityDescription("Creates Hadoop Sequence Files from incoming flow
files")
@SeeAlso(PutHDFS.class)
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
--
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
index 4a52fb7..aa03e73 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
@@ -29,6 +29,8 @@ import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.AccessControlException;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
@@ -44,6 +46,7 @@ import org.apache.nifi.processor.util.StandardValidators;
import org.apache.nifi.util.StopWatch;
@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
@Tags({"hadoop", "hdfs", "get", "ingest", "fetch", "source"})
@CapabilityDescription("Retrieves a file from HDFS. The content of the
incoming FlowFile is replaced by the content of the file in HDFS. "
+ "The file in HDFS is left intact without any changes being made to
it.")
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
--
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
index de776d4..4c9deea 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
@@ -41,6 +41,8 @@ import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathFilter;
import org.apache.hadoop.io.compress.CompressionCodec;
import org.apache.hadoop.io.compress.CompressionCodecFactory;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.behavior.WritesAttributes;
@@ -62,6 +64,7 @@ import org.apache.nifi.processor.util.StandardValidators;
import org.apache.nifi.util.StopWatch;
@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_ALLOWED)
@Tags({"hadoop",
[16/19] nifi git commit: NIFI-810: Addressed several checkstyle violations
NIFI-810: Addressed several checkstyle violations
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/ccfb57fe
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/ccfb57fe
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/ccfb57fe
Branch: refs/heads/master
Commit: ccfb57fe9ff43f11319dcb1625bfc78b1d88f56a
Parents: b974445
Author: Mark Payne
Authored: Wed Oct 7 17:48:51 2015 -0400
Committer: Mark Payne
Committed: Wed Oct 7 17:48:51 2015 -0400
--
.../annotation/behavior/InputRequirement.java | 70 +-
.../nifi/processors/aws/s3/PutS3Object.java | 46 +-
.../apache/nifi/controller/ProcessorNode.java | 88 +--
.../nifi/controller/StandardProcessorNode.java | 10 +-
.../standard/Base64EncodeContent.java | 168 ++---
.../nifi/processors/standard/ControlRate.java | 672 +--
6 files changed, 534 insertions(+), 520 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/ccfb57fe/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/InputRequirement.java
--
diff --git
a/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/InputRequirement.java
b/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/InputRequirement.java
index 97e6b88..13f442c 100644
---
a/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/InputRequirement.java
+++
b/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/InputRequirement.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.nifi.annotation.behavior;
import java.lang.annotation.Documented;
@@ -21,31 +37,31 @@ import java.lang.annotation.Target;
@Retention(RetentionPolicy.RUNTIME)
@Inherited
public @interface InputRequirement {
- Requirement value();
-
- public static enum Requirement {
- /**
-* This value is used to indicate that the Processor requires
input from other Processors
-* in order to run. As a result, the Processor will not be
valid if it does not have any
-* incoming connections.
-*/
- INPUT_REQUIRED,
-
- /**
-* This value is used to indicate that the Processor will
consume data from an incoming
-* connection but does not require an incoming connection in
order to perform its task.
-* If the {@link InputRequirement} annotation is not present,
this is the default value
-* that is used.
-*/
- INPUT_ALLOWED,
-
- /**
-* This value is used to indicate that the Processor is a
"Source Processor" and does
-* not accept incoming connections. Because the Processor does
not pull FlowFiles from
-* an incoming connection, it can be very confusing for users
who create incoming connections
-* to the Processor. As a result, this value can be used in
order to clarify that incoming
-* connections will not be used. This prevents the user from
even creating such a connection.
-*/
- INPUT_FORBIDDEN;
- }
+Requirement value();
+
+public static enum Requirement {
+/**
+ * This value is used to indicate that the Processor requires input
from other Processors
+ * in order to run. As a result, the Processor will not be valid if it
does not have any
+ * incoming connections.
+ */
+INPUT_REQUIRED,
+
+/**
+ * This value is used to indicate that the Processor will consume data
from an incoming
+ * connection but does not require an incoming connection in order to
perform its task.
+ * If the {@link InputRequirement} annotation is not present, this is
the default value
+ * that is used.
+ */
+INPUT_ALLOWED,
+
+/**
+ * This value is used
[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection
[ https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973290#comment-14973290 ] ASF subversion and git services commented on NIFI-810: -- Commit 0636f0e731cd28299edd3a6e9db90de5045ab662 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0636f0e ] NIFI-810: Merged master into branch > Create Annotation that indicates that a Processor cannot be scheduled to run > without an incoming connection > --- > > Key: NIFI-810 > URL: https://issues.apache.org/jira/browse/NIFI-810 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > > Currently, if a Processor has no incoming connections but is started, it will > continually without ever accomplishing anything. We should have an > annotation, perhaps @RequiresInput, that indicates that the Processor should > not be scheduled to run unless it has an incoming connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[01/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
Repository: nifi
Updated Branches:
refs/heads/master f2c4f2d2a -> 8a8006085
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
index 68155d1..98a56bf 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
@@ -16,33 +16,7 @@
*/
package org.apache.nifi.processors.standard;
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessorInitializationContext;
-import org.apache.nifi.processor.DataUnit;
-import org.apache.nifi.processor.ProcessSession;
-import org.apache.nifi.processor.Relationship;
import java.io.BufferedWriter;
-
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.components.Validator;
-import org.apache.nifi.expression.AttributeValueDecorator;
-import org.apache.nifi.flowfile.FlowFile;
-import org.apache.nifi.stream.io.StreamUtils;
-import org.apache.nifi.logging.ProcessorLog;
-import org.apache.nifi.annotation.documentation.CapabilityDescription;
-import org.apache.nifi.annotation.behavior.EventDriven;
-import org.apache.nifi.annotation.behavior.SideEffectFree;
-import org.apache.nifi.annotation.behavior.SupportsBatching;
-import org.apache.nifi.annotation.documentation.Tags;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.processor.io.OutputStreamCallback;
-import org.apache.nifi.processor.io.StreamCallback;
-import org.apache.nifi.processor.util.FlowFileFilters;
-import org.apache.nifi.processor.util.StandardValidators;
-import org.apache.nifi.processors.standard.util.NLKBufferedReader;
-import org.apache.nifi.util.StopWatch;
-
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
@@ -58,9 +32,37 @@ import java.util.concurrent.TimeUnit;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.expression.AttributeValueDecorator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.FlowFileFilters;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.util.NLKBufferedReader;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
@EventDriven
@SideEffectFree
@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
@Tags({"Text", "Regular Expression", "Update", "Change", "Replace", "Modify",
"Regex"})
@CapabilityDescription("Updates the content of a FlowFile by evaluating a
Regular Expression against it and replacing the section of "
+ "the content that matches the Regular Expression with some alternate
value.")
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java
index 0
[13/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/4afd8f88 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/4afd8f88 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/4afd8f88 Branch: refs/heads/master Commit: 4afd8f88f8a34cf87f2a06221667166a54c99a15 Parents: 31fba6b Author: Mark Payne Authored: Fri Sep 25 11:39:28 2015 -0400 Committer: Mark Payne Committed: Wed Oct 7 17:26:14 2015 -0400 -- .../annotation/behavior/InputRequirement.java | 51 + .../src/main/asciidoc/developer-guide.adoc | 11 + .../nifi/processors/avro/ConvertAvroToJSON.java |3 + .../processors/avro/ExtractAvroMetadata.java| 29 +- .../apache/nifi/processors/avro/SplitAvro.java | 27 +- .../nifi/processors/aws/s3/FetchS3Object.java |3 + .../nifi/processors/aws/s3/PutS3Object.java |6 +- .../apache/nifi/processors/aws/sns/PutSNS.java |3 + .../nifi/processors/aws/sqs/DeleteSQS.java |3 + .../apache/nifi/processors/aws/sqs/GetSQS.java |5 +- .../apache/nifi/processors/aws/sqs/PutSQS.java |5 +- .../nifi/processors/flume/ExecuteFlumeSink.java | 14 +- .../processors/flume/ExecuteFlumeSource.java| 14 +- .../apache/nifi/controller/ProcessorNode.java | 89 +- .../nifi/controller/StandardProcessorNode.java | 2440 +- .../org/apache/nifi/processors/GeoEnrichIP.java |3 + .../hadoop/CreateHadoopSequenceFile.java|4 +- .../nifi/processors/hadoop/FetchHDFS.java |3 + .../apache/nifi/processors/hadoop/GetHDFS.java |3 + .../apache/nifi/processors/hadoop/ListHDFS.java |3 + .../apache/nifi/processors/hadoop/PutHDFS.java |3 + .../processors/hl7/ExtractHL7Attributes.java|3 + .../apache/nifi/processors/hl7/RouteHL7.java|3 + .../processors/image/ExtractImageMetadata.java | 36 +- .../nifi/processors/image/ResizeImage.java | 38 +- .../apache/nifi/processors/kafka/GetKafka.java | 21 +- .../apache/nifi/processors/kafka/PutKafka.java | 10 +- .../nifi/processors/kite/ConvertCSVToAvro.java | 16 +- .../nifi/processors/kite/ConvertJSONToAvro.java | 14 +- .../processors/kite/StoreInKiteDataset.java |9 +- .../nifi/processors/yandex/YandexTranslate.java |3 + .../nifi-pcap-processors/.gitignore |1 + .../nifi/processors/twitter/GetTwitter.java |5 +- .../apache/nifi/processors/solr/GetSolr.java| 43 +- .../processors/solr/PutSolrContentStream.java | 33 +- .../standard/Base64EncodeContent.java | 171 +- .../processors/standard/CompressContent.java| 15 +- .../nifi/processors/standard/ControlRate.java | 683 ++--- .../standard/ConvertCharacterSet.java |3 + .../processors/standard/ConvertJSONToSQL.java |3 + .../processors/standard/DetectDuplicate.java|3 + .../processors/standard/DistributeLoad.java |3 + .../processors/standard/DuplicateFlowFile.java |3 + .../nifi/processors/standard/EncodeContent.java | 15 +- .../processors/standard/EncryptContent.java |3 + .../processors/standard/EvaluateJsonPath.java | 38 +- .../nifi/processors/standard/EvaluateXPath.java | 29 +- .../processors/standard/EvaluateXQuery.java | 25 +- .../processors/standard/ExecuteProcess.java |3 + .../nifi/processors/standard/ExecuteSQL.java|3 + .../standard/ExecuteStreamCommand.java |7 +- .../nifi/processors/standard/ExtractText.java |3 + .../processors/standard/GenerateFlowFile.java | 11 +- .../apache/nifi/processors/standard/GetFTP.java | 13 +- .../nifi/processors/standard/GetFile.java |7 +- .../nifi/processors/standard/GetHTTP.java |3 + .../nifi/processors/standard/GetJMSQueue.java |3 + .../nifi/processors/standard/GetJMSTopic.java |3 + .../nifi/processors/standard/GetSFTP.java |7 +- .../processors/standard/HandleHttpRequest.java |7 +- .../processors/standard/HandleHttpResponse.java |5 +- .../nifi/processors/standard/HashAttribute.java |5 +- .../nifi/processors/standard/HashContent.java |5 +- .../processors/standard/IdentifyMimeType.java |5 +- .../nifi/processors/standard/InvokeHTTP.java|3 + .../nifi/processors/standard/ListenHTTP.java| 16 +- .../nifi/processors/standard/ListenUDP.java | 18 +- .../nifi/processors/standard/LogAttribute.java | 16 +- .../nifi/processors/standard/MergeContent.java | 11 +- .../nifi/processors/standard/ModifyBytes.java | 14 +- .../processors/standard/MonitorActivity.java| 31 +- .../nifi/processors/standard/PostHTTP.java |3 + .../nifi/processors/standard/PutEmail.java |3 + .../apache/nifi/processors/standard/PutFTP.ja
[11/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
--
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
index cbcc54d..385ac73 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
@@ -23,7 +23,8 @@ import java.util.List;
import java.util.Set;
import org.apache.hadoop.io.SequenceFile;
-import org.apache.hadoop.io.SequenceFile.CompressionType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.SideEffectFree;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.SeeAlso;
@@ -57,6 +58,7 @@ import
org.apache.nifi.processors.hadoop.util.SequenceFileWriter;
*
*/
@SideEffectFree
+@InputRequirement(Requirement.INPUT_REQUIRED)
@Tags({"hadoop", "sequence file", "create", "sequencefile"})
@CapabilityDescription("Creates Hadoop Sequence Files from incoming flow
files")
@SeeAlso(PutHDFS.class)
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
--
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
index 4a52fb7..aa03e73 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
@@ -29,6 +29,8 @@ import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.AccessControlException;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
@@ -44,6 +46,7 @@ import org.apache.nifi.processor.util.StandardValidators;
import org.apache.nifi.util.StopWatch;
@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
@Tags({"hadoop", "hdfs", "get", "ingest", "fetch", "source"})
@CapabilityDescription("Retrieves a file from HDFS. The content of the
incoming FlowFile is replaced by the content of the file in HDFS. "
+ "The file in HDFS is left intact without any changes being made to
it.")
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
--
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
index de776d4..4c9deea 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
@@ -41,6 +41,8 @@ import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathFilter;
import org.apache.hadoop.io.compress.CompressionCodec;
import org.apache.hadoop.io.compress.CompressionCodecFactory;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.behavior.WritesAttributes;
@@ -62,6 +64,7 @@ import org.apache.nifi.processor.util.StandardValidators;
import org.apache.nifi.util.StopWatch;
@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_ALLOWED)
@Tags({"hadoop",
[19/19] nifi git commit: Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/8a800608 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/8a800608 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/8a800608 Branch: refs/heads/master Commit: 8a8006085190aae1125c883f2aab16a4ada9beb8 Parents: 0636f0e f2c4f2d Author: Mark Payne Authored: Sun Oct 25 11:11:49 2015 -0400 Committer: Mark Payne Committed: Sun Oct 25 11:11:49 2015 -0400 -- .../hadoop/AbstractHadoopProcessor.java | 92 +--- 1 file changed, 79 insertions(+), 13 deletions(-) --
[04/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
index cbd0f88..0c39eda 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
@@ -16,14 +16,6 @@
*/
package org.apache.nifi.controller;
-import org.apache.nifi.annotation.behavior.EventDriven;
-import org.apache.nifi.annotation.behavior.SideEffectFree;
-import org.apache.nifi.annotation.behavior.SupportsBatching;
-import org.apache.nifi.annotation.behavior.TriggerSerially;
-import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
-import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
-import org.apache.nifi.annotation.documentation.CapabilityDescription;
-
import static java.util.Objects.requireNonNull;
import java.util.ArrayList;
@@ -43,6 +35,17 @@ import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.components.ValidationContext;
import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.connectable.Connectable;
@@ -61,8 +64,6 @@ import org.apache.nifi.processor.Processor;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.scheduling.SchedulingStrategy;
import org.apache.nifi.util.FormatUtils;
-import org.apache.commons.lang3.builder.EqualsBuilder;
-import org.apache.commons.lang3.builder.HashCodeBuilder;
import org.quartz.CronExpression;
import org.slf4j.LoggerFactory;
@@ -73,1185 +74,1242 @@ import org.slf4j.LoggerFactory;
*/
public class StandardProcessorNode extends ProcessorNode implements
Connectable {
-public static final String BULLETIN_OBSERVER_ID = "bulletin-observer";
-
-public static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS;
-public static final String DEFAULT_YIELD_PERIOD = "1 sec";
-public static final String DEFAULT_PENALIZATION_PERIOD = "30 sec";
-private final AtomicReference processGroup;
-private final Processor processor;
-private final AtomicReference identifier;
-private final Map destinations;
-private final Map> connections;
-private final AtomicReference>
undefinedRelationshipsToTerminate;
-private final AtomicReference> incomingConnectionsRef;
-private final ReentrantReadWriteLock rwLock;
-private final Lock readLock;
-private final Lock writeLock;
-private final AtomicBoolean isolated;
-private final AtomicBoolean lossTolerant;
-private final AtomicReference scheduledState;
-private final AtomicReference comments;
-private final AtomicReference name;
-private final AtomicReference position;
-private final AtomicReference annotationData;
-private final AtomicReference schedulingPeriod; // stored as
string so it's presented to user as they entered it
-private final AtomicReference yieldPeriod;
-private final AtomicReference penalizationPeriod;
-private final AtomicReference> style;
-private final AtomicInteger concurrentTaskCount;
-private final AtomicLong yieldExpiration;
-private final AtomicLong schedulingNanos;
-private final boolean triggerWhenEmpty;
-private final boolean sideEffectFree;
-private final boolean triggeredSerially;
-private final boolean triggerWhenAnyDestinationAvailable;
-private final boolean eventDrivenSupported;
-private final boolean batchSupported;
-private final ValidationContextFactory validationContextFactory;
-private final ProcessScheduler processScheduler;
-private long runNanos
[15/19] nifi git commit: NIFI-810: rebased from master
NIFI-810: rebased from master Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/b974445d Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/b974445d Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/b974445d Branch: refs/heads/master Commit: b974445ddd38ec7e84995225b86987e6af1af52c Parents: 5ecdb18 2215bc8 Author: Mark Payne Authored: Wed Oct 7 17:28:39 2015 -0400 Committer: Mark Payne Committed: Wed Oct 7 17:28:39 2015 -0400 -- .../nifi/controller/StandardProcessorNode.java| 18 +- 1 file changed, 9 insertions(+), 9 deletions(-) --
[05/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/034ee6de Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/034ee6de Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/034ee6de Branch: refs/heads/master Commit: 034ee6de6bc4c6923a835fbeaab4fb05fd694434 Parents: 96764ed Author: Mark Payne Authored: Fri Sep 25 11:39:28 2015 -0400 Committer: Mark Payne Committed: Fri Sep 25 11:39:58 2015 -0400 -- .../annotation/behavior/InputRequirement.java | 51 + .../src/main/asciidoc/developer-guide.adoc | 11 + .../nifi/processors/avro/ConvertAvroToJSON.java |3 + .../processors/avro/ExtractAvroMetadata.java| 29 +- .../apache/nifi/processors/avro/SplitAvro.java | 27 +- .../nifi/processors/aws/s3/FetchS3Object.java |3 + .../nifi/processors/aws/s3/PutS3Object.java |6 +- .../apache/nifi/processors/aws/sns/PutSNS.java |3 + .../nifi/processors/aws/sqs/DeleteSQS.java |3 + .../apache/nifi/processors/aws/sqs/GetSQS.java |5 +- .../apache/nifi/processors/aws/sqs/PutSQS.java |5 +- .../nifi/processors/flume/ExecuteFlumeSink.java | 14 +- .../processors/flume/ExecuteFlumeSource.java| 14 +- .../apache/nifi/controller/ProcessorNode.java | 89 +- .../nifi/controller/StandardProcessorNode.java | 2440 +- .../org/apache/nifi/processors/GeoEnrichIP.java |3 + .../hadoop/CreateHadoopSequenceFile.java|4 +- .../nifi/processors/hadoop/FetchHDFS.java |3 + .../apache/nifi/processors/hadoop/GetHDFS.java |3 + .../apache/nifi/processors/hadoop/ListHDFS.java |3 + .../apache/nifi/processors/hadoop/PutHDFS.java |3 + .../processors/hl7/ExtractHL7Attributes.java|3 + .../apache/nifi/processors/hl7/RouteHL7.java|3 + .../processors/image/ExtractImageMetadata.java | 36 +- .../nifi/processors/image/ResizeImage.java | 38 +- .../apache/nifi/processors/kafka/GetKafka.java | 21 +- .../apache/nifi/processors/kafka/PutKafka.java | 10 +- .../nifi/processors/kite/ConvertCSVToAvro.java | 16 +- .../nifi/processors/kite/ConvertJSONToAvro.java | 14 +- .../processors/kite/StoreInKiteDataset.java |9 +- .../nifi/processors/yandex/YandexTranslate.java |3 + .../nifi-pcap-processors/.gitignore |1 + .../nifi/processors/twitter/GetTwitter.java |5 +- .../apache/nifi/processors/solr/GetSolr.java| 43 +- .../processors/solr/PutSolrContentStream.java | 33 +- .../standard/Base64EncodeContent.java | 171 +- .../processors/standard/CompressContent.java| 15 +- .../nifi/processors/standard/ControlRate.java | 683 ++--- .../standard/ConvertCharacterSet.java |3 + .../processors/standard/ConvertJSONToSQL.java |3 + .../processors/standard/DetectDuplicate.java|3 + .../processors/standard/DistributeLoad.java |3 + .../processors/standard/DuplicateFlowFile.java |3 + .../nifi/processors/standard/EncodeContent.java | 15 +- .../processors/standard/EncryptContent.java |3 + .../processors/standard/EvaluateJsonPath.java | 38 +- .../nifi/processors/standard/EvaluateXPath.java | 29 +- .../processors/standard/EvaluateXQuery.java | 25 +- .../processors/standard/ExecuteProcess.java |3 + .../nifi/processors/standard/ExecuteSQL.java|3 + .../standard/ExecuteStreamCommand.java |7 +- .../nifi/processors/standard/ExtractText.java |3 + .../processors/standard/GenerateFlowFile.java | 11 +- .../apache/nifi/processors/standard/GetFTP.java | 13 +- .../nifi/processors/standard/GetFile.java |7 +- .../nifi/processors/standard/GetHTTP.java |3 + .../nifi/processors/standard/GetJMSQueue.java |3 + .../nifi/processors/standard/GetJMSTopic.java |3 + .../nifi/processors/standard/GetSFTP.java |7 +- .../processors/standard/HandleHttpRequest.java |7 +- .../processors/standard/HandleHttpResponse.java |5 +- .../nifi/processors/standard/HashAttribute.java |5 +- .../nifi/processors/standard/HashContent.java |5 +- .../processors/standard/IdentifyMimeType.java |5 +- .../nifi/processors/standard/InvokeHTTP.java|3 + .../nifi/processors/standard/ListenHTTP.java| 16 +- .../nifi/processors/standard/ListenUDP.java | 18 +- .../nifi/processors/standard/LogAttribute.java | 16 +- .../nifi/processors/standard/MergeContent.java | 11 +- .../nifi/processors/standard/ModifyBytes.java | 14 +- .../processors/standard/MonitorActivity.java| 31 +- .../nifi/processors/standard/PostHTTP.java |3 + .../nifi/processors/standard/PutEmail.java |3 + .../apache/nifi/processors/standard/PutFTP.j
[10/19] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
index b825972..39dc725 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
@@ -28,11 +28,13 @@ import java.util.concurrent.TimeUnit;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.SeeAlso;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.distributed.cache.client.Deserializer;
import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
@@ -52,6 +54,7 @@ import org.apache.nifi.processor.util.StandardValidators;
@EventDriven
@SupportsBatching
@Tags({"hash", "dupe", "duplicate", "dedupe"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
@CapabilityDescription("Caches a value, computed from FlowFile attributes, for
each incoming FlowFile and determines if the cached value has already been
seen. "
+ "If so, routes the FlowFile to 'duplicate' with an attribute named
'original.identifier' that specifies the original FlowFile's"
+ "\"description\", which is specified in the
property. If the FlowFile is not determined to be a duplicate, the Processor "
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
index afff3c4..73ada84 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
@@ -32,9 +32,11 @@ import java.util.concurrent.atomic.AtomicReference;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.annotation.behavior.DynamicProperty;
import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
import org.apache.nifi.annotation.behavior.SideEffectFree;
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.behavior.DynamicRelationship;
import org.apache.nifi.annotation.documentation.Tags;
@@ -57,6 +59,7 @@ import org.apache.nifi.processor.util.StandardValidators;
@EventDriven
@SideEffectFree
@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
@TriggerWhenAnyDestinationAvailable
@Tags({"distribute", "load balance", "route", "round robin", "weighted"})
@CapabilityDescription("Distributes FlowFiles to downstream processors based
on a Distribution Strategy. If using the Round Robin "
http://git-wip-us.apache.org/repos/asf/nifi/blob/4afd8f88/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
index 7400821..021a94f 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-proce
[07/19] nifi git commit: NIFI-810: - Adding basic support for preventing connection when appropriate. - Updating validation when [dis]connecting processors.
NIFI-810:
- Adding basic support for preventing connection when appropriate.
- Updating validation when [dis]connecting processors.
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/13edcfda
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/13edcfda
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/13edcfda
Branch: refs/heads/master
Commit: 13edcfda2ef830e1b160b31f7cd3bea874ccd3f0
Parents: 4afd8f8
Author: Matt Gilman
Authored: Fri Sep 25 17:46:58 2015 -0400
Committer: Mark Payne
Committed: Wed Oct 7 17:26:14 2015 -0400
--
.../org/apache/nifi/web/api/dto/ProcessorDTO.java| 15 +++
.../org/apache/nifi/controller/TemplateManager.java | 1 +
.../java/org/apache/nifi/web/api/dto/DtoFactory.java | 1 +
.../src/main/webapp/js/nf/canvas/nf-actions.js | 4 +++-
.../src/main/webapp/js/nf/canvas/nf-canvas-utils.js | 13 ++---
.../js/nf/canvas/nf-connection-configuration.js | 8 ++--
6 files changed, 36 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/13edcfda/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
index c65c46a..866d77c 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ProcessorDTO.java
@@ -36,6 +36,7 @@ public class ProcessorDTO extends NiFiComponentDTO {
private String description;
private Boolean supportsParallelProcessing;
private Boolean supportsEventDriven;
+private String inputRequirement;
private ProcessorConfigDTO config;
@@ -121,6 +122,20 @@ public class ProcessorDTO extends NiFiComponentDTO {
}
/**
+ * @return the input requirement of this processor
+ */
+@ApiModelProperty(
+value = "The input requirement for this processor."
+)
+public String getInputRequirement() {
+return inputRequirement;
+}
+
+public void setInputRequirement(String inputRequirement) {
+this.inputRequirement = inputRequirement;
+}
+
+/**
* @return whether this processor supports event driven scheduling
*/
@ApiModelProperty(
http://git-wip-us.apache.org/repos/asf/nifi/blob/13edcfda/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
index 7b8e173..a332e05 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/TemplateManager.java
@@ -320,6 +320,7 @@ public class TemplateManager {
// remove validation errors
processorDTO.setValidationErrors(null);
+processorDTO.setInputRequirement(null);
}
}
http://git-wip-us.apache.org/repos/asf/nifi/blob/13edcfda/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
index 76bce6f..16b114e 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java
@@ -1402,6 +1402,7 @@ public final class DtoFactory {
dto.setPosition(createPositionDto(node.getPosition()));
dto.setStyle(node
[jira] [Updated] (NIFI-1056) ExecuteSQL fails when trying to pull a NUMERIC value from Oracle database
[ https://issues.apache.org/jira/browse/NIFI-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Kurc updated NIFI-1056: Affects Version/s: 0.3.0 > ExecuteSQL fails when trying to pull a NUMERIC value from Oracle database > - > > Key: NIFI-1056 > URL: https://issues.apache.org/jira/browse/NIFI-1056 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.3.0 >Reporter: Mark Payne >Assignee: Mark Payne > Labels: oracle, sql > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-1056-Do-not-treat-BigDecimal-BigInteger-the-sam.patch > > > We received the following error message from the mailing list: > 2015-10-20 15:59:59,859 ERROR [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] failed to process due to > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence; rolling back session: > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,860 ERROR [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] failed to process session > due to org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence: > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,860 WARN [Timer-Driven Process Thread-9] > o.a.nifi.processors.standard.ExecuteSQL > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] Processor > Administratively Yielded for 1 sec due to processing failure > 2015-10-20 15:59:59,860 WARN [Timer-Driven Process Thread-9] > o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding > ExecuteSQL[id=f92d313d-fc87-42a9-ac24-ce7d9b6972c9] due to uncaught > Exception: org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > 2015-10-20 15:59:59,864 WARN [Timer-Driven Process Thread-9] > o.a.n.c.t.ContinuallyRunProcessorTask > org.apache.avro.file.DataFileWriter$AppendWriteException: > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > java.lang.CharSequence > at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296) > ~[na:na] > at > org.apache.nifi.processors.standard.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:87) > ~[na:na] > at > org.apache.nifi.processors.standard.ExecuteSQL$1.process(ExecuteSQL.java:142) > ~[na:na] > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.processors.standard.ExecuteSQL.onTrigger(ExecuteSQL.java:136) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127) > [nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49) > [nifi-framework-core-0.3.0.jar:0.3.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) > [nifi-framework-core-0.3.0.jar:0.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_79] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [na:1.7.0_79] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [na:1.7.0_79] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [na:1.7.0_79] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_79] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_79] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] > Caused by: java.lang.ClassCastException: java.math.BigDec
[25/41] nifi git commit: NIFI-1016 Naive fix so org.apache.nifi.prioritizer.PriorityAttributePrioritizerTest builds
NIFI-1016 Naive fix so
org.apache.nifi.prioritizer.PriorityAttributePrioritizerTest builds
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/92005422
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/92005422
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/92005422
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 9200542298c33bd7527a5329205527445613b2b4
Parents: 88b1b84
Author: Tony Kurc
Authored: Sun Oct 18 20:34:00 2015 -0400
Committer: Tony Kurc
Committed: Sun Oct 18 20:34:36 2015 -0400
--
.../PriorityAttributePrioritizerTest.java | 17 -
1 file changed, 8 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/92005422/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/test/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizerTest.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/test/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizerTest.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/test/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizerTest.java
index d7d278c..7098551 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/test/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizerTest.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/test/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizerTest.java
@@ -16,14 +16,13 @@
*/
package org.apache.nifi.prioritizer;
-import org.apache.nifi.prioritizer.PriorityAttributePrioritizer;
-
import static org.junit.Assert.assertEquals;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.atomic.AtomicLong;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
@@ -48,13 +47,13 @@ public class PriorityAttributePrioritizerTest {
@BeforeClass
public static void init() {
-attrsPri1.put(PriorityAttributePrioritizer.PRIORITY_ATTR, "1");
-attrsPri2.put(PriorityAttributePrioritizer.PRIORITY_ATTR, "2");
-attrsPrin1.put(PriorityAttributePrioritizer.PRIORITY_ATTR, "-1");
-attrsPriA.put(PriorityAttributePrioritizer.PRIORITY_ATTR, "A");
-attrsPriB.put(PriorityAttributePrioritizer.PRIORITY_ATTR, "B");
-attrsPriLP.put(PriorityAttributePrioritizer.PRIORITY_ATTR,
"5432123456789");
-attrsPriLN.put(PriorityAttributePrioritizer.PRIORITY_ATTR,
"-5432123456789");
+attrsPri1.put(CoreAttributes.PRIORITY.key(), "1");
+attrsPri2.put(CoreAttributes.PRIORITY.key(), "2");
+attrsPrin1.put(CoreAttributes.PRIORITY.key(), "-1");
+attrsPriA.put(CoreAttributes.PRIORITY.key(), "A");
+attrsPriB.put(CoreAttributes.PRIORITY.key(), "B");
+attrsPriLP.put(CoreAttributes.PRIORITY.key(), "5432123456789");
+attrsPriLN.put(CoreAttributes.PRIORITY.key(), "-5432123456789");
}
@Test
[07/41] nifi git commit: Remove the process to check if the target key exists or not
Remove the process to check if the target key exists or not
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/d32a32a9
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/d32a32a9
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/d32a32a9
Branch: refs/heads/NIFI-810-InputRequirement
Commit: d32a32a92d29707d22899862594a2679d3fb2273
Parents: c492a1a
Author: Yu ISHIKAWA
Authored: Mon Sep 7 23:32:48 2015 +0900
Committer: Yu ISHIKAWA
Committed: Mon Sep 7 23:32:48 2015 +0900
--
.../nifi/processors/aws/s3/DeleteS3Object.java | 15 ---
.../nifi/processors/aws/s3/TestDeleteS3Object.java | 14 ++
2 files changed, 2 insertions(+), 27 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/d32a32a9/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
index dffcab8..c8950c3 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
@@ -65,11 +65,6 @@ public class DeleteS3Object extends AbstractS3Processor {
new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE,
REL_NOT_FOUND)));
@Override
-public Set getRelationships() {
-return relationships;
-}
-
-@Override
protected List getSupportedPropertyDescriptors() {
return properties;
}
@@ -89,16 +84,6 @@ public class DeleteS3Object extends AbstractS3Processor {
final AmazonS3 s3 = getClient();
-// Checks if the key exists or not
-// If there is no such a key, then throws a exception
-try {
- s3.getObjectMetadata(bucket, key);
-} catch (final AmazonServiceException ase) {
-getLogger().error("Not found such a S3 object for {}; routing to
not found", new Object[]{flowFile, ase});
-session.transfer(flowFile, REL_NOT_FOUND);
-return;
-}
-
// Deletes a key on Amazon S3
try {
if (versionId == null) {
http://git-wip-us.apache.org/repos/asf/nifi/blob/d32a32a9/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
index 04d9e61..082a80d 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
@@ -22,12 +22,9 @@ import java.io.IOException;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
-import java.util.Set;
-
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
-import static org.junit.Assert.assertEquals;
import com.amazonaws.auth.PropertiesCredentials;
import com.amazonaws.services.s3.AmazonS3Client;
@@ -36,7 +33,7 @@ import com.amazonaws.services.s3.model.CreateBucketRequest;
import com.amazonaws.services.s3.model.DeleteBucketRequest;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.PutObjectResult;
-import org.apache.nifi.processor.Relationship;
+
import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;
@@ -128,14 +125,7 @@ public class TestDeleteS3Object {
runner.enqueue(new byte[0], attrs);
runner.run(1);
-runner.assertAllFlowFilesTransferred(DeleteS3Object.REL_NOT_FOUND, 1);
-}
-
-@Test
-public void testGetRelationships() {
-DeleteS3Object deleter = new DeleteS3Object();
-Set relationships = deleter.getRelationships();
-assertEquals(relationships.size(), 3);
+runner.assertAllFlowFilesTransferred(DeleteS3Object.REL_FAILURE, 1);
}
// Uploads a test file
[12/41] nifi git commit: NIFI-442: NIFI-828: - Always selecting the first item in the new component table. - Enabling adding the selected component by typing Enter. - Removing the 'filter by' in the n
NIFI-442: NIFI-828: - Always selecting the first item in the new component table. - Enabling adding the selected component by typing Enter. - Removing the 'filter by' in the new component dialogs and instead just searching every field. Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/2583d786 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/2583d786 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/2583d786 Branch: refs/heads/NIFI-810-InputRequirement Commit: 2583d7869acedcbe536d4dd5eaf2e7c8f9cf149c Parents: 31fba6b Author: Matt Gilman Authored: Wed Oct 7 19:11:55 2015 -0400 Committer: Matt Gilman Committed: Wed Oct 7 19:11:55 2015 -0400 -- .../canvas/new-controller-service-dialog.jsp| 1 - .../partials/canvas/new-processor-dialog.jsp| 1 - .../canvas/new-reporting-task-dialog.jsp| 1 - .../css/new-controller-service-dialog.css | 9 -- .../main/webapp/css/new-processor-dialog.css| 9 -- .../webapp/css/new-reporting-task-dialog.css| 9 -- .../webapp/js/nf/canvas/nf-canvas-toolbox.js| 60 .../src/main/webapp/js/nf/canvas/nf-settings.js | 140 +-- 8 files changed, 105 insertions(+), 125 deletions(-) -- http://git-wip-us.apache.org/repos/asf/nifi/blob/2583d786/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-controller-service-dialog.jsp -- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-controller-service-dialog.jsp b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-controller-service-dialog.jsp index eb54ace..9463cb9 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-controller-service-dialog.jsp +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-controller-service-dialog.jsp @@ -20,7 +20,6 @@ - Displaying  of http://git-wip-us.apache.org/repos/asf/nifi/blob/2583d786/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-processor-dialog.jsp -- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-processor-dialog.jsp b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-processor-dialog.jsp index df7766c..ad33c1c 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-processor-dialog.jsp +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-processor-dialog.jsp @@ -20,7 +20,6 @@ - Displaying  of http://git-wip-us.apache.org/repos/asf/nifi/blob/2583d786/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-reporting-task-dialog.jsp -- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-reporting-task-dialog.jsp b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-reporting-task-dialog.jsp index cfb3992..2025aeb 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-reporting-task-dialog.jsp +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/new-reporting-task-dialog.jsp @@ -20,7 +20,6 @@ - Displaying  of http://git-wip-us.apache.org/repos/asf/nifi/blob/2583d786/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/css/new-controller-service-dialog.css -- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/css/new-controller-service-dialog.css b/nifi-nar-bundles/n
[04/41] nifi git commit: Add @Override annotations
Add @Override annotations
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/213f507f
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/213f507f
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/213f507f
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 213f507f53b3afa3eef3398ebdd3006fa1cbe292
Parents: 6cbc6db
Author: Yu ISHIKAWA
Authored: Wed Sep 2 13:02:39 2015 +0900
Committer: Yu ISHIKAWA
Committed: Wed Sep 2 13:02:39 2015 +0900
--
.../java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/213f507f/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
index 85fc70e..2cc00db 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
@@ -56,10 +56,12 @@ public class DeleteS3Object extends AbstractS3Processor {
Arrays.asList(KEY, BUCKET, ACCESS_KEY, SECRET_KEY,
CREDENTAILS_FILE, REGION, TIMEOUT, VERSION_ID,
FULL_CONTROL_USER_LIST, READ_USER_LIST, WRITE_USER_LIST,
READ_ACL_LIST, WRITE_ACL_LIST, OWNER));
+@Override
protected List getSupportedPropertyDescriptors() {
return properties;
}
+@Override
public void onTrigger(final ProcessContext context, final ProcessSession
session) {
FlowFile flowFile = session.get();
if (flowFile == null) {
[34/41] nifi git commit: NIFI-972 attribute to indicate rows count and cleanup
NIFI-972 attribute to indicate rows count and cleanup
Signed-off-by: Toivo Adams
Signed-off-by: Mark Payne
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/a9e53250
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/a9e53250
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/a9e53250
Branch: refs/heads/NIFI-810-InputRequirement
Commit: a9e5325047fbe3ad8a7d53b664e7944e39fcf658
Parents: ba3225f
Author: Toivo Adams
Authored: Fri Oct 23 12:44:27 2015 +0300
Committer: Mark Payne
Committed: Fri Oct 23 09:28:03 2015 -0400
--
.../apache/nifi/processors/standard/ExecuteSQL.java| 9 -
.../nifi/processors/standard/util/JdbcCommon.java | 13 -
2 files changed, 8 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/a9e53250/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
index 45fd1a8..2a13f32 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
@@ -53,9 +53,13 @@ import org.apache.nifi.util.StopWatch;
+ " Streaming is used so arbitrarily large result sets are supported. This
processor can be scheduled to run on " +
"a timer, or cron expression, using the standard scheduling methods,
or it can be triggered by an incoming FlowFile. " +
"If it is triggered by an incoming FlowFile, then attributes of that
FlowFile will be available when evaluating the " +
-"select query.")
+"select query. " +
+"FlowFile attribute 'executesql.row.count' indicates how many rows
were selected."
+)
public class ExecuteSQL extends AbstractProcessor {
+public static final String RESULT_ROW_COUNT = "executesql.row.count";
+
// Relationships
public static final Relationship REL_SUCCESS = new Relationship.Builder()
.name("success")
@@ -153,6 +157,9 @@ public class ExecuteSQL extends AbstractProcessor {
}
});
+// set attribute how many rows were selected
+outgoing = session.putAttribute(outgoing, RESULT_ROW_COUNT,
nrOfRows.get().toString());
+
logger.info("{} contains {} Avro records", new Object[] {
nrOfRows.get() });
logger.info("Transferred {} to 'success'", new Object[] { outgoing
});
session.getProvenanceReporter().modifyContent(outgoing, "Retrieved
" + nrOfRows.get() + " rows", stopWatch.getElapsed(TimeUnit.MILLISECONDS));
http://git-wip-us.apache.org/repos/asf/nifi/blob/a9e53250/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
index de3d5d1..9cf9338 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
@@ -134,53 +134,41 @@ public class JdbcCommon {
case NCHAR:
case NVARCHAR:
case VARCHAR:
-//
builder.name(meta.getColumnName(i)).type().stringType().noDefault();
-//
builder.name(meta.getColumnName(i)).type().nullable().stringType().noDefault();
-//
builder.name(meta.getColumnName(i)).type().stringType().stringDefault(null);
builder.name(meta.getColumnName(i)).type().unionOf().nullBuilder().endNull().and().stringType().endUnion().noDefault();
-
-
break;
case BOOLEAN:
-//
builder.name(meta.getColumnName(i)).type().nullable().booleanType().noDefault();
builder.name(meta.getColumnName(i)).type().unionOf().nullBuilder().endNull().and().booleanType().endUnion().noDefault()
[19/41] nifi git commit: NIFI-612 Remove FlowUnmarshaller. Builds, runs, could not find any latent references using reflection or other ways of loading classes without direct references. This closes #
NIFI-612 Remove FlowUnmarshaller. Builds, runs, could not find any latent
references using reflection or other ways of loading classes without direct
references. This closes #103. Signed off by Tony Kurc
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/9a8d763d
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/9a8d763d
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/9a8d763d
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 9a8d763d8dddb8feb7e4b176cd97a7790436f60c
Parents: ce7d098
Author: Venkatesh Sellappa
Authored: Sat Oct 17 09:43:10 2015 -0400
Committer: Tony Kurc
Committed: Sat Oct 17 09:43:10 2015 -0400
--
.../nifi/controller/FlowUnmarshaller.java | 77
1 file changed, 77 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/9a8d763d/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowUnmarshaller.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowUnmarshaller.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowUnmarshaller.java
deleted file mode 100644
index c8d90d7..000
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowUnmarshaller.java
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.controller;
-
-import java.io.IOException;
-import java.util.HashSet;
-import java.util.Objects;
-import java.util.Set;
-
-import javax.xml.parsers.DocumentBuilder;
-import javax.xml.parsers.DocumentBuilderFactory;
-import javax.xml.parsers.ParserConfigurationException;
-
-import org.apache.nifi.encrypt.StringEncryptor;
-import org.apache.nifi.stream.io.ByteArrayInputStream;
-import org.apache.nifi.web.api.dto.FlowSnippetDTO;
-import org.apache.nifi.web.api.dto.ProcessGroupDTO;
-
-import org.w3c.dom.Document;
-import org.w3c.dom.Element;
-import org.w3c.dom.NodeList;
-import org.xml.sax.SAXException;
-
-public class FlowUnmarshaller {
-
-/**
- * Interprets the given byte array as an XML document that conforms to the
Flow Configuration schema and returns a FlowSnippetDTO representing the flow
- *
- * @param flowContents contents
- * @param encryptor encryptor
- * @return snippet dto
- * @throws NullPointerException if flowContents is null
- * @throws IOException ioe
- * @throws SAXException sax
- * @throws ParserConfigurationException pe
- */
-public static FlowSnippetDTO unmarshal(final byte[] flowContents, final
StringEncryptor encryptor) throws IOException, SAXException,
ParserConfigurationException {
-if (Objects.requireNonNull(flowContents).length == 0) {
-return new FlowSnippetDTO();
-}
-
-final DocumentBuilderFactory dbf =
DocumentBuilderFactory.newInstance();
-dbf.setNamespaceAware(true);
-
-final DocumentBuilder docBuilder = dbf.newDocumentBuilder();
-final Document document = docBuilder.parse(new
ByteArrayInputStream(flowContents));
-final FlowSnippetDTO flowDto = new FlowSnippetDTO();
-
-final NodeList nodeList = document.getElementsByTagName("rootGroup");
-if (nodeList.getLength() == 0) {
-return flowDto;
-}
-if (nodeList.getLength() > 1) {
-throw new IllegalArgumentException("Contents contain multiple
rootGroup elements");
-}
-
-final Set rootGroupSet = new HashSet<>();
-flowDto.setProcessGroups(rootGroupSet);
-rootGroupSet.add(FlowFromDOMFactory.getProcessGroup(null, (Element)
nodeList.item(0), encryptor));
-
-return flowDto;
-}
-}
[20/41] nifi git commit: NIFI-995 TestGetFile.testAttributes() failed in case of a NTFS partition in Linux, because Files.setPosixFilePermissions() did not have any effect on the files and did not thr
NIFI-995 TestGetFile.testAttributes() failed in case of a NTFS partition in
Linux, because Files.setPosixFilePermissions() did not have any effect on the
files and did not throw Exception. closes #95. Signed off by Tony Kurc
(tk...@apache.org)
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/22924c65
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/22924c65
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/22924c65
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 22924c656bfdcf749bd618f055ffa816aca17ae5
Parents: 9a8d763
Author: Joe
Authored: Sat Oct 17 10:46:49 2015 -0400
Committer: Tony Kurc
Committed: Sat Oct 17 10:46:49 2015 -0400
--
.../nifi/processors/standard/TestGetFile.java | 21 ++--
1 file changed, 19 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/22924c65/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
index 018cbdc..eb8a764 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
@@ -26,6 +26,7 @@ import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.PosixFilePermissions;
import java.text.DateFormat;
import java.text.ParseException;
@@ -33,6 +34,7 @@ import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;
import java.util.Locale;
+import java.util.Set;
import org.apache.nifi.flowfile.attributes.CoreAttributes;
import org.apache.nifi.util.MockFlowFile;
@@ -158,8 +160,23 @@ public class TestGetFile {
boolean verifyPermissions = false;
try {
-Files.setPosixFilePermissions(targetPath,
PosixFilePermissions.fromString("r--r-"));
-verifyPermissions = true;
+// If you mount an NTFS partition in Linux, you are unable to
change the permissions of the files,
+// because every file has the same permissions, controlled by the
'fmask' and 'dmask' mount options.
+// Executing a chmod command will not fail, but it does not change
the file's permissions.
+// From Java perspective the NTFS mount point, as a FileStore
supports the 'unix' and 'posix' file
+// attribute views, but the setPosixFilePermissions() has no
effect.
+//
+// If you set verifyPermissions to true without the following
extra check, the test case will fail
+// on a file system, where Nifi source is located on a NTFS mount
point in Linux.
+// The purpose of the extra check is to ensure, that
setPosixFilePermissions() changes the file's
+// permissions, and set verifyPermissions, after we are convinced.
+Set perms =
PosixFilePermissions.fromString("r--r-");
+Files.setPosixFilePermissions(targetPath, perms);
+Set permsAfterSet =
Files.getPosixFilePermissions(targetPath);
+if (perms.equals(permsAfterSet)) {
+ verifyPermissions = true;
+}
+
} catch (Exception donothing) {
}
[15/41] nifi git commit: NIFI-1024 Correcting references to nifi.cluster.manager.protocol.port
NIFI-1024 Correcting references to nifi.cluster.manager.protocol.port Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/f6d34279 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/f6d34279 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/f6d34279 Branch: refs/heads/NIFI-810-InputRequirement Commit: f6d3427955c48052bc214e08efcb24934d9a4cba Parents: f798127 Author: Aldrin Piri Authored: Tue Oct 6 13:00:30 2015 -0400 Committer: Aldrin Piri Committed: Fri Oct 9 14:22:00 2015 -0400 -- nifi-docs/src/main/asciidoc/administration-guide.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/nifi/blob/f6d34279/nifi-docs/src/main/asciidoc/administration-guide.adoc -- diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi-docs/src/main/asciidoc/administration-guide.adoc index 75523bf..fb109c9 100644 --- a/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -336,7 +336,7 @@ For the NCM, the minimum properties to configure are as follows: * Under the Web Properties, set either the http or https port that you want the NCM to run on. If the NCM and one of the nodes are on the same server, make sure this port is different from the web port used by the node. * Under the Cluster Manager Properties, set the following: ** nifi.cluster.is.manager - Set this to _true_. -** nifi.cluster.protocol.manager.port - Set this to an open port that is higher than 1024 (anything lower requires root). Take note of this setting, as you will need to reference it when you set up the nodes. +** nifi.cluster.manager.protocol.port - Set this to an open port that is higher than 1024 (anything lower requires root). Take note of this setting, as you will need to reference it when you set up the nodes. For Node 1, the minimum properties to configure are as follows: @@ -344,7 +344,7 @@ For Node 1, the minimum properties to configure are as follows: * Under Cluster Node Properties, set the following: ** nifi.cluster.is.node - Set this to _true_. ** nifi.cluster.node.address - Set this to the fully qualified hostname of the node. If left blank, it defaults to "localhost". -** nifi.cluster.node.protocol.port - Set this to an open port that is higher than 1024 (anything lower requires root). If Node 1 and the NCM are on the same server, make sure this port is different from the nifi.cluster.protocol.manager.port. +** nifi.cluster.node.protocol.port - Set this to an open port that is higher than 1024 (anything lower requires root). If Node 1 and the NCM are on the same server, make sure this port is different from the nifi.cluster.manager.protocol.port. ** nifi.cluster.node.unicast.manager.address - Set this to the NCM's fully qualified hostname. ** nifi.cluster.node.unicast.manager.protocol.port - Set this to exactly the same port that was set on the NCM for the property nifi.cluster.manager.protocol.port.
[16/41] nifi git commit: Merge remote-tracking branch 'yu/NIFI-774'
Merge remote-tracking branch 'yu/NIFI-774' Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/fad81d87 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/fad81d87 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/fad81d87 Branch: refs/heads/NIFI-810-InputRequirement Commit: fad81d872d51d797bcc681977cfcb5fbda482257 Parents: f6d3427 eb1d6b5 Author: danbress Authored: Mon Oct 12 09:55:52 2015 -0400 Committer: danbress Committed: Mon Oct 12 09:55:52 2015 -0400 -- .../processors/aws/AbstractAWSProcessor.java| 2 +- .../nifi/processors/aws/s3/DeleteS3Object.java | 98 + .../org.apache.nifi.processor.Processor | 1 + .../processors/aws/s3/TestDeleteS3Object.java | 140 +++ 4 files changed, 240 insertions(+), 1 deletion(-) --
[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection
[ https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973284#comment-14973284 ] ASF subversion and git services commented on NIFI-810: -- Commit 0636f0e731cd28299edd3a6e9db90de5045ab662 in nifi's branch refs/heads/NIFI-810-InputRequirement from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0636f0e ] NIFI-810: Merged master into branch > Create Annotation that indicates that a Processor cannot be scheduled to run > without an incoming connection > --- > > Key: NIFI-810 > URL: https://issues.apache.org/jira/browse/NIFI-810 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > > Currently, if a Processor has no incoming connections but is started, it will > continually without ever accomplishing anything. We should have an > annotation, perhaps @RequiresInput, that indicates that the Processor should > not be scheduled to run unless it has an incoming connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[37/41] nifi git commit: Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/5d90c9be Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/5d90c9be Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/5d90c9be Branch: refs/heads/NIFI-810-InputRequirement Commit: 5d90c9be07b40455c831e5a602eeeb6660cdd8c6 Parents: bd506b1 a5a5bad Author: Mark Payne Authored: Fri Oct 23 09:52:33 2015 -0400 Committer: Mark Payne Committed: Fri Oct 23 09:52:33 2015 -0400 -- .../nifi/processors/avro/ConvertAvroToJSON.java | 49 ++-- .../processors/avro/TestConvertAvroToJSON.java | 47 +-- 2 files changed, 89 insertions(+), 7 deletions(-) --
[23/41] nifi git commit: NIFI-1016: Use centralized priority attribute name in PriorityAttributePrioritizer. closes #98.
NIFI-1016: Use centralized priority attribute name in
PriorityAttributePrioritizer. closes #98.
Signed-off-by: joewitt
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/ad73a23a
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/ad73a23a
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/ad73a23a
Branch: refs/heads/NIFI-810-InputRequirement
Commit: ad73a23affe32e77a5295c829a4958da440d24cb
Parents: bd47f36
Author: Joe
Authored: Thu Oct 1 14:14:37 2015 +0200
Committer: joewitt
Committed: Sun Oct 18 19:42:46 2015 -0400
--
.../nifi-standard-bundle/nifi-standard-prioritizers/pom.xml | 4
.../apache/nifi/prioritizer/PriorityAttributePrioritizer.java | 7 +++
2 files changed, 7 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/ad73a23a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/pom.xml
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/pom.xml
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/pom.xml
index 598c0db..ede0d13 100644
--- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/pom.xml
+++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/pom.xml
@@ -29,6 +29,10 @@
org.apache.nifi
+nifi-utils
+
+
+org.apache.nifi
nifi-mock
test
http://git-wip-us.apache.org/repos/asf/nifi/blob/ad73a23a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/main/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizer.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/main/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizer.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/main/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizer.java
index 75470c8..3d27930 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/main/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizer.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/main/java/org/apache/nifi/prioritizer/PriorityAttributePrioritizer.java
@@ -20,6 +20,7 @@ import java.util.regex.Pattern;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.flowfile.FlowFilePrioritizer;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
/**
* This prioritizer checks each FlowFile for a "priority" attribute and lets
@@ -33,8 +34,6 @@ import org.apache.nifi.flowfile.FlowFilePrioritizer;
*/
public class PriorityAttributePrioritizer implements FlowFilePrioritizer {
-public static final String PRIORITY_ATTR = "priority";
-
private static final Pattern intPattern = Pattern.compile("-?\\d+");
@Override
@@ -47,8 +46,8 @@ public class PriorityAttributePrioritizer implements
FlowFilePrioritizer {
return 1;
}
-String o1Priority = o1.getAttribute(PRIORITY_ATTR);
-String o2Priority = o2.getAttribute(PRIORITY_ATTR);
+String o1Priority = o1.getAttribute(CoreAttributes.PRIORITY.key());
+String o2Priority = o2.getAttribute(CoreAttributes.PRIORITY.key());
if (o1Priority == null && o2Priority == null) {
return -1; // this is not 0 to match FirstInFirstOut
} else if (o2Priority == null) {
[36/41] nifi git commit: NIFI-972: Added additional unit test; deleted lines that were commented out
NIFI-972: Added additional unit test; deleted lines that were commented out
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/bd506b1e
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/bd506b1e
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/bd506b1e
Branch: refs/heads/NIFI-810-InputRequirement
Commit: bd506b1e10ebc2ce025e836f83cb0f77562deba4
Parents: a9e5325
Author: Mark Payne
Authored: Fri Oct 23 09:52:09 2015 -0400
Committer: Mark Payne
Committed: Fri Oct 23 09:52:09 2015 -0400
--
.../processors/standard/util/JdbcCommon.java| 8 ++--
.../standard/util/TestJdbcCommon.java | 42
.../standard/util/TestJdbcTypesDerby.java | 4 --
3 files changed, 47 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/bd506b1e/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
index 9cf9338..937dcab 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
@@ -19,6 +19,7 @@ package org.apache.nifi.processors.standard.util;
import static java.sql.Types.ARRAY;
import static java.sql.Types.BIGINT;
import static java.sql.Types.BINARY;
+import static java.sql.Types.BIT;
import static java.sql.Types.BLOB;
import static java.sql.Types.BOOLEAN;
import static java.sql.Types.CHAR;
@@ -83,7 +84,7 @@ public class JdbcCommon {
if (value == null) {
rec.put(i - 1, null);
-} else if (javaSqlType==BINARY || javaSqlType==VARBINARY
|| javaSqlType==LONGVARBINARY || javaSqlType==ARRAY || javaSqlType==BLOB ||
javaSqlType==CLOB) {
+} else if (javaSqlType == BINARY || javaSqlType ==
VARBINARY || javaSqlType == LONGVARBINARY || javaSqlType == ARRAY ||
javaSqlType == BLOB || javaSqlType == CLOB) {
// bytes requires little bit different handling
byte[] bytes = rs.getBytes(i);
ByteBuffer bb = ByteBuffer.wrap(bytes);
@@ -104,7 +105,7 @@ public class JdbcCommon {
// The different types that we support are numbers
(int, long, double, float),
// as well as boolean values and Strings. Since Avro
doesn't provide
// timestamp types, we want to convert those to
Strings. So we will cast anything other
-// than numbers or booleans to strings by using to
toString() method.
+// than numbers or booleans to strings by using the
toString() method.
rec.put(i - 1, value.toString());
}
}
@@ -137,9 +138,10 @@ public class JdbcCommon {
builder.name(meta.getColumnName(i)).type().unionOf().nullBuilder().endNull().and().stringType().endUnion().noDefault();
break;
+case BIT:
case BOOLEAN:
builder.name(meta.getColumnName(i)).type().unionOf().nullBuilder().endNull().and().booleanType().endUnion().noDefault();
- break;
+break;
case INTEGER:
case SMALLINT:
http://git-wip-us.apache.org/repos/asf/nifi/blob/bd506b1e/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
index f54d4ba..9c9532f 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/util/TestJdbcCommon.java
@@ -24,19 +24,26 @@ import java.io.ByteArrayOutputStream;
import java.io.File;
im
[22/41] nifi git commit: NIFI-922: Upgrade to ActiveMQ 5.12.0. closes #81.
NIFI-922: Upgrade to ActiveMQ 5.12.0. closes #81. Signed-off-by: joewitt Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/bd47f36c Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/bd47f36c Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/bd47f36c Branch: refs/heads/NIFI-810-InputRequirement Commit: bd47f36c06db91d497a4b33117c38f17729c Parents: 943ccfe Author: Jean-Baptiste Onofré Authored: Thu Sep 3 15:53:47 2015 +0200 Committer: joewitt Committed: Sun Oct 18 19:13:26 2015 -0400 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/nifi/blob/bd47f36c/pom.xml -- diff --git a/pom.xml b/pom.xml index 1d5a857..74dc4a9 100644 --- a/pom.xml +++ b/pom.xml @@ -512,7 +512,7 @@ org.apache.activemq activemq-client -5.11.1 +5.12.0 org.apache.lucene
[06/41] nifi git commit: Fix a type and modify an error message
Fix a type and modify an error message
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/c492a1aa
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/c492a1aa
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/c492a1aa
Branch: refs/heads/NIFI-810-InputRequirement
Commit: c492a1aaae14efae3506ae43979fb93ba6655438
Parents: 0334f04
Author: Yuu ISHIKAWA
Authored: Wed Sep 2 21:54:02 2015 +0900
Committer: Yuu ISHIKAWA
Committed: Wed Sep 2 21:54:02 2015 +0900
--
.../java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/c492a1aa/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
index 803a6ab..dffcab8 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
@@ -94,7 +94,7 @@ public class DeleteS3Object extends AbstractS3Processor {
try {
s3.getObjectMetadata(bucket, key);
} catch (final AmazonServiceException ase) {
-getLogger().error("Not found sucha a file and folder on Amazon S3
{}", new Object[]{flowFile, ase});
+getLogger().error("Not found such a S3 object for {}; routing to
not found", new Object[]{flowFile, ase});
session.transfer(flowFile, REL_NOT_FOUND);
return;
}
[02/41] nifi git commit: Ignore the test suite
Ignore the test suite
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/7e683493
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/7e683493
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/7e683493
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 7e6834937b01f035666b113a6c2735e4156ea4f3
Parents: d1dbd37
Author: Yuu ISHIKAWA
Authored: Tue Sep 1 22:36:50 2015 +0900
Committer: Yuu ISHIKAWA
Committed: Tue Sep 1 22:36:50 2015 +0900
--
.../org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/7e683493/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
index dfe6edb..cac55e5 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
@@ -25,17 +25,17 @@ import java.util.Map;
import org.junit.AfterClass;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import com.amazonaws.auth.PropertiesCredentials;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
+
import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;
-@Ignore("For local testing only - interacts with S3 so the credentials file
must be configured and all necessary buckets created")
+//@Ignore("For local testing only - interacts with S3 so the credentials file
must be configured and all necessary buckets created")
public class TestDeleteS3Object {
private static final String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
[11/41] nifi git commit: NIFI-988: Test cases for PutDistributedMapCache
NIFI-988: Test cases for PutDistributedMapCache
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/ee7d89cb
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/ee7d89cb
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/ee7d89cb
Branch: refs/heads/NIFI-810-InputRequirement
Commit: ee7d89cb01d4661cfff2c4f0d093e38758680a56
Parents: 6b1328f
Author: Joe
Authored: Wed Sep 23 14:32:37 2015 +0200
Committer: Joe
Committed: Wed Sep 23 14:32:37 2015 +0200
--
.../standard/TestPutDistributedMapCache.java| 280 +++
1 file changed, 280 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/ee7d89cb/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutDistributedMapCache.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutDistributedMapCache.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutDistributedMapCache.java
new file mode 100644
index 000..8347e7f
--- /dev/null
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutDistributedMapCache.java
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static org.junit.Assert.assertEquals;
+
+public class TestPutDistributedMapCache {
+
+private TestRunner runner;
+private MockCacheClient service;
+private PutDistributedMapCache processor;
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(PutDistributedMapCache.class);
+
+service = new MockCacheClient();
+runner.addControllerService("service", service);
+runner.enableControllerService(service);
+runner.setProperty(PutDistributedMapCache.DISTRIBUTED_CACHE_SERVICE,
"service");
+}
+
+@Test
+public void testNoCacheKey() throws InitializationException {
+
+runner.setProperty(PutDistributedMapCache.CACHE_ENTRY_IDENTIFIER,
"${caheKeyAttribute}");
+runner.enqueue(new byte[]{});
+
+runner.run();
+
+// no cache key attribute
+
runner.assertAllFlowFilesTransferred(PutDistributedMapCache.REL_FAILURE, 1);
+runner.assertTransferCount(PutDistributedMapCache.REL_FAILURE, 1);
+runner.clearTransferState();
+}
+
+@Test
+public void testSingleFlowFile() throws InitializationException,
IOException {
+runner.setProperty(PutDistributedMapCache.CACHE_ENTRY_IDENTIFIER,
"${caheKeyAttribute}");
+
+final Map props = new HashMap<>();
+props.put("caheKeyAttribute", "1");
+
+String flowFileContent = "content";
+runner.enqueue(flowFileContent.getBytes("UTF-8"), props);
+
+runner.run();
+
+
runner.assertAllFlowFilesTransferred(PutDistributedMapCache.REL_SUCCESS, 1);
+runner.assertTransferCount(PutDistributedMapCache.REL_SUCCESS, 1);
+byte[] value = service.get("1", new
PutDistributedMapCache.StringSerializer(), new
PutDistributedMapCache.
[10/41] nifi git commit: NIFI-988: PutDisributedMapCache processor implementation
NIFI-988: PutDisributedMapCache processor implementation
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/6b1328f3
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/6b1328f3
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/6b1328f3
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 6b1328f3f181a27a5856d26983ed3329ee317522
Parents: 14eaeeb
Author: Joe
Authored: Wed Sep 23 13:16:02 2015 +0200
Committer: Joe
Committed: Wed Sep 23 14:31:13 2015 +0200
--
.../standard/PutDistributedMapCache.java| 244 +++
.../org.apache.nifi.processor.Processor | 1 +
2 files changed, 245 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/6b1328f3/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDistributedMapCache.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDistributedMapCache.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDistributedMapCache.java
new file mode 100644
index 000..8e50c9f
--- /dev/null
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDistributedMapCache.java
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+import org.apache.nifi.expression.AttributeExpression.ResultType;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+
+@EventDriven
+@SupportsBatching
+@Tags({"map", "cache", "put", "distributed"})
+@CapabilityDescription("Gets the content of a FlowFile and puts it to a
distributed map cache, using a cache key " +
+"computed from FlowFile attributes. If the cache already contains the
entry and the cache update strategy is " +
+"'keep original' the entry is not replaced.'")
+@WritesAttribute(attribute = "cached", description = "All FlowFiles will have
an attribute 'cached'. The value of this " +
+"attribute is true, is the FlowFile is cached, otherwise false.")
+@SeeAlso(classNames =
{"org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService",
"org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer"})
+public class PutDistributedMapCache extends AbstractProcessor {
+
+public static final String CACHED_ATTRIBUTE_NAME = "cached";
+
+// Identifies the distributed map cache client
+public static final PropertyDescriptor DISTRIBUTED_CACHE_SERVICE = new
PropertyDescriptor.Builder()
+.name("Distributed Cache Service")
+.description
[18/41] nifi git commit: NIFI-1043 fix for erroneous whitespace in maven archetype. This closes #100. Signed off by Tony Kurc
NIFI-1043 fix for erroneous whitespace in maven archetype. This closes #100.
Signed off by Tony Kurc
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/ce7d098a
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/ce7d098a
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/ce7d098a
Branch: refs/heads/NIFI-810-InputRequirement
Commit: ce7d098a485d3a3a58f52e269f48be4969e13f9b
Parents: 49ee06b
Author: Wouter de Bie
Authored: Sat Oct 17 09:00:18 2015 -0400
Committer: Tony Kurc
Committed: Sat Oct 17 09:00:18 2015 -0400
--
.../src/main/java/MyProcessor.java | 11 ---
1 file changed, 4 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/ce7d098a/nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
--
diff --git
a/nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
b/nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
index 7b70dca..ca5d903 100644
---
a/nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
+++
b/nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
@@ -84,13 +84,10 @@ public class MyProcessor extends AbstractProcessor {
@Override
public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {
- FlowFile flowFile = session.get();
- if ( flowFile == null ) {
- return;
- }
-
+FlowFile flowFile = session.get();
+if ( flowFile == null ) {
+return;
+}
// TODO implement
-
}
-
}
[29/41] nifi git commit: NIFI-1042 Adds restlistener.remote.source.ip attribute to ListenHTTPServlet. This closes #105
NIFI-1042 Adds restlistener.remote.source.ip attribute to ListenHTTPServlet.
This closes #105
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/518670db
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/518670db
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/518670db
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 518670dbf66a20855357fc379f516f488921d0c5
Parents: b809031
Author: Andre F de Miranda
Authored: Mon Oct 19 23:43:31 2015 -0400
Committer: Tony Kurc
Committed: Mon Oct 19 23:43:44 2015 -0400
--
.../apache/nifi/processors/standard/servlets/ListenHTTPServlet.java | 1 +
1 file changed, 1 insertion(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/518670db/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java
index 79d3887..d740f93 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java
@@ -269,6 +269,7 @@ public class ListenHTTPServlet extends HttpServlet {
flowFile = session.putAllAttributes(flowFile, attributes);
session.getProvenanceReporter().receive(flowFile,
request.getRequestURL().toString(), sourceSystemFlowFileIdentifier, "Remote
DN=" + foundSubject, transferMillis);
+flowFile = session.putAttribute(flowFile,
"restlistener.remote.source.host", request.getRemoteHost());
flowFile = session.putAttribute(flowFile,
"restlistener.remote.user.dn", foundSubject);
flowFileSet.add(flowFile);
[39/41] nifi git commit: NIFI-1010: If database driver does not support getting table name for column from ResultSetMetadata, then just use a default name
NIFI-1010: If database driver does not support getting table name for column
from ResultSetMetadata, then just use a default name
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/88fc8d28
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/88fc8d28
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/88fc8d28
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 88fc8d28a0502fa7d2d6646f31cf33e131be
Parents: 22924c6
Author: Mark Payne
Authored: Sun Oct 18 22:57:04 2015 -0400
Committer: Mark Payne
Committed: Fri Oct 23 10:25:59 2015 -0400
--
.../org/apache/nifi/processors/standard/util/JdbcCommon.java | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/88fc8d28/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
index 6fc69ff..753513b 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
@@ -51,6 +51,7 @@ import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.DatumWriter;
+import org.apache.commons.lang3.StringUtils;
/**
* JDBC / SQL common functions.
@@ -95,7 +96,10 @@ public class JdbcCommon {
public static Schema createSchema(final ResultSet rs) throws SQLException {
final ResultSetMetaData meta = rs.getMetaData();
final int nrOfColumns = meta.getColumnCount();
-final String tableName = meta.getTableName(1);
+String tableName = meta.getTableName(1);
+if (StringUtils.isBlank(tableName)) {
+tableName = "NiFi_ExecuteSQL_Record";
+}
final FieldAssembler builder =
SchemaBuilder.record(tableName).namespace("any.data").fields();
[40/41] nifi git commit: Merge branch 'NIFI-1010'
Merge branch 'NIFI-1010' Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/d63cd6bd Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/d63cd6bd Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/d63cd6bd Branch: refs/heads/NIFI-810-InputRequirement Commit: d63cd6bd2f9adc3dfc4c7fe168e38081a5b13564 Parents: 0fc5d30 88fc8d2 Author: Mark Payne Authored: Fri Oct 23 14:39:16 2015 -0400 Committer: Mark Payne Committed: Fri Oct 23 14:39:16 2015 -0400 -- .../org/apache/nifi/processors/standard/util/JdbcCommon.java | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/nifi/blob/d63cd6bd/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java --
[28/41] nifi git commit: NIFI-1046 shell clean up: +variable braces for consistency. Reviewed by Tony Kurc (tk...@apache.org). This closes #106
NIFI-1046 shell clean up: +variable braces for consistency. Reviewed by Tony
Kurc (tk...@apache.org). This closes #106
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/b8090311
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/b8090311
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/b8090311
Branch: refs/heads/NIFI-810-InputRequirement
Commit: b809031195405c04ec7a950c293482225f64e8dc
Parents: 9aa716b
Author: Alex Moundalexis
Authored: Mon Oct 19 11:14:32 2015 -0700
Committer: Tony Kurc
Committed: Mon Oct 19 20:36:58 2015 -0400
--
.../src/main/resources/bin/nifi.sh | 76 ++--
1 file changed, 38 insertions(+), 38 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/b8090311/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
index 1958e34..8863982 100755
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
@@ -57,35 +57,35 @@ detectOS() {
;;
esac
# For AIX, set an environment variable
-if $aix; then
+if ${aix}; then
export LDR_CNTRL=MAXDATA=0xB000@DSA
- echo $LDR_CNTRL
+ echo ${LDR_CNTRL}
fi
}
unlimitFD() {
# Use the maximum available, or set MAX_FD != -1 to use that
-if [ "x$MAX_FD" = "x" ]; then
+if [ "x${MAX_FD}" = "x" ]; then
MAX_FD="maximum"
fi
# Increase the maximum file descriptors if we can
-if [ "$os400" = "false" ] && [ "$cygwin" = "false" ]; then
+if [ "${os400}" = "false" ] && [ "${cygwin}" = "false" ]; then
MAX_FD_LIMIT=$(ulimit -H -n)
-if [ "$MAX_FD_LIMIT" != 'unlimited' ]; then
+if [ "${MAX_FD_LIMIT}" != 'unlimited' ]; then
if [ $? -eq 0 ]; then
-if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ]; then
+if [ "${MAX_FD}" = "maximum" -o "${MAX_FD}" = "max" ]; then
# use the system max
-MAX_FD="$MAX_FD_LIMIT"
+MAX_FD="${MAX_FD_LIMIT}"
fi
-ulimit -n $MAX_FD > /dev/null
+ulimit -n ${MAX_FD} > /dev/null
# echo "ulimit -n" `ulimit -n`
if [ $? -ne 0 ]; then
-warn "Could not set maximum file descriptor limit: $MAX_FD"
+warn "Could not set maximum file descriptor limit:
${MAX_FD}"
fi
else
-warn "Could not query system maximum file descriptor limit:
$MAX_FD_LIMIT"
+warn "Could not query system maximum file descriptor limit:
${MAX_FD_LIMIT}"
fi
fi
fi
@@ -96,24 +96,24 @@ unlimitFD() {
locateJava() {
# Setup the Java Virtual Machine
if $cygwin ; then
-[ -n "$JAVA" ] && JAVA=$(cygpath --unix "$JAVA")
-[ -n "$JAVA_HOME" ] && JAVA_HOME=$(cygpath --unix "$JAVA_HOME")
+[ -n "${JAVA}" ] && JAVA=$(cygpath --unix "${JAVA}")
+[ -n "${JAVA_HOME}" ] && JAVA_HOME=$(cygpath --unix "${JAVA_HOME}")
fi
-if [ "x$JAVA" = "x" ] && [ -r /etc/gentoo-release ] ; then
+if [ "x${JAVA}" = "x" ] && [ -r /etc/gentoo-release ] ; then
JAVA_HOME=$(java-config --jre-home)
fi
-if [ "x$JAVA" = "x" ]; then
-if [ "x$JAVA_HOME" != "x" ]; then
-if [ ! -d "$JAVA_HOME" ]; then
-die "JAVA_HOME is not valid: $JAVA_HOME"
+if [ "x${JAVA}" = "x" ]; then
+if [ "x${JAVA_HOME}" != "x" ]; then
+if [ ! -d "${JAVA_HOME}" ]; then
+die "JAVA_HOME is not valid: ${JAVA_HOME}"
fi
-JAVA="$JAVA_HOME/bin/java"
+JAVA="${JAVA_HOME}/bin/java"
else
warn "JAVA_HOME not set; results may vary"
JAVA=$(type java)
-JAVA=$(expr "$JAVA" : '.* \(/.*\)$')
-if [ "x$JAVA" = "x" ]; then
+JAVA=$(expr "${JAVA}" : '.* \(/.*\)$')
+if [ "x${JAVA}" = "x" ]; then
die "java command not found"
fi
fi
@@ -138,35 +138,35 @@ install() {
SVC_NAME=$2
fi
-SVC_FILE=/etc/init.d/$SVC_NAME
-cp "$0" "$SVC_FILE"
-sed -i s:NIFI_HOME=.*:NIFI_HOME="$NIFI_HOME": "$SVC_FILE"
-sed -i s:PROGNAME=.*:PROGNAME="${SCRIPT_NAME}": "$SVC_FILE"
+
[jira] [Commented] (NIFI-1010) When using MSSqlServer Driver with ExecuteSQL always recieve Avro parse error: Empty: Name
[ https://issues.apache.org/jira/browse/NIFI-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973283#comment-14973283 ] ASF subversion and git services commented on NIFI-1010: --- Commit d63cd6bd2f9adc3dfc4c7fe168e38081a5b13564 in nifi's branch refs/heads/NIFI-810-InputRequirement from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d63cd6b ] Merge branch 'NIFI-1010' > When using MSSqlServer Driver with ExecuteSQL always recieve Avro parse > error: Empty: Name > -- > > Key: NIFI-1010 > URL: https://issues.apache.org/jira/browse/NIFI-1010 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.3.0 > Environment: MSSqlServer >Reporter: Scott >Assignee: Mark Payne > Labels: ExecuteSQL, MSSql > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-1010-If-database-driver-does-not-support-gettin.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When using ExecuteSQL with the MSSqlServerDriver from Microsoft you always > get a SchemParseException , Empty: Name. > This is located in / nifi-nar-bundles / nifi-standard-bundle / > nifi-standard-processors / src / main / java / org / apache / nifi / > processors / standard / util / JdbcCommon.java line 98. > The line should read: final String tableName = meta.getTableName(1) == "" ? > "table" : meta.getTableName(1); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[38/41] nifi git commit: NIFI-1055: Fixed checkstyle violations
NIFI-1055: Fixed checkstyle violations
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/0fc5d304
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/0fc5d304
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/0fc5d304
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 0fc5d3046178836365f710d312cef6568126a99d
Parents: 5d90c9b
Author: Mark Payne
Authored: Fri Oct 23 09:59:24 2015 -0400
Committer: Mark Payne
Committed: Fri Oct 23 10:08:44 2015 -0400
--
.../nifi/processors/avro/ConvertAvroToJSON.java | 42
.../processors/kite/TestCSVToAvroProcessor.java | 1 -
.../nifi/processors/standard/ListenHTTP.java| 108 +--
.../standard/PutDistributedMapCache.java| 96 +
.../standard/TestPutDistributedMapCache.java| 31 +++---
5 files changed, 140 insertions(+), 138 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/0fc5d304/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
--
diff --git
a/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
b/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
index 016750b..f0ba71a 100644
---
a/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
+++
b/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
@@ -50,7 +50,7 @@ import org.apache.nifi.processor.io.StreamCallback;
@SideEffectFree
@SupportsBatching
-@Tags({ "json", "avro", "binary" })
+@Tags({"json", "avro", "binary"})
@CapabilityDescription("Converts a Binary Avro record into a JSON object. This
processor provides a direct mapping of an Avro field to a JSON field, such "
+ "that the resulting JSON will have the same hierarchical structure as
the Avro document. Note that the Avro schema information will be lost, as this "
+ "is not a translation from binary Avro to JSON formatted Avro. The
output JSON is encoded the UTF-8 encoding. If an incoming FlowFile contains a
stream of "
@@ -60,41 +60,41 @@ public class ConvertAvroToJSON extends AbstractProcessor {
protected static final String CONTAINER_ARRAY = "array";
protected static final String CONTAINER_NONE = "none";
-static final PropertyDescriptor CONTAINER_OPTIONS
-= new PropertyDescriptor.Builder()
-.name("JSON container options")
-.description("Determines how stream of records is exposed: either
as a sequence of single Objects (" + CONTAINER_NONE + ") (i.e. writing every
Object to a new line), or as an array of Objects (" + CONTAINER_ARRAY + ").")
-.allowableValues(CONTAINER_NONE, CONTAINER_ARRAY)
-.required(true)
-.defaultValue(CONTAINER_ARRAY)
-.build();
+static final PropertyDescriptor CONTAINER_OPTIONS = new
PropertyDescriptor.Builder()
+.name("JSON container options")
+.description("Determines how stream of records is exposed: either as a
sequence of single Objects (" + CONTAINER_NONE
++ ") (i.e. writing every Object to a new line), or as an array of
Objects (" + CONTAINER_ARRAY + ").")
+.allowableValues(CONTAINER_NONE, CONTAINER_ARRAY)
+.required(true)
+.defaultValue(CONTAINER_ARRAY)
+.build();
static final Relationship REL_SUCCESS = new Relationship.Builder()
-.name("success")
-.description("A FlowFile is routed to this relationship after it
has been converted to JSON")
-.build();
+.name("success")
+.description("A FlowFile is routed to this relationship after it has
been converted to JSON")
+.build();
static final Relationship REL_FAILURE = new Relationship.Builder()
-.name("failure")
-.description("A FlowFile is routed to this relationship if it
cannot be parsed as Avro or cannot be converted to JSON for any reason")
-.build();
+.name("failure")
+.description("A FlowFile is routed to this relationship if it cannot
be parsed as Avro or cannot be converted to JSON for any reason")
+.build();
-
private List properties;
-
+
@Override
protected void init(ProcessorInitializationContext context) {
super.init(context);
-
+
final List properties = new ArrayList<>();
properties.add(CONTAINER_OPTIONS);
this.properties = Collections.unmodifiableList(properties);
-
[14/41] nifi git commit: NIFI-1031 resolve reporting task error in flow.xml
NIFI-1031 resolve reporting task error in flow.xml Signed-off-by: Aldrin Piri Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/f7981272 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/f7981272 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/f7981272 Branch: refs/heads/NIFI-810-InputRequirement Commit: f798127290c5a0bf7fba9ea0aae8cbb733e67f40 Parents: b4bfcc1 Author: Mike Moser Authored: Thu Oct 8 20:50:28 2015 -0400 Committer: Aldrin Piri Committed: Fri Oct 9 10:53:23 2015 -0400 -- .../nifi-framework-core/src/main/resources/FlowConfiguration.xsd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/nifi/blob/f7981272/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd -- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd index 1809554..56f08a6 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd @@ -364,7 +364,7 @@ - +
[30/41] nifi git commit: NIFI-944 Added support and unit tests for escaped characters in ConvertCSVtoAvro processor properties. This closes #87. Reviewed by Tony Kurc
NIFI-944 Added support and unit tests for escaped characters in
ConvertCSVtoAvro processor properties. This closes #87. Reviewed by Tony Kurc
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/e68fdca5
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/e68fdca5
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/e68fdca5
Branch: refs/heads/NIFI-810-InputRequirement
Commit: e68fdca517eac53700c0f38bbaa7a893cfc28d9c
Parents: 518670d
Author: Joe
Authored: Tue Oct 20 23:48:54 2015 -0400
Committer: Tony Kurc
Committed: Wed Oct 21 00:01:38 2015 -0400
--
.../nifi/processors/kite/ConvertCSVToAvro.java | 17 -
.../processors/kite/TestCSVToAvroProcessor.java | 40
2 files changed, 55 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/e68fdca5/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/ConvertCSVToAvro.java
--
diff --git
a/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/ConvertCSVToAvro.java
b/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/ConvertCSVToAvro.java
index 6c20a8f..ea84daa 100644
---
a/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/ConvertCSVToAvro.java
+++
b/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/ConvertCSVToAvro.java
@@ -30,6 +30,7 @@ import org.apache.avro.Schema;
import org.apache.avro.file.CodecFactory;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData.Record;
+import org.apache.commons.lang3.StringEscapeUtils;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.lifecycle.OnScheduled;
@@ -53,6 +54,7 @@ import org.kitesdk.data.spi.DefaultConfiguration;
import org.kitesdk.data.spi.filesystem.CSVFileReader;
import org.kitesdk.data.spi.filesystem.CSVProperties;
+
import static
org.apache.nifi.processor.util.StandardValidators.createLongValidator;
@Tags({"kite", "csv", "avro"})
@@ -66,11 +68,15 @@ public class ConvertCSVToAvro extends AbstractKiteProcessor
{
@Override
public ValidationResult validate(String subject, String input,
ValidationContext context) {
+// Allows special, escaped characters as input, which is then
unescaped and converted to a single character.
+// Examples for special characters: \t (or \u0009), \f.
+input = unescapeString(input);
+
return new ValidationResult.Builder()
.subject(subject)
.input(input)
-.explanation("Only single characters are supported")
-.valid(input.length() == 1)
+.explanation("Only non-null single characters are
supported")
+.valid(input.length() == 1 && input.charAt(0) != 0)
.build();
}
};
@@ -295,4 +301,11 @@ public class ConvertCSVToAvro extends
AbstractKiteProcessor {
session.transfer(incomingCSV, FAILURE);
}
}
+
+private static String unescapeString(String input) {
+if (input.length() > 1) {
+input = StringEscapeUtils.unescapeJava(input);
+}
+return input;
+}
}
http://git-wip-us.apache.org/repos/asf/nifi/blob/e68fdca5/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
--
diff --git
a/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
b/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
index 43dea6e..0cde23c 100644
---
a/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
+++
b/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
@@ -48,9 +48,49 @@ public class TestCSVToAvroProcessor {
public static final String FAILURE_CONTENT = ""
+ ",blue,\n"; // invalid, ID is missing
+public static final String TSV_CONTENT = ""
++ "1\tgreen\n"
++ "\tblue\t\n" + // invalid, ID is missing
+"2\tgrey\t12.95";
+
public static final String FAILURE_SUMMARY = "" +
[31/41] nifi git commit: Merge branch 'NIFI-988' of https://github.com/ImpressTV/nifi into NIFI-988
Merge branch 'NIFI-988' of https://github.com/ImpressTV/nifi into NIFI-988 Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/97441ea0 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/97441ea0 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/97441ea0 Branch: refs/heads/NIFI-810-InputRequirement Commit: 97441ea0c23ebdc7c51939ff28ba270012c0fe55 Parents: 22924c6 ee7d89c Author: Mark Payne Authored: Wed Oct 21 10:18:05 2015 -0400 Committer: Mark Payne Committed: Wed Oct 21 10:18:05 2015 -0400 -- .../standard/PutDistributedMapCache.java| 244 .../org.apache.nifi.processor.Processor | 1 + .../standard/TestPutDistributedMapCache.java| 280 +++ 3 files changed, 525 insertions(+) --
[09/41] nifi git commit: Remove `REL_NOT_FOUND`
Remove `REL_NOT_FOUND`
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/eb1d6b55
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/eb1d6b55
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/eb1d6b55
Branch: refs/heads/NIFI-810-InputRequirement
Commit: eb1d6b554cc589d7a62b4bf452bf4ebfbad1f8ff
Parents: f718b4b
Author: Yu ISHIKAWA
Authored: Fri Sep 11 18:05:20 2015 +0900
Committer: Yu ISHIKAWA
Committed: Fri Sep 11 18:05:20 2015 +0900
--
.../org/apache/nifi/processors/aws/s3/DeleteS3Object.java | 10 +-
1 file changed, 1 insertion(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/eb1d6b55/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
index c8950c3..836e0d8 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
@@ -18,15 +18,14 @@ package org.apache.nifi.processors.aws.s3;
import java.util.Arrays;
import java.util.Collections;
-import java.util.HashSet;
import java.util.List;
-import java.util.Set;
import java.util.concurrent.TimeUnit;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.DeleteObjectRequest;
import com.amazonaws.services.s3.model.DeleteVersionRequest;
+
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.SeeAlso;
@@ -35,7 +34,6 @@ import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
-import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.util.StandardValidators;
@@ -46,9 +44,6 @@ import org.apache.nifi.processor.util.StandardValidators;
"And the FlowFiles are checked if exists or not before deleting.")
public class DeleteS3Object extends AbstractS3Processor {
-public static final Relationship REL_NOT_FOUND = new
Relationship.Builder().name("not found")
-.description("FlowFiles are routed to 'not found' if it doesn't
exist on Amazon S3").build();
-
public static final PropertyDescriptor VERSION_ID = new
PropertyDescriptor.Builder()
.name("Version")
.description("The Version of the Object to delete")
@@ -61,9 +56,6 @@ public class DeleteS3Object extends AbstractS3Processor {
Arrays.asList(KEY, BUCKET, ACCESS_KEY, SECRET_KEY,
CREDENTAILS_FILE, REGION, TIMEOUT, VERSION_ID,
FULL_CONTROL_USER_LIST, READ_USER_LIST, WRITE_USER_LIST,
READ_ACL_LIST, WRITE_ACL_LIST, OWNER));
-public static final Set relationships =
Collections.unmodifiableSet(
-new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE,
REL_NOT_FOUND)));
-
@Override
protected List getSupportedPropertyDescriptors() {
return properties;
[26/41] nifi git commit: NIFI-1046 shell clean up: remove backticks from nifi.sh. Reviewed by Tony Kurc (tk...@apache.org)
NIFI-1046 shell clean up: remove backticks from nifi.sh. Reviewed by Tony Kurc
(tk...@apache.org)
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/58d9b025
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/58d9b025
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/58d9b025
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 58d9b0251734f124dae40443a295821ca2f653b3
Parents: 9200542
Author: Alex Moundalexis
Authored: Mon Oct 19 10:42:28 2015 -0700
Committer: Tony Kurc
Committed: Mon Oct 19 20:33:52 2015 -0400
--
.../src/main/resources/bin/nifi.sh | 22 ++--
1 file changed, 11 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/58d9b025/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
index 2d60d5a..f8d90c4 100755
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
@@ -21,8 +21,8 @@
# Script structure inspired from Apache Karaf and other Apache projects with
similar startup approaches
-NIFI_HOME=`cd $(dirname "$0") && cd .. && pwd`
-PROGNAME=`basename "$0"`
+NIFI_HOME=$(cd $(dirname "$0") && cd .. && pwd)
+PROGNAME=$(basename "$0")
warn() {
@@ -40,7 +40,7 @@ detectOS() {
aix=false;
os400=false;
darwin=false;
-case "`uname`" in
+case "$(uname)" in
CYGWIN*)
cygwin=true
;;
@@ -69,7 +69,7 @@ unlimitFD() {
# Increase the maximum file descriptors if we can
if [ "$os400" = "false" ] && [ "$cygwin" = "false" ]; then
-MAX_FD_LIMIT=`ulimit -H -n`
+MAX_FD_LIMIT=$(ulimit -H -n)
if [ "$MAX_FD_LIMIT" != 'unlimited' ]; then
if [ $? -eq 0 ]; then
if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ]; then
@@ -94,12 +94,12 @@ unlimitFD() {
locateJava() {
# Setup the Java Virtual Machine
if $cygwin ; then
-[ -n "$JAVA" ] && JAVA=`cygpath --unix "$JAVA"`
-[ -n "$JAVA_HOME" ] && JAVA_HOME=`cygpath --unix "$JAVA_HOME"`
+[ -n "$JAVA" ] && JAVA=$(cygpath --unix "$JAVA")
+[ -n "$JAVA_HOME" ] && JAVA_HOME=$(cygpath --unix "$JAVA_HOME")
fi
if [ "x$JAVA" = "x" ] && [ -r /etc/gentoo-release ] ; then
-JAVA_HOME=`java-config --jre-home`
+JAVA_HOME=$(java-config --jre-home)
fi
if [ "x$JAVA" = "x" ]; then
if [ "x$JAVA_HOME" != "x" ]; then
@@ -109,8 +109,8 @@ locateJava() {
JAVA="$JAVA_HOME/bin/java"
else
warn "JAVA_HOME not set; results may vary"
-JAVA=`type java`
-JAVA=`expr "$JAVA" : '.* \(/.*\)$'`
+JAVA=$(type java)
+JAVA=$(expr "$JAVA" : '.* \(/.*\)$')
if [ "x$JAVA" = "x" ]; then
die "java command not found"
fi
@@ -160,8 +160,8 @@ run() {
exit 1
fi;
-NIFI_HOME=`cygpath --path --windows "$NIFI_HOME"`
-BOOTSTRAP_CONF=`cygpath --path --windows "$BOOTSTRAP_CONF"`
+NIFI_HOME=$(cygpath --path --windows "$NIFI_HOME")
+BOOTSTRAP_CONF=$(cygpath --path --windows "$BOOTSTRAP_CONF")
else
if [ -n "$run_as" ]; then
if id -u "$run_as" >/dev/null 2>&1; then
[08/41] nifi git commit: Modify an error message
Modify an error message
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/f718b4bf
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/f718b4bf
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/f718b4bf
Branch: refs/heads/NIFI-810-InputRequirement
Commit: f718b4bf46f441fde2f02065b6d55ac294fe60ce
Parents: d32a32a
Author: Yu ISHIKAWA
Authored: Mon Sep 7 23:34:22 2015 +0900
Committer: Yu ISHIKAWA
Committed: Mon Sep 7 23:34:22 2015 +0900
--
.../java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/f718b4bf/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
index 082a80d..603b06a 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
@@ -56,7 +56,7 @@ public class TestDeleteS3Object {
CreateBucketRequest request = new CreateBucketRequest(TEST_BUCKET,
TEST_REGION);
client.createBucket(request);
} catch (final AmazonS3Exception e) {
-System.out.println(TEST_BUCKET + " already exists.");
+System.out.println("Can't create the key " + TEST_BUCKET + ":" +
e.toString());
} catch (final IOException e) {
System.out.println(CREDENTIALS_FILE + " doesn't exist.");
}
[32/41] nifi git commit: Merge branch 'NIFI-988'
Merge branch 'NIFI-988' Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/da28b81e Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/da28b81e Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/da28b81e Branch: refs/heads/NIFI-810-InputRequirement Commit: da28b81eece1277c7500f28708155239ae317e57 Parents: e68fdca 97441ea Author: Mark Payne Authored: Wed Oct 21 10:44:06 2015 -0400 Committer: Mark Payne Committed: Wed Oct 21 10:44:06 2015 -0400 -- .../standard/PutDistributedMapCache.java| 244 .../org.apache.nifi.processor.Processor | 1 + .../standard/TestPutDistributedMapCache.java| 280 +++ 3 files changed, 525 insertions(+) --
[21/41] nifi git commit: NIFI-1035 AbstractKiteProcessor.getSchema leaks file handles. closes #101. Close file handles when reading schema from resources or paths in AbstractKiteProcessor.getSchema
NIFI-1035 AbstractKiteProcessor.getSchema leaks file handles. closes #101.
Close file handles when reading schema from resources or paths in
AbstractKiteProcessor.getSchema
Signed-off-by: joewitt
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/943ccfed
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/943ccfed
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/943ccfed
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 943ccfedef0cfd8896814db203646e3e4de8cf8f
Parents: 22924c6
Author: Alan Jackoway
Authored: Mon Oct 12 15:30:20 2015 -0700
Committer: joewitt
Committed: Sun Oct 18 18:56:26 2015 -0400
--
.../nifi/processors/kite/AbstractKiteProcessor.java | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/943ccfed/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
--
diff --git
a/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
b/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
index da1c046..f90c089 100644
---
a/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
+++
b/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
@@ -121,14 +121,17 @@ abstract class AbstractKiteProcessor extends
AbstractProcessor {
if ("dataset".equals(uri.getScheme()) ||
"view".equals(uri.getScheme())) {
return
Datasets.load(uri).getDataset().getDescriptor().getSchema();
} else if ("resource".equals(uri.getScheme())) {
-InputStream in =
Resources.getResource(uri.getSchemeSpecificPart())
-.openStream();
-return parseSchema(uri, in);
+try (InputStream in =
Resources.getResource(uri.getSchemeSpecificPart())
+.openStream()) {
+return parseSchema(uri, in);
+}
} else {
// try to open the file
Path schemaPath = new Path(uri);
FileSystem fs = schemaPath.getFileSystem(conf);
-return parseSchema(uri, fs.open(schemaPath));
+try (InputStream in = fs.open(schemaPath)) {
+return parseSchema(uri, in);
+}
}
} catch (DatasetNotFoundException e) {
[41/41] nifi git commit: NIFI-810: Merged master into branch
NIFI-810: Merged master into branch
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/0636f0e7
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/0636f0e7
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/0636f0e7
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 0636f0e731cd28299edd3a6e9db90de5045ab662
Parents: 8e2308b d63cd6b
Author: Mark Payne
Authored: Sun Oct 25 11:02:40 2015 -0400
Committer: Mark Payne
Committed: Sun Oct 25 11:02:40 2015 -0400
--
.../src/main/asciidoc/administration-guide.adoc | 4 +-
.../src/main/java/MyProcessor.java | 11 +-
.../nifi/processors/avro/ConvertAvroToJSON.java | 67 -
.../processors/avro/TestConvertAvroToJSON.java | 47 ++-
.../processors/aws/AbstractAWSProcessor.java| 2 +-
.../nifi/processors/aws/s3/DeleteS3Object.java | 98 ++
.../org.apache.nifi.processor.Processor | 1 +
.../processors/aws/s3/TestDeleteS3Object.java | 141 +
.../nifi/controller/FlowUnmarshaller.java | 77 -
.../src/main/resources/FlowConfiguration.xsd| 2 +-
.../src/main/resources/bin/nifi.sh | 96 +++---
.../canvas/new-controller-service-dialog.jsp| 1 -
.../partials/canvas/new-processor-dialog.jsp| 1 -
.../canvas/new-reporting-task-dialog.jsp| 1 -
.../css/new-controller-service-dialog.css | 9 -
.../main/webapp/css/new-processor-dialog.css| 9 -
.../webapp/css/new-reporting-task-dialog.css| 9 -
.../webapp/js/nf/canvas/nf-canvas-toolbox.js| 60 ++--
.../src/main/webapp/js/nf/canvas/nf-settings.js | 140 +
.../processors/kite/AbstractKiteProcessor.java | 11 +-
.../nifi/processors/kite/ConvertCSVToAvro.java | 296 ++-
.../processors/kite/TestCSVToAvroProcessor.java | 39 +++
.../nifi-standard-prioritizers/pom.xml | 4 +
.../PriorityAttributePrioritizer.java | 7 +-
.../PriorityAttributePrioritizerTest.java | 17 +-
.../nifi-standard-processors/pom.xml| 9 +
.../nifi/processors/standard/ExecuteSQL.java| 9 +-
.../nifi/processors/standard/InvokeHTTP.java| 1 +
.../nifi/processors/standard/ListenHTTP.java| 105 ---
.../standard/PutDistributedMapCache.java| 252
.../servlets/ContentAcknowledgmentServlet.java | 3 +-
.../standard/servlets/ListenHTTPServlet.java| 8 +-
.../processors/standard/util/JdbcCommon.java| 70 -
.../org.apache.nifi.processor.Processor | 1 +
.../nifi/processors/standard/TestGetFile.java | 21 +-
.../standard/TestPutDistributedMapCache.java| 277 +
.../standard/util/TestJdbcCommon.java | 42 +++
.../standard/util/TestJdbcTypesDerby.java | 133 +
.../standard/util/TestJdbcTypesH2.java | 149 ++
pom.xml | 2 +-
40 files changed, 1725 insertions(+), 507 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/0636f0e7/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
--
diff --cc
nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
index b214427,f0ba71a..f0f1630
---
a/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
+++
b/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
@@@ -35,7 -36,8 +38,7 @@@ import org.apache.nifi.annotation.behav
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
--import org.apache.nifi.annotation.documentation.Tags;
+ import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.flowfile.attributes.CoreAttributes;
import org.apache.nifi.processor.AbstractProcessor;
@@@ -47,8 -50,7 +51,7 @@@ import org.apache.nifi.processor.io.Str
@SideEffectFree
@SupportsBatching
- @Tags({ "json", "avro", "binary" })
-@Tags({"json", "avro", "binary"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
@CapabilityDescription("Converts a Binary Avro record into a JSON object.
This processor provides a direct mapping of an Avro field to a JSON field, such
"
+ "that the resulting JSON will have the same hierarchical structure as
the Avro document. Note that the Avro schema information will be lost, as this "
+ "is not a translation from binary Avro to JSON formatted Avro. The
output J
[03/41] nifi git commit: Remove `getSupportedDynamicPropertyDescriptor`
Remove `getSupportedDynamicPropertyDescriptor`
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/6cbc6db8
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/6cbc6db8
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/6cbc6db8
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 6cbc6db860d730954ce8cadc56b6d8446fe085d8
Parents: 7e68349
Author: Yuu ISHIKAWA
Authored: Wed Sep 2 06:49:46 2015 +0900
Committer: Yuu ISHIKAWA
Committed: Wed Sep 2 06:49:46 2015 +0900
--
.../org/apache/nifi/processors/aws/s3/DeleteS3Object.java | 9 -
1 file changed, 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/6cbc6db8/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
index 3be7a15..85fc70e 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
@@ -60,15 +60,6 @@ public class DeleteS3Object extends AbstractS3Processor {
return properties;
}
-protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final
String propertyDescriptorName) {
-return new PropertyDescriptor.Builder()
-.name(propertyDescriptorName)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.expressionLanguageSupported(true)
-.dynamic(true)
-.build();
-}
-
public void onTrigger(final ProcessContext context, final ProcessSession
session) {
FlowFile flowFile = session.get();
if (flowFile == null) {
[jira] [Commented] (NIFI-1055) Build of master branch fails due to contrib-check violations
[ https://issues.apache.org/jira/browse/NIFI-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973281#comment-14973281 ] ASF subversion and git services commented on NIFI-1055: --- Commit 0fc5d3046178836365f710d312cef6568126a99d in nifi's branch refs/heads/NIFI-810-InputRequirement from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0fc5d30 ] NIFI-1055: Fixed checkstyle violations > Build of master branch fails due to contrib-check violations > > > Key: NIFI-1055 > URL: https://issues.apache.org/jira/browse/NIFI-1055 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.4.0 >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[13/41] nifi git commit: NIFI-1030 made InvokeHttp basic auth password a sensitive value
NIFI-1030 made InvokeHttp basic auth password a sensitive value
Signed-off-by: Aldrin Piri
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/b4bfcc1f
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/b4bfcc1f
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/b4bfcc1f
Branch: refs/heads/NIFI-810-InputRequirement
Commit: b4bfcc1f21fed3209bf4a8f187616cdbb3d1a5c9
Parents: 2583d78
Author: Joseph Percivall
Authored: Thu Oct 8 15:33:16 2015 -0400
Committer: Aldrin Piri
Committed: Thu Oct 8 15:46:12 2015 -0400
--
.../main/java/org/apache/nifi/processors/standard/InvokeHTTP.java | 1 +
1 file changed, 1 insertion(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/b4bfcc1f/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
index f16eb9c..bf1fb4e 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
@@ -277,6 +277,7 @@ public final class InvokeHTTP extends AbstractProcessor {
.displayName("Basic Authentication Password")
.description("The password to be used by the client to
authenticate against the Remote URL.")
.required(false)
+.sensitive(true)
.addValidator(StandardValidators.createRegexMatchingValidator(Pattern.compile("^[\\x20-\\x7e\\x80-\\xff]+$")))
.build();
[01/41] nifi git commit: [NIFI-774] Create a DeleteS3Object Processor
Repository: nifi
Updated Branches:
refs/heads/NIFI-810-InputRequirement 8e2308b78 -> 0636f0e73
[NIFI-774] Create a DeleteS3Object Processor
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/d1dbd376
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/d1dbd376
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/d1dbd376
Branch: refs/heads/NIFI-810-InputRequirement
Commit: d1dbd37629baeb9a1570f47f7583d07f54b38b5c
Parents: e4e263c
Author: Yu ISHIKAWA
Authored: Thu Jul 23 13:13:15 2015 +0900
Committer: Yuu ISHIKAWA
Committed: Tue Sep 1 22:35:11 2015 +0900
--
.../nifi/processors/aws/s3/DeleteS3Object.java | 108 +++
.../org.apache.nifi.processor.Processor | 1 +
.../processors/aws/s3/TestDeleteS3Object.java | 137 +++
3 files changed, 246 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/d1dbd376/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
new file mode 100644
index 000..3be7a15
--- /dev/null
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.DeleteObjectRequest;
+import com.amazonaws.services.s3.model.DeleteVersionRequest;
+
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@SupportsBatching
+@SeeAlso({PutS3Object.class})
+@Tags({"Amazon", "S3", "AWS", "Archive", "Delete"})
+@CapabilityDescription("Deletes FlowFiles on an Amazon S3 Bucket. " +
+"And the FlowFiles are checked if exists or not before deleting.")
+public class DeleteS3Object extends AbstractS3Processor {
+
+public static final PropertyDescriptor VERSION_ID = new
PropertyDescriptor.Builder()
+.name("Version")
+.description("The Version of the Object to delete")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.required(false)
+.build();
+
+public static final List properties =
Collections.unmodifiableList(
+Arrays.asList(KEY, BUCKET, ACCESS_KEY, SECRET_KEY,
CREDENTAILS_FILE, REGION, TIMEOUT, VERSION_ID,
+FULL_CONTROL_USER_LIST, READ_USER_LIST, WRITE_USER_LIST,
READ_ACL_LIST, WRITE_ACL_LIST, OWNER));
+
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final
String propertyDescriptorName) {
+return new PropertyDescriptor.Builder()
+.name(propertyDescriptorName)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.dynamic(true)
+.build();
+}
+
+public void onTrigger(final ProcessContext context, final ProcessSession
session
[35/41] nifi git commit: NIFI-972 ExecuteSQL bug in createSchema() create Arvo Schema 1
NIFI-972 ExecuteSQL bug in createSchema() create Arvo Schema 1
Signed-off-by: Toivo Adams
Signed-off-by: Mark Payne
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/ba3225fe
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/ba3225fe
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/ba3225fe
Branch: refs/heads/NIFI-810-InputRequirement
Commit: ba3225fe92258a6aca3cb706412ab62955914dc8
Parents: da28b81
Author: Toivo Adams
Authored: Thu Oct 1 17:22:08 2015 +0300
Committer: Mark Payne
Committed: Fri Oct 23 09:28:03 2015 -0400
--
.../nifi-standard-processors/pom.xml| 9 ++
.../processors/standard/util/JdbcCommon.java| 77 --
.../standard/util/TestJdbcTypesDerby.java | 137 +
.../standard/util/TestJdbcTypesH2.java | 149 +++
4 files changed, 357 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/ba3225fe/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml
index 2d94981..b0b3afa 100644
--- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml
+++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml
@@ -190,6 +190,15 @@ language governing permissions and limitations under the
License. -->
derby
test
+
+
+com.h2database
+h2
+1.4.187
+test
+
+
+
http://git-wip-us.apache.org/repos/asf/nifi/blob/ba3225fe/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
index 6fc69ff..de3d5d1 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java
@@ -16,15 +16,20 @@
*/
package org.apache.nifi.processors.standard.util;
+import static java.sql.Types.ARRAY;
import static java.sql.Types.BIGINT;
+import static java.sql.Types.BINARY;
+import static java.sql.Types.BLOB;
import static java.sql.Types.BOOLEAN;
import static java.sql.Types.CHAR;
+import static java.sql.Types.CLOB;
import static java.sql.Types.DATE;
import static java.sql.Types.DECIMAL;
import static java.sql.Types.DOUBLE;
import static java.sql.Types.FLOAT;
import static java.sql.Types.INTEGER;
import static java.sql.Types.LONGNVARCHAR;
+import static java.sql.Types.LONGVARBINARY;
import static java.sql.Types.LONGVARCHAR;
import static java.sql.Types.NCHAR;
import static java.sql.Types.NUMERIC;
@@ -35,10 +40,12 @@ import static java.sql.Types.SMALLINT;
import static java.sql.Types.TIME;
import static java.sql.Types.TIMESTAMP;
import static java.sql.Types.TINYINT;
+import static java.sql.Types.VARBINARY;
import static java.sql.Types.VARCHAR;
import java.io.IOException;
import java.io.OutputStream;
+import java.nio.ByteBuffer;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
@@ -70,17 +77,34 @@ public class JdbcCommon {
long nrOfRows = 0;
while (rs.next()) {
for (int i = 1; i <= nrOfColumns; i++) {
+final int javaSqlType = meta.getColumnType(i);
final Object value = rs.getObject(i);
-// The different types that we support are numbers (int,
long, double, float),
-// as well as boolean values and Strings. Since Avro
doesn't provide
-// timestamp types, we want to convert those to Strings.
So we will cast anything other
-// than numbers or booleans to strings by using to
toString() method.
if (value == null) {
rec.put(i - 1, null);
+
+} else if (javaSqlType==BINARY || javaSqlType==VARBINARY
|| javaSqlType==LONGVARBINARY || javaSqlType==ARRAY || javaSqlType==BLOB ||
javaSqlType==CLOB) {
+// bytes requires little bit different handling
+
[jira] [Commented] (NIFI-1010) When using MSSqlServer Driver with ExecuteSQL always recieve Avro parse error: Empty: Name
[ https://issues.apache.org/jira/browse/NIFI-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973282#comment-14973282 ] ASF subversion and git services commented on NIFI-1010: --- Commit 88fc8d28a0502fa7d2d6646f31cf33e131be in nifi's branch refs/heads/NIFI-810-InputRequirement from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=88fc8d2 ] NIFI-1010: If database driver does not support getting table name for column from ResultSetMetadata, then just use a default name > When using MSSqlServer Driver with ExecuteSQL always recieve Avro parse > error: Empty: Name > -- > > Key: NIFI-1010 > URL: https://issues.apache.org/jira/browse/NIFI-1010 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.3.0 > Environment: MSSqlServer >Reporter: Scott >Assignee: Mark Payne > Labels: ExecuteSQL, MSSql > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-1010-If-database-driver-does-not-support-gettin.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When using ExecuteSQL with the MSSqlServerDriver from Microsoft you always > get a SchemParseException , Empty: Name. > This is located in / nifi-nar-bundles / nifi-standard-bundle / > nifi-standard-processors / src / main / java / org / apache / nifi / > processors / standard / util / JdbcCommon.java line 98. > The line should read: final String tableName = meta.getTableName(1) == "" ? > "table" : meta.getTableName(1); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[17/41] nifi git commit: NIFI-774 Ignore integration style test
NIFI-774 Ignore integration style test
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/49ee06b0
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/49ee06b0
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/49ee06b0
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 49ee06b0a256b9b4b7550acccb6facceb3c31c21
Parents: fad81d8
Author: danbress
Authored: Mon Oct 12 19:59:46 2015 -0400
Committer: danbress
Committed: Mon Oct 12 19:59:46 2015 -0400
--
.../org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/49ee06b0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
index 603b06a..509a31f 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestDeleteS3Object.java
@@ -24,6 +24,7 @@ import java.util.HashMap;
import java.util.Map;
import org.junit.AfterClass;
import org.junit.BeforeClass;
+import org.junit.Ignore;
import org.junit.Test;
import com.amazonaws.auth.PropertiesCredentials;
@@ -38,7 +39,7 @@ import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;
-//@Ignore("For local testing only - interacts with S3 so the credentials file
must be configured and all necessary buckets created")
+@Ignore("For local testing only - interacts with S3 so the credentials file
must be configured and all necessary buckets created")
public class TestDeleteS3Object {
private static final String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
[24/41] nifi git commit: NIFI-747 This closes #104. PR from Venkatesh Sellappa was modified, then code reviewed by Joe Witt (comments in ticket)
NIFI-747 This closes #104. PR from Venkatesh Sellappa
was modified, then code reviewed by Joe Witt (comments in ticket)
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/88b1b844
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/88b1b844
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/88b1b844
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 88b1b844fba5e4dded6242bb17f2096ff4172ed3
Parents: ad73a23
Author: Tony Kurc
Authored: Sun Oct 18 20:04:24 2015 -0400
Committer: Tony Kurc
Committed: Sun Oct 18 20:04:24 2015 -0400
--
.../nifi/processors/standard/ListenHTTP.java| 28
.../servlets/ContentAcknowledgmentServlet.java | 3 +--
.../standard/servlets/ListenHTTPServlet.java| 7 +++--
3 files changed, 29 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/88b1b844/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
index c7842d9..a446eb6 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java
@@ -63,7 +63,7 @@ import org.eclipse.jetty.util.ssl.SslContextFactory;
import org.eclipse.jetty.util.thread.QueuedThreadPool;
@Tags({"ingest", "http", "https", "rest", "listen"})
-@CapabilityDescription("Starts an HTTP Server that is used to receive
FlowFiles from remote sources. The URL of the Service will be
http://{hostname}:{port}/contentListener";)
+@CapabilityDescription("Starts an HTTP Server that is used to receive
FlowFiles from remote sources. The default URI of the Service will be
http://{hostname}:{port}/contentListener";)
public class ListenHTTP extends AbstractSessionFactoryProcessor {
private Set relationships;
@@ -74,6 +74,14 @@ public class ListenHTTP extends
AbstractSessionFactoryProcessor {
.description("Relationship for successfully received FlowFiles")
.build();
+public static final PropertyDescriptor BASE_PATH = new
PropertyDescriptor.Builder()
+.name("Base Path")
+.description("Base path for incoming connections")
+.required(true)
+.defaultValue("contentListener")
+.addValidator(StandardValidators.URI_VALIDATOR)
+
.addValidator(StandardValidators.createRegexMatchingValidator(Pattern.compile("(^[^/]+.*[^/]+$|^[^/]+$|^$)")))
// no start with / or end with /
+.build();
public static final PropertyDescriptor PORT = new
PropertyDescriptor.Builder()
.name("Listening Port")
.description("The Port to listen on for incoming connections")
@@ -113,7 +121,6 @@ public class ListenHTTP extends
AbstractSessionFactoryProcessor {
.required(false)
.build();
-public static final String URI = "/contentListener";
public static final String CONTEXT_ATTRIBUTE_PROCESSOR = "processor";
public static final String CONTEXT_ATTRIBUTE_LOGGER = "logger";
public static final String CONTEXT_ATTRIBUTE_SESSION_FACTORY_HOLDER =
"sessionFactoryHolder";
@@ -122,6 +129,7 @@ public class ListenHTTP extends
AbstractSessionFactoryProcessor {
public static final String CONTEXT_ATTRIBUTE_HEADER_PATTERN =
"headerPattern";
public static final String CONTEXT_ATTRIBUTE_FLOWFILE_MAP = "flowFileMap";
public static final String CONTEXT_ATTRIBUTE_STREAM_THROTTLER =
"streamThrottler";
+public static final String CONTEXT_ATTRIBUTE_BASE_PATH = "basePath";
private volatile Server server = null;
private final ConcurrentMap flowFileMap
= new ConcurrentHashMap<>();
@@ -134,6 +142,7 @@ public class ListenHTTP extends
AbstractSessionFactoryProcessor {
this.relationships = Collections.unmodifiableSet(relationships);
final List descriptors = new ArrayList<>();
+descriptors.add(BASE_PATH);
descriptors.add(PORT);
descriptors.add(MAX_DATA_RATE);
descriptors.add(SSL_CONTEXT_SERVICE);
@@ -170,6 +179,7 @@ public class ListenHTTP extends
AbstractSessionFactoryProcessor {
}
private void createHttpServerFromService(final ProcessContext context)
throws Exception {
+final String basePath = context.getProperty(BASE_PATH).getValue()
[33/41] nifi git commit: NIFI-945 Create a new property (JSON Container) in ConvertAvroToJson, which determines how stream of records is exposed: either as a sequence of single Objects ("none"), writi
NIFI-945 Create a new property (JSON Container) in ConvertAvroToJson, which
determines how stream of records is exposed: either as a sequence of single
Objects ("none"),
writing every Object to a new line, or as an array of Objects.
Let's assume you have an Avro content as stream of records (record1, record2,
...). If JSON container is "none", the converter will expose the records as
sequence of
single JSON objects:
record1
record2
...
recordN
Please bear in mind, that the final output is not a valid JSON content. You can
then forward this content e.g. to Kafka, where every record will be a single
Kafka message.
If JSON container is "array", the output looks like this:
[record1,record2,...,recordN]
It is useful when you want to convert your Avro content to a valid JSON array.
This closes #88
Reviewed and Amended (amendments reviewed by original patch author on github)
by Tony Kurc (tk...@apache.org)
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/a5a5badb
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/a5a5badb
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/a5a5badb
Branch: refs/heads/NIFI-810-InputRequirement
Commit: a5a5badb88311cb29cbe39088b57b9686314a1c6
Parents: da28b81
Author: Joe
Authored: Wed Oct 21 20:11:06 2015 -0400
Committer: Tony Kurc
Committed: Wed Oct 21 20:15:29 2015 -0400
--
.../nifi/processors/avro/ConvertAvroToJSON.java | 49 ++--
.../processors/avro/TestConvertAvroToJSON.java | 47 +--
2 files changed, 89 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/a5a5badb/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
--
diff --git
a/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
b/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
index 8832a73..016750b 100644
---
a/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
+++
b/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/src/main/java/org/apache/nifi/processors/avro/ConvertAvroToJSON.java
@@ -22,7 +22,10 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashSet;
+import java.util.List;
import java.util.Set;
import org.apache.avro.file.DataFileStream;
@@ -34,11 +37,13 @@ import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.behavior.WritesAttribute;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.flowfile.attributes.CoreAttributes;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.exception.ProcessException;
import org.apache.nifi.processor.io.StreamCallback;
@@ -49,9 +54,20 @@ import org.apache.nifi.processor.io.StreamCallback;
@CapabilityDescription("Converts a Binary Avro record into a JSON object. This
processor provides a direct mapping of an Avro field to a JSON field, such "
+ "that the resulting JSON will have the same hierarchical structure as
the Avro document. Note that the Avro schema information will be lost, as this "
+ "is not a translation from binary Avro to JSON formatted Avro. The
output JSON is encoded the UTF-8 encoding. If an incoming FlowFile contains a
stream of "
-+ "multiple Avro records, the resultant FlowFile will contain a JSON Array
containing all of the Avro records.")
++ "multiple Avro records, the resultant FlowFile will contain a JSON Array
containing all of the Avro records or a sequence of JSON Objects")
@WritesAttribute(attribute = "mime.type", description = "Sets the mime type to
application/json")
public class ConvertAvroToJSON extends AbstractProcessor {
+protected static final String CONTAINER_ARRAY = "array";
+protected static final String CONTAINER_NONE = "none";
+
+static final PropertyDescriptor CONTAINER_OPTIONS
+= new PropertyDescriptor.Builder()
+.name("JSON container options")
+.description("Determines
[27/41] nifi git commit: NIFI-1046 shell clean up: quoting prevents globbing & word splitting.
NIFI-1046 shell clean up: quoting prevents globbing & word splitting.
Ignoring the unquoted array expansions on L188/190; suspect they ought to be
quoted per [SC2068](https://github.com/koalaman/shellcheck/wiki/SC2068), but
need to confirm desired arguments.
Reviewed by Tony Kurc (tk...@apache.org)
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/9aa716b0
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/9aa716b0
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/9aa716b0
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 9aa716b0592bfd3552880b4d0042db499307f778
Parents: 58d9b02
Author: Alex Moundalexis
Authored: Mon Oct 19 11:02:39 2015 -0700
Committer: Tony Kurc
Committed: Mon Oct 19 20:36:07 2015 -0400
--
.../src/main/resources/bin/nifi.sh | 26 +++-
1 file changed, 14 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/9aa716b0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
index f8d90c4..1958e34 100755
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
@@ -21,7 +21,9 @@
# Script structure inspired from Apache Karaf and other Apache projects with
similar startup approaches
-NIFI_HOME=$(cd $(dirname "$0") && cd .. && pwd)
+SCRIPT_DIR=$(dirname "$0")
+SCRIPT_NAME=$(basename "$0")
+NIFI_HOME=$(cd "${SCRIPT_DIR}" && cd .. && pwd)
PROGNAME=$(basename "$0")
@@ -137,21 +139,21 @@ install() {
fi
SVC_FILE=/etc/init.d/$SVC_NAME
-cp $0 $SVC_FILE
-sed -i s:NIFI_HOME=.*:NIFI_HOME="$NIFI_HOME": $SVC_FILE
-sed -i s:PROGNAME=.*:PROGNAME=$(basename "$0"): $SVC_FILE
-rm -f /etc/rc2.d/S65${SVC_NAME}
-ln -s /etc/init.d/$SVC_NAME /etc/rc2.d/S65${SVC_NAME}
-rm -f /etc/rc2.d/K65${SVC_NAME}
-ln -s /etc/init.d/$SVC_NAME /etc/rc2.d/K65${SVC_NAME}
-echo Service $SVC_NAME installed
+cp "$0" "$SVC_FILE"
+sed -i s:NIFI_HOME=.*:NIFI_HOME="$NIFI_HOME": "$SVC_FILE"
+sed -i s:PROGNAME=.*:PROGNAME="${SCRIPT_NAME}": "$SVC_FILE"
+rm -f "/etc/rc2.d/S65${SVC_NAME}"
+ln -s "/etc/init.d/$SVC_NAME" "/etc/rc2.d/S65${SVC_NAME}"
+rm -f "/etc/rc2.d/K65${SVC_NAME}"
+ln -s "/etc/init.d/$SVC_NAME" "/etc/rc2.d/K65${SVC_NAME}"
+echo "Service $SVC_NAME installed"
}
run() {
BOOTSTRAP_CONF="$NIFI_HOME/conf/bootstrap.conf";
-run_as=$(grep run.as ${BOOTSTRAP_CONF} | cut -d'=' -f2)
+run_as=$(grep run.as "${BOOTSTRAP_CONF}" | cut -d'=' -f2)
sudo_cmd_prefix=""
if $cygwin; then
@@ -183,9 +185,9 @@ run() {
# run 'start' in the background because the process will continue to run,
monitoring NiFi.
# all other commands will terminate quickly so want to just wait for them
if [ "$1" = "start" ]; then
-(cd $NIFI_HOME && ${sudo_cmd_prefix} "$JAVA" -cp
"$NIFI_HOME"/conf/:"$NIFI_HOME"/lib/bootstrap/* -Xms12m -Xmx24m
-Dorg.apache.nifi.bootstrap.config.file="$BOOTSTRAP_CONF"
org.apache.nifi.bootstrap.RunNiFi $@ &)
+(cd "$NIFI_HOME" && ${sudo_cmd_prefix} "$JAVA" -cp
"$NIFI_HOME"/conf/:"$NIFI_HOME"/lib/bootstrap/* -Xms12m -Xmx24m
-Dorg.apache.nifi.bootstrap.config.file="$BOOTSTRAP_CONF"
org.apache.nifi.bootstrap.RunNiFi $@ &)
else
-(cd $NIFI_HOME && ${sudo_cmd_prefix} "$JAVA" -cp
"$NIFI_HOME"/conf/:"$NIFI_HOME"/lib/bootstrap/* -Xms12m -Xmx24m
-Dorg.apache.nifi.bootstrap.config.file="$BOOTSTRAP_CONF"
org.apache.nifi.bootstrap.RunNiFi $@)
+(cd "$NIFI_HOME" && ${sudo_cmd_prefix} "$JAVA" -cp
"$NIFI_HOME"/conf/:"$NIFI_HOME"/lib/bootstrap/* -Xms12m -Xmx24m
-Dorg.apache.nifi.bootstrap.config.file="$BOOTSTRAP_CONF"
org.apache.nifi.bootstrap.RunNiFi $@)
fi
# Wait just a bit (3 secs) to wait for the logging to finish and then echo
a new-line.
[05/41] nifi git commit: Add a relation for "not found" and that is transfered if a target key doesn't exist on S3
Add a relation for "not found" and that is transfered if a target key doesn't
exist on S3
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/0334f046
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/0334f046
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/0334f046
Branch: refs/heads/NIFI-810-InputRequirement
Commit: 0334f04640c7ae0b070214df7e356b2b05a8051c
Parents: 213f507
Author: Yu ISHIKAWA
Authored: Wed Sep 2 13:03:05 2015 +0900
Committer: Yu ISHIKAWA
Committed: Wed Sep 2 13:48:39 2015 +0900
--
.../processors/aws/AbstractAWSProcessor.java| 2 +-
.../nifi/processors/aws/s3/DeleteS3Object.java | 30
.../processors/aws/s3/TestDeleteS3Object.java | 19 +++--
3 files changed, 42 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/0334f046/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
index a781ff9..e2ae31e 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
@@ -54,7 +54,7 @@ public abstract class AbstractAWSProcessor relationships =
Collections.unmodifiableSet(
+public static Set relationships =
Collections.unmodifiableSet(
new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE)));
public static final PropertyDescriptor CREDENTAILS_FILE = new
PropertyDescriptor.Builder()
http://git-wip-us.apache.org/repos/asf/nifi/blob/0334f046/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
--
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
index 2cc00db..803a6ab 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/DeleteS3Object.java
@@ -18,14 +18,15 @@ package org.apache.nifi.processors.aws.s3;
import java.util.Arrays;
import java.util.Collections;
+import java.util.HashSet;
import java.util.List;
+import java.util.Set;
import java.util.concurrent.TimeUnit;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.DeleteObjectRequest;
import com.amazonaws.services.s3.model.DeleteVersionRequest;
-
import org.apache.nifi.annotation.behavior.SupportsBatching;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.SeeAlso;
@@ -34,6 +35,7 @@ import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.util.StandardValidators;
@@ -44,6 +46,9 @@ import org.apache.nifi.processor.util.StandardValidators;
"And the FlowFiles are checked if exists or not before deleting.")
public class DeleteS3Object extends AbstractS3Processor {
+public static final Relationship REL_NOT_FOUND = new
Relationship.Builder().name("not found")
+.description("FlowFiles are routed to 'not found' if it doesn't
exist on Amazon S3").build();
+
public static final PropertyDescriptor VERSION_ID = new
PropertyDescriptor.Builder()
.name("Version")
.description("The Version of the Object to delete")
@@ -56,6 +61,14 @@ public class DeleteS3Object extends AbstractS3Processor {
Arrays.asList(KEY, BUCKET, ACCESS_KEY, SECRET_KEY,
CREDENTAILS_FILE, REGION, TIMEOUT, VERSION_ID,
FULL_CONTROL_USER_LIST, READ_USER_LIST, WRITE_USER_LIST,
READ_ACL_LIST, WRITE_ACL_LIST, OWNER));
+public static final Set relationships =
Collections.unmodifiableSet(
+new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE,
REL_NOT_FOUND)));
+
+@Overrid
