[jira] [Created] (NIFI-997) Kerberos tickets are not being renewed by Hadoop

2015-09-25 Thread Ricky Saltzer (JIRA) 
Ricky Saltzer created NIFI-997:
--

 Summary: Kerberos tickets are not being renewed by Hadoop
 Key: NIFI-997
 URL: https://issues.apache.org/jira/browse/NIFI-997
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Ricky Saltzer
Assignee: Ricky Saltzer


I've discovered after some time of having kerberos enabled processors, that the 
kerberos ticket is not being renewed as it should. This is strange because 
according to HADOOP-6656, this should be automatically taken care of with a 
utility thread. I examined the NiFi jstack and saw that the renewal thread was 
present, so I'm not sure what's going on.

Does NiFi do something with the processor threads that cause child threads to 
suspend? I have a patch that I'm currently testing (currently looking good), 
that will renew the kerberos ticket on getFileSystem() if a threshold is 
reached (e.g. 4 hours). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/5] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree

2015-09-25 Thread markap14 
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
--
diff --git 
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
 
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
index cbd0f88..0c39eda 100644
--- 
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
+++ 
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
@@ -16,14 +16,6 @@
  */
 package org.apache.nifi.controller;
 
-import org.apache.nifi.annotation.behavior.EventDriven;
-import org.apache.nifi.annotation.behavior.SideEffectFree;
-import org.apache.nifi.annotation.behavior.SupportsBatching;
-import org.apache.nifi.annotation.behavior.TriggerSerially;
-import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
-import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
-import org.apache.nifi.annotation.documentation.CapabilityDescription;
-
 import static java.util.Objects.requireNonNull;
 
 import java.util.ArrayList;
@@ -43,6 +35,17 @@ import java.util.concurrent.atomic.AtomicReference;
 import java.util.concurrent.locks.Lock;
 import java.util.concurrent.locks.ReentrantReadWriteLock;
 
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.components.ValidationContext;
 import org.apache.nifi.components.ValidationResult;
 import org.apache.nifi.connectable.Connectable;
@@ -61,8 +64,6 @@ import org.apache.nifi.processor.Processor;
 import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.scheduling.SchedulingStrategy;
 import org.apache.nifi.util.FormatUtils;
-import org.apache.commons.lang3.builder.EqualsBuilder;
-import org.apache.commons.lang3.builder.HashCodeBuilder;
 import org.quartz.CronExpression;
 import org.slf4j.LoggerFactory;
 
@@ -73,1185 +74,1242 @@ import org.slf4j.LoggerFactory;
  */
 public class StandardProcessorNode extends ProcessorNode implements 
Connectable {
 
-public static final String BULLETIN_OBSERVER_ID = "bulletin-observer";
-
-public static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS;
-public static final String DEFAULT_YIELD_PERIOD = "1 sec";
-public static final String DEFAULT_PENALIZATION_PERIOD = "30 sec";
-private final AtomicReference processGroup;
-private final Processor processor;
-private final AtomicReference identifier;
-private final Map destinations;
-private final Map connections;
-private final AtomicReference 
undefinedRelationshipsToTerminate;
-private final AtomicReference incomingConnectionsRef;
-private final ReentrantReadWriteLock rwLock;
-private final Lock readLock;
-private final Lock writeLock;
-private final AtomicBoolean isolated;
-private final AtomicBoolean lossTolerant;
-private final AtomicReference scheduledState;
-private final AtomicReference comments;
-private final AtomicReference name;
-private final AtomicReference position;
-private final AtomicReference annotationData;
-private final AtomicReference schedulingPeriod; // stored as 
string so it's presented to user as they entered it
-private final AtomicReference yieldPeriod;
-private final AtomicReference penalizationPeriod;
-private final AtomicReference> style;
-private final AtomicInteger concurrentTaskCount;
-private final AtomicLong yieldExpiration;
-private final AtomicLong schedulingNanos;
-private final boolean triggerWhenEmpty;
-private final boolean sideEffectFree;
-private final boolean triggeredSerially;
-private final boolean triggerWhenAnyDestinationAvailable;
-private final boolean eventDrivenSupported;
-private final boolean batchSupported;
-private final ValidationContextFactory validationContextFactory;
-

[3/5] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree

2015-09-25 Thread markap14 
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
--
diff --git 
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
 
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
index cbcc54d..385ac73 100644
--- 
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
+++ 
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/CreateHadoopSequenceFile.java
@@ -23,7 +23,8 @@ import java.util.List;
 import java.util.Set;
 
 import org.apache.hadoop.io.SequenceFile;
-import org.apache.hadoop.io.SequenceFile.CompressionType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
 import org.apache.nifi.annotation.behavior.SideEffectFree;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.documentation.SeeAlso;
@@ -57,6 +58,7 @@ import 
org.apache.nifi.processors.hadoop.util.SequenceFileWriter;
  *
  */
 @SideEffectFree
+@InputRequirement(Requirement.INPUT_REQUIRED)
 @Tags({"hadoop", "sequence file", "create", "sequencefile"})
 @CapabilityDescription("Creates Hadoop Sequence Files from incoming flow 
files")
 @SeeAlso(PutHDFS.class)

http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
--
diff --git 
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
 
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
index 4a52fb7..aa03e73 100644
--- 
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
+++ 
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
@@ -29,6 +29,8 @@ import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.security.AccessControlException;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
 import org.apache.nifi.annotation.behavior.SupportsBatching;
 import org.apache.nifi.annotation.behavior.WritesAttribute;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
@@ -44,6 +46,7 @@ import org.apache.nifi.processor.util.StandardValidators;
 import org.apache.nifi.util.StopWatch;
 
 @SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
 @Tags({"hadoop", "hdfs", "get", "ingest", "fetch", "source"})
 @CapabilityDescription("Retrieves a file from HDFS. The content of the 
incoming FlowFile is replaced by the content of the file in HDFS. "
 + "The file in HDFS is left intact without any changes being made to 
it.")

http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
--
diff --git 
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
 
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
index de776d4..4c9deea 100644
--- 
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
+++ 
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFS.java
@@ -41,6 +41,8 @@ import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.PathFilter;
 import org.apache.hadoop.io.compress.CompressionCodec;
 import org.apache.hadoop.io.compress.CompressionCodecFactory;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
 import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
 import org.apache.nifi.annotation.behavior.WritesAttribute;
 import org.apache.nifi.annotation.behavior.WritesAttributes;
@@ -62,6 +64,7 @@ import org.apache.nifi.processor.util.StandardValidators;
 import org.apache.nifi.util.StopWatch;
 
 @TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_ALLOWED)

[1/5] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree

2015-09-25 Thread markap14 
Repository: nifi
Updated Branches:
  refs/heads/NIFI-810-InputRequirement [created] 034ee6de6


http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
index 68155d1..98a56bf 100644
--- 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
+++ 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
@@ -16,33 +16,7 @@
  */
 package org.apache.nifi.processors.standard;
 
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessorInitializationContext;
-import org.apache.nifi.processor.DataUnit;
-import org.apache.nifi.processor.ProcessSession;
-import org.apache.nifi.processor.Relationship;
 import java.io.BufferedWriter;
-
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.components.Validator;
-import org.apache.nifi.expression.AttributeValueDecorator;
-import org.apache.nifi.flowfile.FlowFile;
-import org.apache.nifi.stream.io.StreamUtils;
-import org.apache.nifi.logging.ProcessorLog;
-import org.apache.nifi.annotation.documentation.CapabilityDescription;
-import org.apache.nifi.annotation.behavior.EventDriven;
-import org.apache.nifi.annotation.behavior.SideEffectFree;
-import org.apache.nifi.annotation.behavior.SupportsBatching;
-import org.apache.nifi.annotation.documentation.Tags;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.processor.io.OutputStreamCallback;
-import org.apache.nifi.processor.io.StreamCallback;
-import org.apache.nifi.processor.util.FlowFileFilters;
-import org.apache.nifi.processor.util.StandardValidators;
-import org.apache.nifi.processors.standard.util.NLKBufferedReader;
-import org.apache.nifi.util.StopWatch;
-
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.InputStreamReader;
@@ -58,9 +32,37 @@ import java.util.concurrent.TimeUnit;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.expression.AttributeValueDecorator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.FlowFileFilters;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.util.NLKBufferedReader;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
 @EventDriven
 @SideEffectFree
 @SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
 @Tags({"Text", "Regular Expression", "Update", "Change", "Replace", "Modify", 
"Regex"})
 @CapabilityDescription("Updates the content of a FlowFile by evaluating a 
Regular Expression against it and replacing the section of "
 + "the content that matches the Regular Expression with some alternate 
value.")

http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java

[2/5] nifi git commit: NIFI-810: Created RequiresInput annotation and ensure that processors are invalid if connections do not agree

2015-09-25 Thread markap14 
http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
index b825972..39dc725 100644
--- 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
+++ 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicate.java
@@ -28,11 +28,13 @@ import java.util.concurrent.TimeUnit;
 
 import org.apache.commons.lang3.StringUtils;
 import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
 import org.apache.nifi.annotation.behavior.SupportsBatching;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.documentation.SeeAlso;
 import org.apache.nifi.annotation.documentation.Tags;
 import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
 import org.apache.nifi.components.PropertyDescriptor;
 import org.apache.nifi.distributed.cache.client.Deserializer;
 import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
@@ -52,6 +54,7 @@ import org.apache.nifi.processor.util.StandardValidators;
 @EventDriven
 @SupportsBatching
 @Tags({"hash", "dupe", "duplicate", "dedupe"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
 @CapabilityDescription("Caches a value, computed from FlowFile attributes, for 
each incoming FlowFile and determines if the cached value has already been 
seen. "
 + "If so, routes the FlowFile to 'duplicate' with an attribute named 
'original.identifier' that specifies the original FlowFile's"
 + "\"description\", which is specified in the  
property. If the FlowFile is not determined to be a duplicate, the Processor "

http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
index afff3c4..73ada84 100644
--- 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
+++ 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java
@@ -32,9 +32,11 @@ import java.util.concurrent.atomic.AtomicReference;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.nifi.annotation.behavior.DynamicProperty;
 import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
 import org.apache.nifi.annotation.behavior.SideEffectFree;
 import org.apache.nifi.annotation.behavior.SupportsBatching;
 import org.apache.nifi.annotation.behavior.TriggerWhenAnyDestinationAvailable;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.behavior.DynamicRelationship;
 import org.apache.nifi.annotation.documentation.Tags;
@@ -57,6 +59,7 @@ import org.apache.nifi.processor.util.StandardValidators;
 @EventDriven
 @SideEffectFree
 @SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
 @TriggerWhenAnyDestinationAvailable
 @Tags({"distribute", "load balance", "route", "round robin", "weighted"})
 @CapabilityDescription("Distributes FlowFiles to downstream processors based 
on a Distribution Strategy. If using the Round Robin "

http://git-wip-us.apache.org/repos/asf/nifi/blob/034ee6de/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java
index 7400821..021a94f 100644
---

[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection

2015-09-25 Thread Rob Moran (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908618#comment-14908618
 ] 

Rob Moran commented on NIFI-810:


{quote}
if a Processor has no incoming connections but is started, it will continually 
without ever accomplishing anything.
{quote}

In the UI I believe it would make sense to a user to flag that processor as 
invalid and display the alert icon (yellow triangle). The tooltip could state 
something like:

_A required upstream connection to this processor is not present._

I’ll make additional comments relating to NIFI-891 in that ticket.

> Create Annotation that indicates that a Processor cannot be scheduled to run 
> without an incoming connection
> ---
>
> Key: NIFI-810
> URL: https://issues.apache.org/jira/browse/NIFI-810
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Mark Payne
>
> Currently, if a Processor has no incoming connections but is started, it will 
> continually without ever accomplishing anything. We should have an 
> annotation, perhaps @RequiresInput, that indicates that the Processor should 
> not be scheduled to run unless it has an incoming connection.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-891) Provide mechanisms to handle processors that do not take input connections

2015-09-25 Thread Rob Moran (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908624#comment-14908624
 ] 

Rob Moran commented on NIFI-891:


In the UI I’d recommend not allowing the user to create the connection at all. 
Interaction can be the same as when attempting to create a connection to an 
Input Port. You see the red dotted line and do not see the green border around 
the target. 

More awareness/additional feedback could be provided to the user as well. For 
example, update the add connection icon to a version with a red slash through 
it as they drag over the target. If they attempt to “drop” the connection, 
prompt with a dialog stating why it can’t be done.

For pre-existing connections once the feature is implemented, render the 
processor invalid and explain how to address in a tooltip. Would be nice to 
expand tooltip functionality and provide follow on action directly from them 
(e.g., _Remove connection_)

> Provide mechanisms to handle processors that do not take input connections
> --
>
> Key: NIFI-891
> URL: https://issues.apache.org/jira/browse/NIFI-891
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Reporter: Aldrin Piri
>
> This is the "other half" of functionality of NIFI-810.  
> Some processors (typically those providing List, Get, Listen functionality) 
> take no input connections which can be confusing to users as their is no 
> indication and files will endlessly queue.
> Ideally, processors could be registered as invalid should they have a 
> connection where they will not consume any files.  Ideally, the UI will would 
> prevent the connection from even making these connections.  
> Processors that do not consume messages could also make use of an annotation 
> akin to @IgnoresInput 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-992) Couchbase Server Processors

2015-09-25 Thread Bryan Bende (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908526#comment-14908526
 ] 

Bryan Bende commented on NIFI-992:
--

Koji, thanks for taking the time to contribute! I will start reviewing as soon 
as I can and work with you to get this incorporated.

> Couchbase Server Processors
> ---
>
> Key: NIFI-992
> URL: https://issues.apache.org/jira/browse/NIFI-992
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Koji Kawamura
>  Labels: processor
>
> Processors providing data access interface with a Couchbase Server cluster.
> I've started writing a set of processors for interacting with Couchbase 
> Server. There are several ways to integrate with Couchbase such as:
> 1. Key/Value CRUD operations
> 2. View (Map/Reduce) queries
> 3. N1QL queries
> For the first step, I'm implementing the Key/Value CRUD operations. I will 
> send a pull request once the code and test get clean.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-999) "Tag" extensions based on their designated NiFi API version

2015-09-25 Thread Matt Gilman (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908574#comment-14908574
 ] 

Matt Gilman commented on NIFI-999:
--

Semi related to this effort is the request to support multiple versions of the 
same extension (possibly through the use of a label/tag) [1]. 

[1] 
https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension

> "Tag" extensions based on their designated NiFi API version
> ---
>
> Key: NIFI-999
> URL: https://issues.apache.org/jira/browse/NIFI-999
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Aldrin Piri
>
> Extensions need some form of tagging, whether it is on an individual 
> processor basis via annotation or at the NAR level to specify the version of 
> NiFi they were built against.
> The idea is to catch those processors at server start (or potentially, 
> additionally) at the point of assembly with a Maven plugin) that may be using 
> incompatible APIs.  
> While the importance of this will not be realized until 1.0 and the 
> introduction of breaking changes, a mechanism to ensure this is present for 
> an extension to be valid should be initialized now so that standards are in 
> place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-986) Expose audit information on flow design changes to reporting tasks

2015-09-25 Thread Bryan Bende (JIRA) 

 [ 
https://issues.apache.org/jira/browse/NIFI-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-986:
-
Attachment: NIFI-986.patch

This patch adds a new method to the EventAccess interface:

{code}
List getFlowChanges(int firstActionId, final int maxActions);
{code}

In addition, interfaces were created for the classes in nifi-user-actions and 
placed in nifi-api. 

> Expose audit information on flow design changes to reporting tasks
> --
>
> Key: NIFI-986
> URL: https://issues.apache.org/jira/browse/NIFI-986
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
> Fix For: 0.4.0
>
> Attachments: NIFI-986.patch
>
>
> It would be helpful for reporting tasks to have access to the audit 
> information about flow design changes in order to report this information to 
> external systems. 
> The first use case for this would be reporting flow design changes to Atlas 
> in order for Atlas to capture the flow design metadata.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-998) Add/Update Processor to get data into/out of Rabbit MQ

2015-09-25 Thread Joseph Witt (JIRA) 
Joseph Witt created NIFI-998:


 Summary: Add/Update Processor to get data into/out of Rabbit MQ
 Key: NIFI-998
 URL: https://issues.apache.org/jira/browse/NIFI-998
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Joseph Witt
 Fix For: 0.4.0


Quick look suggests Rabbit MQ has a bit of a unique licensing arranagement with 
GPL2, ASLv2, and MPL 1.1.  But that the release is under MPL 1.1.  Need to look 
further but in general it seems like this might be supportable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-999) "Tag" extensions based on their designated NiFi API version

2015-09-25 Thread Aldrin Piri (JIRA) 
Aldrin Piri created NIFI-999:


 Summary: "Tag" extensions based on their designated NiFi API 
version
 Key: NIFI-999
 URL: https://issues.apache.org/jira/browse/NIFI-999
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Aldrin Piri


Extensions need some form of tagging, whether it is on an individual processor 
basis via annotation or at the NAR level to specify the version of NiFi they 
were built against.

The idea is to catch those processors at server start (or potentially, 
additionally) at the point of assembly with a Maven plugin) that may be using 
incompatible APIs.  

While the importance of this will not be realized until 1.0 and the 
introduction of breaking changes, a mechanism to ensure this is present for an 
extension to be valid should be initialized now so that standards are in place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-980) Add support for HTTP Digest authentication to InvokeHttp

2015-09-25 Thread Joseph Percivall (JIRA) 

 [ 
https://issues.apache.org/jira/browse/NIFI-980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-980:
--
Attachment: NIFI-980.patch

Added Digest Authentication to InvokeHTTP and related test cases

> Add support for HTTP Digest authentication to InvokeHttp
> 
>
> Key: NIFI-980
> URL: https://issues.apache.org/jira/browse/NIFI-980
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Joseph Percivall
>Priority: Minor
> Fix For: 0.4.0
>
> Attachments: NIFI-980.patch
>
>
> Currently InvokeHttp supports Basic authentication over HTTP and HTTPs and 
> two way SSL. It would be nice to also support Digest authentication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-985) Custom log prefix for LogAttribute processor

2015-09-25 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/NIFI-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Mészáros resolved NIFI-985.
---
Resolution: Done

Closed by github PR #90

> Custom log prefix for LogAttribute processor
> 
>
> Key: NIFI-985
> URL: https://issues.apache.org/jira/browse/NIFI-985
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Joe Mészáros
>Priority: Minor
>  Labels: improvement, log
> Attachments: nifi_log_prefix_updated.png
>
>
> When you have a data flow with a bunch of LogAttribute processors it is 
> really hard to identify the right LogAttribute processor in the log file and 
> distinguish the output of different LogAttribute processors.
> There is a new property, called log prefix, which helps the user to set a 
> custom log prefix for each Log Attribute processor, which will appear in the 
> log output of the processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-989) Support size() operation in distributed map cache

2015-09-25 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/NIFI-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906178#comment-14906178
 ] 

Joe Mészáros edited comment on NIFI-989 at 9/25/15 8:27 AM:


[~markap14] From client perspective, I created an 
ExtendedDistributedMapCacheClient interface, which is the super interface of 
DistributedMapCacheClient. I choose this name, instead of 
SizableDistributedMapCacheClient, because it gives us more space. In the future 
we can add a new method, e.g. stat() to the new interface, without change the 
interface name, and do not rename it SizableDistributedMapCacheClientWithStats.


was (Author: joemeszaros):
>From client perspective, I created an ExtendedDistributedMapCacheClient 
>interface, which is the super interface of DistributedMapCacheClient. I choose 
>this name, instead of SizableDistributedMapCacheClient, because it gives us 
>more space. In the future we can add a new method, e.g. stat() to the new 
>interface, without change the interface name, and do not rename it 
>SizableDistributedMapCacheClientWithStats.

> Support size() operation in distributed map cache
> -
>
> Key: NIFI-989
> URL: https://issues.apache.org/jira/browse/NIFI-989
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Joe Mészáros
>Priority: Minor
>  Labels: cache, command-line, distributed, improvement
>
> The distributed map cache server is a great tool for caching along with the 
> cache client (DistributedMapCacheClientService), but after you configure and 
> enable it, it is a black box. You are unable to get any information from the 
> cache e.g how much entries live in the cache.
> The purpose of this issue to extend the cache interface and associated 
> implementations to support the size() operation, which returns the number of 
> entries in the distributed cache. 
> It could be a first step in a direction of a more transparent cache, where 
> the user can understand, what happens with the distributed cache. I mean 
> after the size() operation, it could be really helpful to implement a stats() 
> command, which s used to query the server about statistics it
> maintains and other internal data (e.g. evictions, hit rates, ...). 
> Similar to:
>  - memcached : 
> https://docs.oracle.com/cd/E17952_01/refman-5.0-en/ha-memcached-stats-general.html
> - couchbase: http://blog.couchbase.com/monitoring-couchbase-cluster
> - redis: http://haydenjames.io/using-redis-stat-for-redis-statistics-tracking/
> I implemented a really simple command line tool, which can interact with the 
> cache server from the command line, e.g. get a cache entry and it also able 
> to get the size of the cache, which could be useful, when you would like to 
> debug cache related problems, or just get basic interaction with the cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-604) ExecuteStreamCommand does not support arguments with semicolons

2015-09-25 Thread Mark Payne (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908017#comment-14908017
 ] 

Mark Payne commented on NIFI-604:
-

[~rickysaltzer] - looks good! Tested it and all seems to work well. Passes 
contrib-check. Pushed to master. Thanks for the knocking this out!

> ExecuteStreamCommand does not support arguments with semicolons 
> 
>
> Key: NIFI-604
> URL: https://issues.apache.org/jira/browse/NIFI-604
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Ricky Saltzer
>Assignee: Mark Payne
> Fix For: 0.4.0
>
> Attachments: NIFI-604.1.patch, NIFI-604.2.patch
>
>
> The following code in ExecuteStreamCommand assumes you're not passing 
> semicolons within your argument. This is a problem for people who need to 
> pass semicolons to the executing program as part of the argument. 
> {code}
> 224for (String arg : commandArguments.split(";")) { 
> {code}
> To allow for escaped semicolons, I propose we change this to the following 
> regex.
> {code}
> 224for (String arg : commandArguments.split("[^\\];")) { 
> {code}
> *or*... could we just change the way arguments are passed to make it more 
> similar to how ExecuteCommand works? The whole semicolon per argument took 
> some getting used to, and doesn't seem very intuitive. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-604) ExecuteStreamCommand does not support arguments with semicolons

2015-09-25 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908013#comment-14908013
 ] 

ASF subversion and git services commented on NIFI-604:
--

Commit eb94dab00cf6011cfb59e5e0b08fd840184a5b12 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=eb94dab ]

NIFI-604: Fixed checkstyle violation


> ExecuteStreamCommand does not support arguments with semicolons 
> 
>
> Key: NIFI-604
> URL: https://issues.apache.org/jira/browse/NIFI-604
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Ricky Saltzer
>Assignee: Mark Payne
> Fix For: 0.4.0
>
> Attachments: NIFI-604.1.patch, NIFI-604.2.patch
>
>
> The following code in ExecuteStreamCommand assumes you're not passing 
> semicolons within your argument. This is a problem for people who need to 
> pass semicolons to the executing program as part of the argument. 
> {code}
> 224for (String arg : commandArguments.split(";")) { 
> {code}
> To allow for escaped semicolons, I propose we change this to the following 
> regex.
> {code}
> 224for (String arg : commandArguments.split("[^\\];")) { 
> {code}
> *or*... could we just change the way arguments are passed to make it more 
> similar to how ExecuteCommand works? The whole semicolon per argument took 
> some getting used to, and doesn't seem very intuitive. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-604) ExecuteStreamCommand does not support arguments with semicolons

2015-09-25 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908011#comment-14908011
 ] 

ASF subversion and git services commented on NIFI-604:
--

Commit 193dbe65335700cf577d9090797ebf12b3105023 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=193dbe6 ]

Merge branch 'NIFI-604' of https://github.com/rickysaltzer/nifi into NIFI-604


> ExecuteStreamCommand does not support arguments with semicolons 
> 
>
> Key: NIFI-604
> URL: https://issues.apache.org/jira/browse/NIFI-604
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Ricky Saltzer
>Assignee: Mark Payne
> Fix For: 0.4.0
>
> Attachments: NIFI-604.1.patch, NIFI-604.2.patch
>
>
> The following code in ExecuteStreamCommand assumes you're not passing 
> semicolons within your argument. This is a problem for people who need to 
> pass semicolons to the executing program as part of the argument. 
> {code}
> 224for (String arg : commandArguments.split(";")) { 
> {code}
> To allow for escaped semicolons, I propose we change this to the following 
> regex.
> {code}
> 224for (String arg : commandArguments.split("[^\\];")) { 
> {code}
> *or*... could we just change the way arguments are passed to make it more 
> similar to how ExecuteCommand works? The whole semicolon per argument took 
> some getting used to, and doesn't seem very intuitive. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-988) PutDistributedMapCache processor

2015-09-25 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908041#comment-14908041
 ] 

ASF GitHub Bot commented on NIFI-988:
-

Github user joemeszaros commented on the pull request:

https://github.com/apache/nifi/pull/92#issuecomment-143226518
  
I started to implement nifi-tools, including a very simple map cache 
client. You can find the project on my github page 
[here](https://github.com/joemeszaros/nifi-tools)


> PutDistributedMapCache processor
> 
>
> Key: NIFI-988
> URL: https://issues.apache.org/jira/browse/NIFI-988
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Joe Mészáros
>Priority: Minor
>  Labels: cache, distributed, feature, new, put
>
> There is a standard controller service, called DistributedMapCacheServer, 
> which provides a distributed cache, and an associated 
> DistributedMapCacheClientService to interact with the cache. But there is not 
> any standard processor, which puts data into the cache, and helps the user to 
> leverage the distributed cache capabilities.
> The purpose of PutDistributedMapCache is very similar to the egress 
> processors: it gets the content of a FlowFile and puts it to a distributed 
> map cache, using a cache key computed from FlowFile attributes. If the cache 
> already contains the entry and the cache update strategy is 'keep original' 
> the entry is not replaced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/5] nifi git commit: NIFI-604: Removed unused import, as it caused checkstyle violation

2015-09-25 Thread markap14 
NIFI-604: Removed unused import, as it caused checkstyle violation


Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/24c08078
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/24c08078
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/24c08078

Branch: refs/heads/master
Commit: 24c080788fe92af8a66fb8f2f2392273bd8f4553
Parents: 193dbe6
Author: Mark Payne 
Authored: Thu Sep 24 10:06:39 2015 -0400
Committer: Mark Payne 
Committed: Thu Sep 24 10:06:39 2015 -0400

--
 .../standard/TestExecuteStreamCommand.java  | 435 +--
 1 file changed, 217 insertions(+), 218 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/nifi/blob/24c08078/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
index f01f404..8f945ac 100644
--- 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
+++ 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
@@ -16,14 +16,9 @@
  */
 package org.apache.nifi.processors.standard;
 
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.standard.util.ArgumentUtils;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.BeforeClass;
-import org.junit.Ignore;
-import org.junit.Test;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
 
 import java.io.File;
 import java.io.FileInputStream;
@@ -35,220 +30,224 @@ import java.util.List;
 import java.util.Set;
 import java.util.regex.Pattern;
 
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
+import org.apache.nifi.processors.standard.util.ArgumentUtils;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
 
 /**
  *
  */
 public class TestExecuteStreamCommand {
-@BeforeClass
-public static void init() {
-System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "info");
-System.setProperty("org.slf4j.simpleLogger.showDateTime", "true");
-
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.ExecuteStreamCommand",
 "debug");
-
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.TestExecuteStreamCommand",
 "debug");
-}
-
-@Test
-public void testExecuteJar() throws Exception {
-File exJar = new 
File("src/test/resources/ExecuteCommand/TestSuccess.jar");
-File dummy = new 
File("src/test/resources/ExecuteCommand/1000bytes.txt");
-String jarPath = exJar.getAbsolutePath();
-exJar.setExecutable(true);
-final TestRunner controller = 
TestRunners.newTestRunner(ExecuteStreamCommand.class);
-controller.setValidateExpressionUsage(false);
-controller.enqueue(dummy.toPath());
-controller.setProperty(ExecuteStreamCommand.EXECUTION_COMMAND, "java");
-controller.setProperty(ExecuteStreamCommand.EXECUTION_ARGUMENTS, 
"-jar;" + jarPath);
-controller.run(1);
-
controller.assertTransferCount(ExecuteStreamCommand.ORIGINAL_RELATIONSHIP, 1);
-
controller.assertTransferCount(ExecuteStreamCommand.OUTPUT_STREAM_RELATIONSHIP, 
1);
-
-List flowFiles = 
controller.getFlowFilesForRelationship(ExecuteStreamCommand.OUTPUT_STREAM_RELATIONSHIP);
-MockFlowFile outputFlowFile = flowFiles.get(0);
-byte[] byteArray = outputFlowFile.toByteArray();
-String result = new String(byteArray);
-assertTrue(Pattern.compile("Test was a 
success\r?\n").matcher(result).find());
-assertEquals("0", outputFlowFile.getAttribute("execution.status"));
-assertEquals("java", outputFlowFile.getAttribute("execution.command"));
-assertEquals("-jar;", 
outputFlowFile.getAttribute("execution.command.args").substring(0, 5));
-String attribute = 
outputFlowFile.getAttribute("execution.command.args");
-String

[1/5] nifi git commit: NIFI-604: Custom Argument Delimiters ExecuteStreamCommand / ExecuteProcess

2015-09-25 Thread markap14 
Repository: nifi
Updated Branches:
  refs/heads/master e53ab391f -> 96764ed6a


NIFI-604: Custom Argument Delimiters ExecuteStreamCommand / ExecuteProcess

- Unified the way ExecuteStreamCommand and ExecuteProcess handle arguments
- Argument delimiters can now be specified. Their default being what they were 
using before (; and space)


Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/9cefc4a5
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/9cefc4a5
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/9cefc4a5

Branch: refs/heads/master
Commit: 9cefc4a5a54adf63bf4eb67c4b8211f2a2b5dd14
Parents: d39848f
Author: ricky 
Authored: Thu Aug 27 15:02:37 2015 -0400
Committer: ricky 
Committed: Thu Aug 27 15:26:56 2015 -0400

--
 .../nifi/processor/util/StandardValidators.java | 28 +++
 .../processors/standard/ExecuteProcess.java | 62 +---
 .../standard/ExecuteStreamCommand.java  | 17 -
 .../processors/standard/util/ArgumentUtils.java | 77 
 .../processors/standard/TestExecuteProcess.java | 13 ++--
 .../standard/TestExecuteStreamCommand.java  | 22 ++
 6 files changed, 166 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/nifi/blob/9cefc4a5/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/StandardValidators.java
--
diff --git 
a/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/StandardValidators.java
 
b/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/StandardValidators.java
index 37ba7d8..438 100644
--- 
a/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/StandardValidators.java
+++ 
b/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/StandardValidators.java
@@ -602,6 +602,34 @@ public class StandardValidators {
 }
 }
 
+public static class StringLengthValidator implements Validator {
+private final int minimum;
+private final int maximum;
+
+public StringLengthValidator(int minimum, int maximum) {
+this.minimum = minimum;
+this.maximum = maximum;
+}
+
+@Override
+public ValidationResult validate(final String subject, final String 
value, final ValidationContext context) {
+if (value.length() < minimum || value.length() > maximum) {
+return new ValidationResult.Builder()
+  .subject(subject)
+  .valid(false)
+  .input(value)
+  .explanation(String.format("String length invalid [min: %d, 
max: %d]", minimum, maximum))
+  .build();
+} else {
+return new ValidationResult.Builder()
+  .valid(true)
+  .input(value)
+  .subject(subject)
+  .build();
+}
+}
+}
+
 public static class DirectoryExistsValidator implements Validator {
 
 private final boolean allowEL;

http://git-wip-us.apache.org/repos/asf/nifi/blob/9cefc4a5/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteProcess.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteProcess.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteProcess.java
index f6085e7..c8a67a0 100644
--- 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteProcess.java
+++ 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteProcess.java
@@ -48,6 +48,7 @@ import org.apache.nifi.annotation.documentation.Tags;
 import org.apache.nifi.annotation.lifecycle.OnScheduled;
 import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
 import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
 import org.apache.nifi.flowfile.FlowFile;
 import org.apache.nifi.logging.ProcessorLog;
 import org.apache.nifi.processor.AbstractProcessor;
@@ -57,6 +58,7 @@ import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.processor.exception.ProcessException;
 import org.apache.nifi.processor.io.OutputStreamCallback;
 import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.util.ArgumentUtils;

[4/5] nifi git commit: NIFI-604: Fixed checkstyle violation

2015-09-25 Thread markap14 
NIFI-604: Fixed checkstyle violation


Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/eb94dab0
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/eb94dab0
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/eb94dab0

Branch: refs/heads/master
Commit: eb94dab00cf6011cfb59e5e0b08fd840184a5b12
Parents: 24c0807
Author: Mark Payne 
Authored: Thu Sep 24 11:51:44 2015 -0400
Committer: Mark Payne 
Committed: Thu Sep 24 11:51:44 2015 -0400

--
 .../standard/TestExecuteStreamCommand.java  | 414 +--
 1 file changed, 207 insertions(+), 207 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/nifi/blob/eb94dab0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
index 8f945ac..0f13ba2 100644
--- 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
+++ 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java
@@ -42,212 +42,212 @@ import org.junit.Test;
  *
  */
 public class TestExecuteStreamCommand {
-   @BeforeClass
-   public static void init() {
-   System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", 
"info");
-   System.setProperty("org.slf4j.simpleLogger.showDateTime", 
"true");
-   
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.ExecuteStreamCommand",
 "debug");
-   
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.TestExecuteStreamCommand",
 "debug");
-   }
-
-   @Test
-   public void testExecuteJar() throws Exception {
-   File exJar = new 
File("src/test/resources/ExecuteCommand/TestSuccess.jar");
-   File dummy = new 
File("src/test/resources/ExecuteCommand/1000bytes.txt");
-   String jarPath = exJar.getAbsolutePath();
-   exJar.setExecutable(true);
-   final TestRunner controller = 
TestRunners.newTestRunner(ExecuteStreamCommand.class);
-   controller.setValidateExpressionUsage(false);
-   controller.enqueue(dummy.toPath());
-   controller.setProperty(ExecuteStreamCommand.EXECUTION_COMMAND, 
"java");
-   
controller.setProperty(ExecuteStreamCommand.EXECUTION_ARGUMENTS, "-jar;" + 
jarPath);
-   controller.run(1);
-   
controller.assertTransferCount(ExecuteStreamCommand.ORIGINAL_RELATIONSHIP, 1);
-   
controller.assertTransferCount(ExecuteStreamCommand.OUTPUT_STREAM_RELATIONSHIP, 
1);
-
-   List flowFiles = 
controller.getFlowFilesForRelationship(ExecuteStreamCommand.OUTPUT_STREAM_RELATIONSHIP);
-   MockFlowFile outputFlowFile = flowFiles.get(0);
-   byte[] byteArray = outputFlowFile.toByteArray();
-   String result = new String(byteArray);
-   assertTrue(Pattern.compile("Test was a 
success\r?\n").matcher(result).find());
-   assertEquals("0", 
outputFlowFile.getAttribute("execution.status"));
-   assertEquals("java", 
outputFlowFile.getAttribute("execution.command"));
-   assertEquals("-jar;", 
outputFlowFile.getAttribute("execution.command.args").substring(0, 5));
-   String attribute = 
outputFlowFile.getAttribute("execution.command.args");
-   String expected = "src" + File.separator + "test" + 
File.separator + "resources" + File.separator + "ExecuteCommand" + 
File.separator + "TestSuccess.jar";
-   assertEquals(expected, attribute.substring(attribute.length() - 
expected.length()));
-
-   MockFlowFile originalFlowFile = 
controller.getFlowFilesForRelationship(ExecuteStreamCommand.ORIGINAL_RELATIONSHIP).get(0);
-   assertEquals(outputFlowFile.getAttribute("execution.status"), 
originalFlowFile.getAttribute("execution.status"));
-   assertEquals(outputFlowFile.getAttribute("execution.command"), 
originalFlowFile.getAttribute("execution.command"));
-   
assertEquals(outputFlowFile.getAttribute("execution.command.args"), 
originalFlowFile.getAttribute("execution.command.args"));
-   }
-
-   @Test
-   public void testExecuteJarWithBadPath()

[2/5] nifi git commit: Merge branch 'NIFI-604' of https://github.com/rickysaltzer/nifi into NIFI-604

2015-09-25 Thread markap14 
Merge branch 'NIFI-604' of https://github.com/rickysaltzer/nifi into NIFI-604


Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/193dbe65
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/193dbe65
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/193dbe65

Branch: refs/heads/master
Commit: 193dbe65335700cf577d9090797ebf12b3105023
Parents: 26f8009 9cefc4a
Author: Mark Payne 
Authored: Wed Sep 23 16:03:46 2015 -0400
Committer: Mark Payne 
Committed: Wed Sep 23 16:03:46 2015 -0400

--
 .../nifi/processor/util/StandardValidators.java | 28 +++
 .../processors/standard/ExecuteProcess.java | 62 +---
 .../standard/ExecuteStreamCommand.java  | 17 -
 .../processors/standard/util/ArgumentUtils.java | 77 
 .../processors/standard/TestExecuteProcess.java | 13 ++--
 .../standard/TestExecuteStreamCommand.java  | 22 ++
 6 files changed, 166 insertions(+), 53 deletions(-)
--

[5/5] nifi git commit: Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/nifi

2015-09-25 Thread markap14 
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/nifi


Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/96764ed6
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/96764ed6
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/96764ed6

Branch: refs/heads/master
Commit: 96764ed6a14fadbc925e64c47f10e1b76256a389
Parents: eb94dab e53ab39
Author: Mark Payne 
Authored: Fri Sep 25 09:15:34 2015 -0400
Committer: Mark Payne 
Committed: Fri Sep 25 09:15:34 2015 -0400

--
 .../nifi/processors/standard/LogAttribute.java  |  28 -
 .../processors/attributes/UpdateAttribute.java  |  84 +++
 .../additionalDetails.html  |  25 -
 .../update/attributes/TestUpdateAttribute.java  | 104 ---
 4 files changed, 204 insertions(+), 37 deletions(-)
--

[jira] [Commented] (NIFI-604) ExecuteStreamCommand does not support arguments with semicolons

2015-09-25 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908010#comment-14908010
 ] 

ASF subversion and git services commented on NIFI-604:
--

Commit 193dbe65335700cf577d9090797ebf12b3105023 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=193dbe6 ]

Merge branch 'NIFI-604' of https://github.com/rickysaltzer/nifi into NIFI-604


> ExecuteStreamCommand does not support arguments with semicolons 
> 
>
> Key: NIFI-604
> URL: https://issues.apache.org/jira/browse/NIFI-604
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Ricky Saltzer
>Assignee: Mark Payne
> Fix For: 0.4.0
>
> Attachments: NIFI-604.1.patch, NIFI-604.2.patch
>
>
> The following code in ExecuteStreamCommand assumes you're not passing 
> semicolons within your argument. This is a problem for people who need to 
> pass semicolons to the executing program as part of the argument. 
> {code}
> 224for (String arg : commandArguments.split(";")) { 
> {code}
> To allow for escaped semicolons, I propose we change this to the following 
> regex.
> {code}
> 224for (String arg : commandArguments.split("[^\\];")) { 
> {code}
> *or*... could we just change the way arguments are passed to make it more 
> similar to how ExecuteCommand works? The whole semicolon per argument took 
> some getting used to, and doesn't seem very intuitive. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-604) ExecuteStreamCommand does not support arguments with semicolons

2015-09-25 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908012#comment-14908012
 ] 

ASF subversion and git services commented on NIFI-604:
--

Commit 24c080788fe92af8a66fb8f2f2392273bd8f4553 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=24c0807 ]

NIFI-604: Removed unused import, as it caused checkstyle violation


> ExecuteStreamCommand does not support arguments with semicolons 
> 
>
> Key: NIFI-604
> URL: https://issues.apache.org/jira/browse/NIFI-604
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Ricky Saltzer
>Assignee: Mark Payne
> Fix For: 0.4.0
>
> Attachments: NIFI-604.1.patch, NIFI-604.2.patch
>
>
> The following code in ExecuteStreamCommand assumes you're not passing 
> semicolons within your argument. This is a problem for people who need to 
> pass semicolons to the executing program as part of the argument. 
> {code}
> 224for (String arg : commandArguments.split(";")) { 
> {code}
> To allow for escaped semicolons, I propose we change this to the following 
> regex.
> {code}
> 224for (String arg : commandArguments.split("[^\\];")) { 
> {code}
> *or*... could we just change the way arguments are passed to make it more 
> similar to how ExecuteCommand works? The whole semicolon per argument took 
> some getting used to, and doesn't seem very intuitive. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-604) ExecuteStreamCommand does not support arguments with semicolons

2015-09-25 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908014#comment-14908014
 ] 

ASF GitHub Bot commented on NIFI-604:
-

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/78


> ExecuteStreamCommand does not support arguments with semicolons 
> 
>
> Key: NIFI-604
> URL: https://issues.apache.org/jira/browse/NIFI-604
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Ricky Saltzer
>Assignee: Mark Payne
> Fix For: 0.4.0
>
> Attachments: NIFI-604.1.patch, NIFI-604.2.patch
>
>
> The following code in ExecuteStreamCommand assumes you're not passing 
> semicolons within your argument. This is a problem for people who need to 
> pass semicolons to the executing program as part of the argument. 
> {code}
> 224for (String arg : commandArguments.split(";")) { 
> {code}
> To allow for escaped semicolons, I propose we change this to the following 
> regex.
> {code}
> 224for (String arg : commandArguments.split("[^\\];")) { 
> {code}
> *or*... could we just change the way arguments are passed to make it more 
> similar to how ExecuteCommand works? The whole semicolon per argument took 
> some getting used to, and doesn't seem very intuitive. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)