[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67722269
  
--- Diff: 
nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/put/AbstractPutEventProcessor.java
 ---
@@ -119,7 +119,24 @@
 .defaultValue("10 seconds")
 .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
 .build();
-
+public static final PropertyDescriptor OUTGOING_MESSAGE_DELIMITER = 
new PropertyDescriptor.Builder()
+.name("Outgoing Message Delimiter")
+.description("Specifies the delimiter to use when sending 
messages out over the same TCP stream. The delimiter is appended to each 
FlowFile message "
++ "that is transmitted over the stream so that the 
receiver can determine when one message ends and the next message begins. Users 
should "
++ "ensure that the FlowFile content does not contain 
the delimiter character to avoid errors. If it is not possible to define a 
delimiter "
++ "character that is not present in the FlowFile 
content then the user must use another processor to change the encoding of the 
data e.g. Base64 "
++ "encoding.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+public static final PropertyDescriptor CONNECTION_PER_FLOWFILE = new 
PropertyDescriptor.Builder()
--- End diff --

While I'm not sure about use-cases I don't see a problem adding it as a 
property which defaults to false. I assume @aviewfromspace1 had a specific 
use-case in mind when adding the property and since it is optional I like 
keeping it.

Also PutTCP is marked as "TriggerWhenEmpty" so it will properly 
pruneIdleSenders just like PutSplunk.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #542: NIFI-1895 for 0.x

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/542#discussion_r67723205
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java
 ---
@@ -89,6 +89,25 @@
 .defaultValue(COMPLEX_FIELD_TEXT.getValue())
 .build();
 
+protected static final String STRING_ENCODING_VALUE = "String";
+protected static final String BYTES_ENCODING_VALUE = "Bytes";
+
+protected static final AllowableValue FIELD_ENCODING_STRING = new 
AllowableValue(STRING_ENCODING_VALUE, STRING_ENCODING_VALUE,
+"Stores the value of each field as a UTF-8 String.");
--- End diff --

That works for me.

That's fair, if questions come in to the mailing list asking to clarify 
then we can address the description.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #542: NIFI-1895 for 0.x

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/542
  
Specifically looking at Bryan's commit: I looked through the code and built 
locally (since travis failed for un-related issues) and it looks good. 

+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67776824
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
  

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r6337
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutTCP.java
 ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.processors.standard.util.TestPutTCPCommon;
+
+public class TestPutTCP extends TestPutTCPCommon {
+
+public TestPutTCP() {
+super();
+ssl = false;
+}
+
+@Override
+public void configureProperties(String host, int port, String 
outgoingMessageDelimiter, boolean connectionPerFlowFile, boolean expectValid) {
+runner.setProperty(PutTCP.HOSTNAME, host);
+runner.setProperty(PutTCP.PORT, Integer.toString(port));
+if (outgoingMessageDelimiter != null) {
+runner.setProperty(PutTCP.OUTGOING_MESSAGE_DELIMITER, 
outgoingMessageDelimiter);
+}
+if (connectionPerFlowFile) {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "true");
+} else {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "false");
+}
--- End diff --

In general, I like the IF/ELSE block better but actually I think neither 
are appropriate here and it would better to just do 
`String.valueOf(connectionPerFlowFile)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67782856
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
  

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67782966
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
  

[GitHub] nifi issue #252: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/252
  
@jskora were you able to rebase and open a new PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67799164
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
  

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67800576
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
  

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67801359
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
  

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67803466
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
  

[GitHub] nifi issue #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/543
  
Thanks @olegz, will merge it in


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/543
  
Also, for the sake of transparency, we had a discussion offline debating 
the various directions to go to handle the "consistent inconsistency" and had 
the idea for a way for the user to check a box in the UI for whether to enter 
literal or interpreted text. I created a ticket for it here[1].

For now, in order to keep consistent with the other processors in this area 
that do automatically convert, the PutTCP processor will convert a "\n" entered 
in the UI to a new line character (also tabs and carriage returns).

[1] https://issues.apache.org/jira/browse/NIFI-2069


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #555: NIFI-2067 ignored intermittently failing MemoryTest

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/555
  
In cases where validateWarnWhenPercentThresholdReached fails would 
validateWarnWhenSizeThresholdReached fail too? Wondering if we should ignore 
both.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #555: NIFI-2067 ignored intermittently failing MemoryTest

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/555
  
Nope I am missing it, sorry about that. Once the travis build finishes 
successfully I am +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/548#discussion_r67942458
  
--- Diff: 
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java
 ---
@@ -265,7 +266,13 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 transferQueue(session);
 }
 
-private void transferQueue(ProcessSession session){
+// Protected visibility for unit testing
+protected BlockingQueue getMqttQueue() {
+return mqttQueue;
+}
+
+// Protected visibility for unit testing
+protected void transferQueue(ProcessSession session){
--- End diff --

Instead of making these protected I would prefer to use reflection to 
reference them. You can see Oleg and I's discussion on the issue here: 
https://github.com/apache/nifi/pull/392#discussion_r63536461


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/548#discussion_r67948120
  
--- Diff: 
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java
 ---
@@ -289,8 +296,10 @@ public void process(final OutputStream out) throws 
IOException {
 String transitUri = new 
StringBuilder(broker).append(mqttMessage.getTopic()).toString();
 session.getProvenanceReporter().receive(messageFlowfile, 
transitUri);
 session.transfer(messageFlowfile, REL_MESSAGE);
-mqttQueue.remove(mqttMessage);
 session.commit();
+if (!mqttQueue.remove(mqttMessage)) {
+logger.warn("Mqtt message " + mqttMessage + " had already 
been removed from queue, possible duplication of flow files");
--- End diff --

Thinking through this error message and how it would be useful to a user, I 
would like some kind of UUID to output here (like a message id) but there 
aren't any in MQTT. So I think it would be better to word the warning a 
different way by referencing the FlowFile that could potentially contain 
duplicate data. By having a warning message with the UUID of the FlowFile 
containing potentially duplicate data the user may be able to correlate that 
using provenance to find the other FlowFile with duplicate data.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #549: NIFI-2046 - Adding reference to @TriggerSerially in Process...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/549
  
+1

LGTM, will merge into 0.x and master, thanks @brosander 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/548#discussion_r67957067
  
--- Diff: 
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java
 ---
@@ -288,9 +289,13 @@ public void process(final OutputStream out) throws 
IOException {
 
 String transitUri = new 
StringBuilder(broker).append(mqttMessage.getTopic()).toString();
 session.getProvenanceReporter().receive(messageFlowfile, 
transitUri);
+String uuid = 
messageFlowfile.getAttribute(CoreAttributes.UUID.key());
--- End diff --

A bit of a nit pick, this is only referenced in the logger statement below 
so it should be created in there, not outside of it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/548#discussion_r67957184
  
--- Diff: 
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java
 ---
@@ -288,9 +289,13 @@ public void process(final OutputStream out) throws 
IOException {
 
 String transitUri = new 
StringBuilder(broker).append(mqttMessage.getTopic()).toString();
 session.getProvenanceReporter().receive(messageFlowfile, 
transitUri);
+String uuid = 
messageFlowfile.getAttribute(CoreAttributes.UUID.key());
 session.transfer(messageFlowfile, REL_MESSAGE);
-mqttQueue.remove(mqttMessage);
 session.commit();
+if (!mqttQueue.remove(mqttMessage)) {
--- End diff --

In order to avoid concatenating the Strings in the logger message 
needlessly, this should check if warn is enabled too: 
`... && getLogger().isWarnEnabled()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67962838
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67963044
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67963320
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67963557
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67964066
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
--- End diff --

Here it has "." making it seem like there 
is automatically a "." added but in the property descriptor it says the "." or 
"-" is not automatically added. I agree that we shouldn't lock the user into 
using ".", "-", etc. so this should be changed to reflect that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67966203
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67967967
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67968149
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67969095
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/image/ExtractImageMetadataTest.java
 ---
@@ -37,7 +37,7 @@
 private static String BMP_HEADER = "BMP Header.";
 private static String JPEG_HEADER = "JPEG.";
 private static String GIF_HEADER = "GIF Header.";
-private static String PNG_HEADER = "PNG.";
+private static String PNG_HEADER = "PNG-";
--- End diff --

Did the PNG header change when changing version of metadata extractor? If 
so, does that constitute breaking backwards compatibility?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67969885
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/media/TestExtractMediaMetadata.java
 ---
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestExtractMediaMetadata {
+
+@Test
+public void testProperties() {
+final TestRunner runner = TestRunners.newTestRunner(new 
ExtractMediaMetadata());
+ProcessContext context = runner.getProcessContext();
+Map propertyValues = 
context.getProperties();
+assertEquals(6, propertyValues.size());
+}
+
+@Test
+public void testRelationships() {
+final TestRunner runner = TestRunners.newTestRunner(new 
ExtractMediaMetadata());
+ProcessContext context = runner.getProcessContext();
+Set relationships = 
context.getAvailableRelationships();
+assertEquals(2, relationships.size());
+assertTrue(relationships.contains(ExtractMediaMetadata.SUCCESS));
+assertTrue(relationships.contains(ExtractMediaMetadata.FAILURE));
+}
+
+@Test
+public void testTextBytes() throws IOException {
+final TestRunner runner = TestRunners.newTestRunner(new 
ExtractMediaMetadata());
+runner.setProperty(ExtractMediaMetadata.MIME_TYPE_FILTER, 
"text/.*");
+runner.setProperty(ExtractMediaMetadata.METADATA_KEY_FILTER, "");
+runner.setProperty(ExtractMediaMetadata.METADATA_KEY_PREFIX, 
"txt.");
+runner.assertValid();
+
+final Map attrs = new HashMap<>();
+attrs.put("filename", "test1.txt");
+runner.enqueue("test1".getBytes(), attrs);
+runner.run();
+
+runner.assertAllFlowFilesTransferred(ExtractMediaMetadata.SUCCESS, 
1);
+runner.assertTransferCount(ExtractMediaMetadata.FAILURE, 0);
+
+final List successFiles = 
runner.getFlowFilesForRelationship(ExtractMediaMetadata.SUCCESS);
+MockFlowFile flowFile0 = successFiles.get(0);
+flowFile0.assertAttributeExists("filename");
+flowFile0.assertAttributeEquals("filename", "test1.txt");
+flowFile0.assertAttributeExists("txt.Content-Type");
+
assertTrue(flowFile0.getAttribute("txt.Content-Type").startsWith("text/plain"));
+flowFile0.assertAttributeExists("txt.X-Parsed-By");
+
assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.DefaultParser"));
+
assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.txt.TXTParser"));
+flowFile0.assertAttributeExists("txt.Content-Encoding");
+flowFile0.assertAttributeEquals("txt.Content-Encoding", 
"ISO-8859-1");
+flowFile0.assertContentEquals("test1".getBytes("UTF-8"));
+}
+
+@Test
+public void te

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67982895
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r67982943
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r68052320
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r68052582
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r68056970
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/media/TestExtractMediaMetadata.java
 ---
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestExtractMediaMetadata {
+
+@Test
+public void testProperties() {
+final TestRunner runner = TestRunners.newTestRunner(new 
ExtractMediaMetadata());
+ProcessContext context = runner.getProcessContext();
+Map propertyValues = 
context.getProperties();
+assertEquals(6, propertyValues.size());
+}
+
+@Test
+public void testRelationships() {
+final TestRunner runner = TestRunners.newTestRunner(new 
ExtractMediaMetadata());
+ProcessContext context = runner.getProcessContext();
+Set relationships = 
context.getAvailableRelationships();
+assertEquals(2, relationships.size());
+assertTrue(relationships.contains(ExtractMediaMetadata.SUCCESS));
+assertTrue(relationships.contains(ExtractMediaMetadata.FAILURE));
+}
+
+@Test
+public void testTextBytes() throws IOException {
+final TestRunner runner = TestRunners.newTestRunner(new 
ExtractMediaMetadata());
+runner.setProperty(ExtractMediaMetadata.MIME_TYPE_FILTER, 
"text/.*");
+runner.setProperty(ExtractMediaMetadata.METADATA_KEY_FILTER, "");
+runner.setProperty(ExtractMediaMetadata.METADATA_KEY_PREFIX, 
"txt.");
+runner.assertValid();
+
+final Map attrs = new HashMap<>();
+attrs.put("filename", "test1.txt");
+runner.enqueue("test1".getBytes(), attrs);
+runner.run();
+
+runner.assertAllFlowFilesTransferred(ExtractMediaMetadata.SUCCESS, 
1);
+runner.assertTransferCount(ExtractMediaMetadata.FAILURE, 0);
+
+final List successFiles = 
runner.getFlowFilesForRelationship(ExtractMediaMetadata.SUCCESS);
+MockFlowFile flowFile0 = successFiles.get(0);
+flowFile0.assertAttributeExists("filename");
+flowFile0.assertAttributeEquals("filename", "test1.txt");
+flowFile0.assertAttributeExists("txt.Content-Type");
+
assertTrue(flowFile0.getAttribute("txt.Content-Type").startsWith("text/plain"));
+flowFile0.assertAttributeExists("txt.X-Parsed-By");
+
assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.DefaultParser"));
+
assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.txt.TXTParser"));
+flowFile0.assertAttributeExists("txt.Content-Encoding");
+flowFile0.assertAttributeEquals("txt.Content-Encoding", 
"ISO-8859-1");
+flowFile0.assertContentEquals("test1".getBytes("UTF-8"));
+}
+
+@Test
+public void te

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
@jskora when do you think you'll get these changes done and another commit 
posted? We only have a couple more tickets left for 0.7.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/556#discussion_r68074121
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.";)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+   

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
I am trying to parse my downloads folder again and I came across a couple 
exceptions (will post in other comment) but as I try to figure why it happened 
I realize that since ExtractMediaMetatadata doesn't emit a provenance event for 
the failure transfer so don't get any details. It would be nice to include in 
the catch block:
` session.getProvenanceReporter().route(flowFile, FAILURE, "Failed to 
extract media metadata due to " + e);`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
@markap14 it seems odd to limit the usefulness of provenance for the reason 
to "keep data in the flow". Information is being created on "why this flowfile 
failed" but we don't track it in provenance, it seems wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
Future change, I'll add a ticket which is blocked by NIFI-104


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
The only relevant exception I saw processing the download folder was an xml 
parse error related to a underlying limitation of Tika, no need to address:

2016-06-22 11:32:19,774 ERROR [Timer-Driven Process Thread-10] 
o.a.n.p.media.ExtractMediaMetadata 
ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19] Failed to extract 
media metadata from 
StandardFlowFileRecord[uuid=ea365723-636f-4fb7-b871-f14241d686f7,claim=StandardContentClaim
 [resourceClaim=StandardResourceClaim[id=1466609517187-31, container=default, 
section=31], offset=293539, length=888],offset=0,name=prelude.jspf,size=888] 
due to org.apache.nifi.processor.exception.ProcessException: IOException thrown 
from ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19]: 
java.io.IOException: org.apache.tika.exception.TikaException: XML parse error: 
org.apache.nifi.processor.exception.ProcessException: IOException thrown from 
ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19]: 
java.io.IOException: org.apache.tika.exception.TikaException: XML parse error
2016-06-22 11:32:19,776 ERROR [Timer-Driven Process Thread-10] 
o.a.n.p.media.ExtractMediaMetadata
org.apache.nifi.processor.exception.ProcessException: IOException thrown 
from ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19]: 
java.io.IOException: org.apache.tika.exception.TikaException: XML parse error
at 
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1873)
 ~[na:na]
at 
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822)
 ~[na:na]
at 
org.apache.nifi.processors.media.ExtractMediaMetadata.onTrigger(ExtractMediaMetadata.java:211)
 ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
 [nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_74]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_74]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_74]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Caused by: java.io.IOException: org.apache.tika.exception.TikaException: 
XML parse error
at 
org.apache.nifi.processors.media.ExtractMediaMetadata$2.process(ExtractMediaMetadata.java:218)
 ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
 ~[na:na]
... 14 common frames omitted
Caused by: org.apache.tika.exception.TikaException: XML parse error
at org.apache.tika.parser.xml.XMLParser.parse(XMLParser.java:78) 
~[tika-parsers-1.8.jar:1.8]
at 
org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:281) 
~[tika-core-1.8.jar:1.8]
at 
org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:281) 
~[tika-core-1.8.jar:1.8]
at 
org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120) 
~[tika-core-1.8.jar:1.8]
at 
org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:136) 
~[tika-core-1.8.jar:1.8]
at 
org.apache.nifi.processors.media.ExtractMediaMetadata.tika_parse(ExtractMediaMetadata.java:245)
 ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.processors.media.ExtractMediaMetadata.access$000(ExtractMediaMetadata.java:74)
 ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
According to the latest comment from @bbende, yes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
@jskora what about changing it so there is a media and image bundles? 
Reading this comment[1] it's what we decided and the only way not to break 
backwards compatibility.

[1] https://github.com/apache/nifi/pull/556#discussion_r68044015


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
Ah sorry @jskora for misinterpreting it, changes look good. I am gonna do 
another round of testing and then should be good to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/556
  
+1

I reviewed the code, built locally, analyzed ~25gb of randoms files with it 
and it performed well. Looks good to me @jskora. Can you squash your commits 
and merge it into 0.x and master?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #548: NIFI-2045 - Removing mqtt message from queue after after se...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/548
  
+1

Comments addressed, builds and addresses issue. Thanks @brosander will 
merge it into 0.x and master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #568: NIFI-2092 Fixing unit tests which try to clean up di...

[JPercivall]

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi/pull/568

NIFI-2092 Fixing unit tests which try to clean up directories before …

…stopping the FlowController that has locks opened.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi NIFI-2092

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/568.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #568


commit 8ed0ddadcd3d494997f231aafb406fa62d23824e
Author: jpercivall 
Date:   2016-06-23T16:42:16Z

NIFI-2092 Fixing unit tests which try to clean up directories before 
stopping the FlowController that has locks opened.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #569: NIFI-2097 Changing RemoteProcessGroupStatusTest.java...

[JPercivall]

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi/pull/569

NIFI-2097 Changing RemoteProcessGroupStatusTest.java to use a random open 
port ...

... instead of a statically defined one

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi NIFI-2097

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/569.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #569


commit 81f303a943193a17daa03ae1fa7c5da3c984a057
Author: jpercivall 
Date:   2016-06-23T16:55:58Z

NIFI-2097 Changing RemoteProcessGroupStatusTest to run on a random open 
port instead of a statically defined one

commit b664fc648a9250dc745a7d60d47d6889d8d633cf
Author: jpercivall 
Date:   2016-06-23T17:21:56Z

NIFI-2097 Changing RemoteProcessGroupStatusTest.java to use a random open 
port instead of a statically defined one




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #570: NIFI-2098 Adding ignore annotation to testManyFilesO...

[JPercivall]

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi/pull/570

NIFI-2098 Adding ignore annotation to testManyFilesOpened unit test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi NIFI-2098

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/570.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #570


commit f029ac28e8cd8d84d2390f8dbb0b8c01a2e6a37b
Author: jpercivall 
Date:   2016-06-23T17:45:21Z

NIFI-2098 Adding ignore annotation to testManyFilesOpened unit test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #568: NIFI-2092 Fixing unit tests which try to clean up directori...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/568
  
Got it, thanks @olegz 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #568: NIFI-2092 Fixing unit tests which try to clean up di...

[JPercivall]

Github user JPercivall closed the pull request at:

https://github.com/apache/nifi/pull/568


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #569: NIFI-2097 Changing RemoteProcessGroupStatusTest.java to use...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/569
  
Thanks @olegz, closing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #569: NIFI-2097 Changing RemoteProcessGroupStatusTest.java...

[JPercivall]

Github user JPercivall closed the pull request at:

https://github.com/apache/nifi/pull/569


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #573: NIFI-2089: Ensure streams are closed before attempting to r...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/573
  
+1 

Fixes errors seen on Windows, code looks good and passes contrib-check. 
Thanks Mark


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #586: NIFI-2118 Addresssing License/Notice issues in nifi-...

[JPercivall]

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi/pull/586

NIFI-2118 Addresssing License/Notice issues in nifi-assumbly

Jython's license and notice information can be found here: 
https://hg.python.org/jython/file/tip

Jolt's license can be found here (no notice): 
https://github.com/bazaarvoice/jolt/tree/jolt-0.0.20

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi NIFI-2118

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #586


commit 7e9887996b93654b2671c794bd8558da76d10805
Author: jpercivall 
Date:   2016-06-27T01:03:46Z

NIFI-2118 Addresssing License/Notice issues in nifi-assumbly




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #19: MINIFI-40 - Defaulting to empty string for key...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/19#discussion_r68591275
  
--- Diff: minifi-bootstrap/pom.xml ---
@@ -43,6 +43,10 @@ limitations under the License.
 
 
 org.apache.nifi
+nifi-properties
--- End diff --

Instead of adding another pom dependency for one static helper method I 
would prefer to just create a new private one that does it:
private static boolean nullOrEmpty(String input) {
return input == null || input.isEmpty();
}


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #20: MINIFI-43 Fixing provenance instantiation and ...

[JPercivall]

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi-minifi/pull/20

MINIFI-43 Fixing provenance instantiation and making some default con…

…structors private to prevent same problem in other classes

I made the no parameter constructor private when there are required 
properties with no default value. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi-minifi MINIFI-43

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi/pull/20.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20


commit 1c5c2d49e5e0477a0b6dbb58cbd2c3ecf73b4a52
Author: Joseph Percivall 
Date:   2016-06-27T16:52:58Z

MINIFI-43 Fixing provenance instantiation and making some default 
constructors private to prevent same problem in other classes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/576
  
There have already been comments asking for a multi-get function for 
ElasticSearch[1] could this be added to FetchElasticSearchHTTP now and be added 
as a new feature to PutElasticSearch later (Create a Jira for it)?

[1] 
https://community.hortonworks.com/questions/41951/how-to-get-all-values-with-expression-language-in.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68660039
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68666372
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68666289
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68671269
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68777286
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68780983
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68780468
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68785240
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68799710
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r68943688
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java
 ---
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.Kernel32Util;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@TriggerSerially
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class ConsumeWindowsEventLog extends 
AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Que

[GitHub] nifi issue #525: NIFI-1976 - Windows Event Log native processor

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/525
  
+1

Did a contrib check build, reviewed the code and any comments were 
addressed. To test I ran three ConsumeWindowsEventLog processors following 
multiple Windows channels and did a query to limit one of them. I verified the 
results by looking at the built in Event Viewer program. Thanks @brosander for 
the contribution! I will rebase and merge it into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69157552
  
--- Diff: minifi-commons/minifi-commons-schema/pom.xml ---
@@ -0,0 +1,46 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+org.apache.nifi.minifi
+minifi-commons
+0.0.1-SNAPSHOT
+
+
+minifi-commons-schema
+jar
+
+
+
+org.yaml
+snakeyaml
+1.17
+
+
+org.apache.nifi
+nifi-framework-core
+0.6.0
+
+
--- End diff --

I don't think this is applicable for the new assembly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69159093
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/SchemaLoader.java
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.schema;
+
+import 
org.apache.nifi.minifi.commons.schema.exception.SchemaLoaderException;
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.error.YAMLException;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+public class SchemaLoader {
+public static Map loadYamlAsMap(InputStream 
sourceStream) throws IOException, SchemaLoaderException {
+try {
+Yaml yaml = new Yaml();
+
+// Parse the YAML file
+final Object loadedObject = yaml.load(sourceStream);
+
+// Verify the parsed object is a Map structure
+if (loadedObject instanceof Map) {
+return (Map) loadedObject;
+} else {
+throw new SchemaLoaderException("Provided YAML 
configuration is not a Map");
+}
+} catch (YAMLException e ) {
+throw new IOException(e);
+} finally {
+sourceStream.close();
+}
+}
+
+public static ConfigSchema loadConfigSchema(InputStream sourceStream) 
throws IOException, SchemaLoaderException {
--- End diff --

Since this is just a wrapper around loadYamlAsMap and has the requirement 
that the input is Yaml, I'd prefer to have "Yaml" in somewhere in the method 
name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69162642
  
--- Diff: minifi-toolkit/minifi-toolkit-assembly/NOTICE ---
@@ -0,0 +1,21 @@
+Apache NiFi
+Copyright 2014-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===
+Apache Software License v2
+===
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Apache NiFi
+The following NOTICE information applies:
+  Apache NiFi
+  Copyright 2014-2016 The Apache Software Foundation
+
+  (ASLv2) Jetty
+The following NOTICE information applies:
+   Jetty Web Container
--- End diff --

Is Jetty packaged in the toolkit-assembly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69165759
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

I think I originally should have made this class Abstract. One of the main 
things is the toMap method. In this class all it does it return a LinkedHashMap 
and that just feels off to me. I see that others are overriding it and just 
using this to get a LinkedHashMap. Also I think it is counter intuitive to have 
the ability to initialize a BaseSchema. 

Overall I believe all the "Value Access/Interpretation helper methods" 
should be make static and toMap would be abstract. Thoughts?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69166517
  
--- Diff: minifi-toolkit/minifi-toolkit-assembly/pom.xml ---
@@ -0,0 +1,68 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+minifi-toolkit
+org.apache.nifi.minifi
+0.0.1-SNAPSHOT
+
+minifi-toolkit-assembly
+pom
+This is the assembly of Apache NiFi - MiNiFi
--- End diff --

Should be changed to reflect it being the toolkit


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69178841
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

Right the "Validation Issue helper methods" access the validation issues, 
which is something every schema will have, so it's good to keep them 
implemented how they are. 

The "Value Access/Interpretation helper methods" are all methods that do 
not necessarily rely on the internal representation of each schema but instead 
are helper methods for accessing/interpreting values. So they can be static.

For toMap I agree that it's good to encourage the use of LinkedHashMap. 
Maybe a protected method getMap for the schemas to use. Then toMap would be 
abstract.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69180850
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

Ah very true, ok I'm on board with not making them static.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69210176
  
--- Diff: 
minifi-toolkit/minifi-toolkit-assembly/src/main/resources/config.sh ---
@@ -0,0 +1,133 @@
+#!/bin/sh
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 
2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+# chkconfig: 2345 20 80
+# description: Apache NiFi - MiNiFi
+#
+
+# Script structure inspired from Apache Karaf and other Apache projects 
with similar startup approaches
+
+SCRIPT_DIR=$(dirname "$0")
+SCRIPT_NAME=$(basename "$0")
+MINIFI_TOOLKIT_HOME=$(cd "${SCRIPT_DIR}" && cd .. && pwd)
+PROGNAME=$(basename "$0")
+
+
+warn() {
+echo "${PROGNAME}: $*"
+}
+
+die() {
+warn "$*"
+exit 1
+}
+
+detectOS() {
+# OS specific support (must be 'true' or 'false').
+cygwin=false;
+aix=false;
+os400=false;
+darwin=false;
+case "$(uname)" in
+CYGWIN*)
+cygwin=true
+;;
+AIX*)
+aix=true
+;;
+OS400*)
+os400=true
+;;
+Darwin)
+darwin=true
+;;
+esac
+# For AIX, set an environment variable
+if ${aix}; then
+ export LDR_CNTRL=MAXDATA=0xB000@DSA
+ echo ${LDR_CNTRL}
+fi
+}
+
+locateJava() {
+# Setup the Java Virtual Machine
+if $cygwin ; then
+[ -n "${JAVA}" ] && JAVA=$(cygpath --unix "${JAVA}")
+[ -n "${JAVA_HOME}" ] && JAVA_HOME=$(cygpath --unix "${JAVA_HOME}")
+fi
+
+if [ "x${JAVA}" = "x" ] && [ -r /etc/gentoo-release ] ; then
+JAVA_HOME=$(java-config --jre-home)
+fi
+if [ "x${JAVA}" = "x" ]; then
+if [ "x${JAVA_HOME}" != "x" ]; then
+if [ ! -d "${JAVA_HOME}" ]; then
+die "JAVA_HOME is not valid: ${JAVA_HOME}"
+fi
+JAVA="${JAVA_HOME}/bin/java"
+else
+warn "JAVA_HOME not set; results may vary"
+JAVA=$(type java)
+JAVA=$(expr "${JAVA}" : '.* \(/.*\)$')
+if [ "x${JAVA}" = "x" ]; then
+die "java command not found"
+fi
+fi
+fi
+# if command is env, attempt to add more to the classpath
+if [ "$1" = "env" ]; then
+[ "x${TOOLS_JAR}" =  "x" ] && [ -n "${JAVA_HOME}" ] && 
TOOLS_JAR=$(find -H "${JAVA_HOME}" -name "tools.jar")
+[ "x${TOOLS_JAR}" =  "x" ] && [ -n "${JAVA_HOME}" ] && 
TOOLS_JAR=$(find -H "${JAVA_HOME}" -name "classes.jar")
+if [ "x${TOOLS_JAR}" =  "x" ]; then
+ warn "Could not locate tools.jar or classes.jar. Please set 
manually to avail all command features."
+fi
+fi
--- End diff --

Is this block left over from the minifi.sh? I tried running it and got this:
Joseph-Percivall:bin jpercivall$ ./config.sh env

Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home
MiNiFi Toolkit home: 
/Users/jpercivall/projects/edge/nifi-minifi/minifi-toolkit/minifi-toolkit-assembly/target/minifi-0.0.1-SNAPSHOT-bin/minifi-toolkit-0.0.1-SNAPSHOT


Usage:

java org.apache.nifi.minifi.toolkit.configuration.ConfigMain  
options

Valid commands include:
transform: Transform template xml into MiNiFi config YAML
validate: Validate config YAML



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69214705
  
--- Diff: minifi-toolkit/minifi-toolkit-configuration/pom.xml ---
@@ -0,0 +1,42 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+minifi-toolkit
+org.apache.nifi.minifi
+0.0.1-SNAPSHOT
+
+minifi-toolkit-configuration
+jar
+
+
+
+org.apache.nifi.minifi
+minifi-commons-schema
+${project.version}
+
+
+org.apache.nifi
+nifi-framework-core
+0.6.0
--- End diff --

This should be added to the Dependency Management portion of the root pom 
so that it shares the nifi version set there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69218372
  
--- Diff: minifi-commons/minifi-commons-schema/pom.xml ---
@@ -0,0 +1,46 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+org.apache.nifi.minifi
+minifi-commons
+0.0.1-SNAPSHOT
+
+
+minifi-commons-schema
+jar
+
+
+
+org.yaml
+snakeyaml
+1.17
+
+
+org.apache.nifi
+nifi-framework-core
+0.6.0
+
+
--- End diff --

(Accidentally commented on the wrong pom originally) This should probably 
be removed from this pom too since it will be used in both assemblies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69306232
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/pom.xml
 ---
@@ -56,6 +56,11 @@ language governing permissions and limitations under the 
License. -->
 ${es.version}
 
 
+com.squareup.okhttp
+okhttp
+2.7.1
--- End diff --

What's the motiviation for using this version of OkHttp instead of the 
latest[1][2]?

[1] https://github.com/square/okhttp/tree/parent-3.3.1
[2] https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69307262
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearchProcessor.java
 ---
@@ -17,129 +17,25 @@
 package org.apache.nifi.processors.elasticsearch;
 
 import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
-import org.apache.nifi.components.Validator;
-import org.apache.nifi.logging.ComponentLog;
 import org.apache.nifi.processor.AbstractProcessor;
 import org.apache.nifi.processor.ProcessContext;
 import org.apache.nifi.processor.exception.ProcessException;
 import org.apache.nifi.processor.util.StandardValidators;
 import org.apache.nifi.ssl.SSLContextService;
-import org.apache.nifi.util.StringUtils;
-import org.elasticsearch.client.Client;
-import org.elasticsearch.client.transport.TransportClient;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.transport.InetSocketTransportAddress;
-
-import java.io.File;
-import java.lang.reflect.Constructor;
-import java.lang.reflect.InvocationTargetException;
-import java.lang.reflect.Method;
-import java.net.InetSocketAddress;
-import java.net.MalformedURLException;
-import java.net.URL;
-import java.net.URLClassLoader;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.atomic.AtomicReference;
-
 
+/**
+ * A base class for all Elasticsearch processors
+ */
 public abstract class AbstractElasticsearchProcessor extends 
AbstractProcessor {
 
-/**
- * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
- */
-private static final Validator HOSTNAME_PORT_VALIDATOR = new 
Validator() {
-@Override
-public ValidationResult validate(final String subject, final 
String input, final ValidationContext context) {
-final List esList = Arrays.asList(input.split(","));
-for (String hostnamePort : esList) {
-String[] addresses = hostnamePort.split(":");
-// Protect against invalid input like 
http://127.0.0.1:9300 (URL scheme should not be there)
-if (addresses.length != 2) {
-return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
-"Must be in hostname:port form (no scheme such 
as http://";).valid(false).build();
-}
-}
-return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
-"Valid cluster definition").valid(true).build();
-}
-};
-
-protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
-.name("Cluster Name")
-.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
-.required(true)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.defaultValue("elasticsearch")
-.build();
-
-protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
-.name("ElasticSearch Hosts")
-.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
-+ "host1:port,host2:port,  For example 
testcluster:9300.")
-.required(true)
-.expressionLanguageSupported(false)
-.addValidator(HOSTNAME_PORT_VALIDATOR)
-.build();
-
 public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
 .name("SSL Context Service")
 .description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
-+ "connections. This service only applies if the 
Shield plugin is available.")
++ "connections. This service only applies if the 
Elasticsearch endpoints have been protected by SSLShield plugin is available.")
--- End diff --

This sentence doesn't really make sense to me, specifically the "plugin is 
available" part


-

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69308812
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69309177
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+.descripti

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69311424
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69312240
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69312668
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69313182
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69313807
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69314193
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69314307
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java
 ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.HttpUrl;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "fetch", "read", "get", "http"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69316019
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69316350
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69317035
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69317281
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69317330
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69317662
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69318008
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69318837
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua

[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/576#discussion_r69319085
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
 ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.squareup.okhttp.MediaType;
+import com.squareup.okhttp.OkHttpClient;
+import com.squareup.okhttp.RequestBody;
+import com.squareup.okhttp.Response;
+import com.squareup.okhttp.ResponseBody;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.util.StringUtils;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.node.ArrayNode;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.commons.lang3.StringUtils.trimToEmpty;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", 
"put", "http"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document.")
+public class PutElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("Identifier Attribute")
+.description("The name of the FlowFile attribute containing 
the identifier for the document. If the Index Operation is \"index\", "
++ "this property may be left empty or evalua