[GitHub] nifi issue #584: closing InputStream after execution completes
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/584 Hello Is this PR associated with an Apache NiFi JIRA? If not can you please create one. Also, we'll want to update the commit message to reference that JIRA as well. You can see some of the key details that help make the PR process move along smoothly here: https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide Thanks for contributing! Joe --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments and ExtractEm...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/483 @trixpan i haven't been able to dive into the details of the pr to have more comments just yet. Hopefully others can engage as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/556 @JPercivall I do agree with @jskora interpretation fwiw. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/556 nice work on wrapping this stuff up @jskora and @JPercivall ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/556 @JPercivall @markap14 here is my two cents in reading this exchange. Joe's points are really good about what a user expects and would like to see. Mark's points are true but from a purely how it is implemented now perspective. Basically what we should, imho, being saying is that 'Joe your idea is right but as of now we cannot afford to do that well'. For now, it should cause a bulletin to happen which does give the user a sense of what is happening generally (and recently). And we should make a JIRA to figure out how to best achieve what joe describes. We absolutely want a dataflow manager to be able to see what went wrong. Perhaps we say something as simple as 'we don't index route events' and thus provenance will be ok. Or maybe we do more. Whatever it is we can do more and it should work as joeP had hoped. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r68044679 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/image/ExtractImageMetadataTest.java --- @@ -37,7 +37,7 @@ private static String BMP_HEADER = "BMP Header."; private static String JPEG_HEADER = "JPEG."; private static String GIF_HEADER = "GIF Header."; -private static String PNG_HEADER = "PNG."; +private static String PNG_HEADER = "PNG-"; --- End diff -- Ok. Even that punts the problem because it just means we'll stick with an older dependency. My vote here would be we keep it as-is. Update the migration guide to call this issue out when going to 0.7 (and 1.0). And we update the processor documentation to highlight that the attributes extracted by this processor are not stable and subject to change across upgrades. It isn't great but it is honest. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/556#discussion_r68041998
--- Diff:
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image",
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles
containing audio, video, image, and other file "
++ "types. This processor relies on the Apache Tika project for
file format detection and parsing. It "
++ "extracts a long list of metadata types for media files
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types,
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will
process all of them.")
+.
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r68041768 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/image/ExtractImageMetadataTest.java --- @@ -37,7 +37,7 @@ private static String BMP_HEADER = "BMP Header."; private static String JPEG_HEADER = "JPEG."; private static String GIF_HEADER = "GIF Header."; -private static String PNG_HEADER = "PNG."; +private static String PNG_HEADER = "PNG-"; --- End diff -- When composing Nar's people often think about how various processors do similar things and make sense as a bundle of like things from a user perspective. However, they're really only about classloader isolation and so the thinking should be more purely about their dependencies and if they can/should co-exist nicely. This case sounds like this to me where we're thinking intuitively that these processors should be bundled because they do similar (media processing) things and then we're trying to wrangle their dependencies. This is just a food for thought comment. I'm not suggesting we have to change this one. It may be correct as is. For this specific case, could we perhaps bridge the header/attribute used so that it will map from the new name to the old name? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/556#discussion_r67966608
--- Diff:
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image",
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles
containing audio, video, image, and other file "
++ "types. This processor relies on the Apache Tika project for
file format detection and parsing. It "
++ "extracts a long list of metadata types for media files
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types,
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata "
++ "will be inserted with the attribute name \".\", or \"\" if "
++ "\"Metadata Key Prefix\" is not provided.")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will
process all of them.")
+.
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67800869 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67799555 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67732066 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #248: NIFI-1568: Add Filter Capability to UnpackContent
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/248#discussion_r67604647
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UnpackContent.java
---
@@ -154,75 +171,88 @@ protected void init(final
ProcessorInitializationContext context) {
return properties;
}
-@Override
-public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
-FlowFile flowFile = session.get();
-if (flowFile == null) {
-return;
-}
+@OnStopped
+public void onStopped() {
+unpacker = null;
+fileFilter = null;
+}
-final ComponentLog logger = getLogger();
-String packagingFormat =
context.getProperty(PACKAGING_FORMAT).getValue().toLowerCase();
-if (AUTO_DETECT_FORMAT.equals(packagingFormat)) {
-final String mimeType =
flowFile.getAttribute(CoreAttributes.MIME_TYPE.key());
-if (mimeType == null) {
-logger.error("No mime.type attribute set for {}; routing
to failure", new Object[]{flowFile});
-session.transfer(flowFile, REL_FAILURE);
-return;
-}
+@OnScheduled
+public void onScheduled(ProcessContext context) throws
ProcessException {
+if (fileFilter == null) {
+fileFilter =
Pattern.compile(context.getProperty(FILE_FILTER).getValue());
+tarUnpacker = new TarUnpacker(fileFilter);
+zipUnpacker = new ZipUnpacker(fileFilter);
+flowFileStreamV3Unpacker = new FlowFileStreamUnpacker(new
FlowFileUnpackagerV3());
+flowFileStreamV2Unpacker = new FlowFileStreamUnpacker(new
FlowFileUnpackagerV2());
+flowFileTarUnpacker = new FlowFileStreamUnpacker(new
FlowFileUnpackagerV1());
+}
-switch (mimeType.toLowerCase()) {
-case "application/tar":
-packagingFormat = TAR_FORMAT;
-break;
-case "application/x-tar":
-packagingFormat = TAR_FORMAT;
-break;
-case "application/zip":
-packagingFormat = ZIP_FORMAT;
-break;
-case "application/flowfile-v3":
-packagingFormat = FLOWFILE_STREAM_FORMAT_V3;
-break;
-case "application/flowfile-v2":
-packagingFormat = FLOWFILE_STREAM_FORMAT_V2;
-break;
-case "application/flowfile-v1":
-packagingFormat = FLOWFILE_TAR_FORMAT;
-break;
-default: {
-logger.info("Cannot unpack {} because its mime.type
attribute is set to '{}', which is not a format that can be unpacked; routing
to 'success'", new Object[]{flowFile, mimeType});
-session.transfer(flowFile, REL_SUCCESS);
-return;
-}
-}
+PackageFormat format =
PackageFormat.getFormat(context.getProperty(PACKAGING_FORMAT).getValue());
+if (format != PackageFormat.AUTO_DETECT_FORMAT && unpacker ==
null) {
+initUnpacker(format);
}
+}
-final Unpacker unpacker;
-final boolean addFragmentAttrs;
+public void initUnpacker(PackageFormat packagingFormat) {
switch (packagingFormat) {
case TAR_FORMAT:
-unpacker = new TarUnpacker();
+case X_TAR_FORMAT:
+unpacker = tarUnpacker;
addFragmentAttrs = true;
break;
case ZIP_FORMAT:
-unpacker = new ZipUnpacker();
+unpacker = zipUnpacker;
addFragmentAttrs = true;
break;
case FLOWFILE_STREAM_FORMAT_V2:
-unpacker = new FlowFileStreamUnpacker(new
FlowFileUnpackagerV2());
+unpacker = flowFileStreamV2Unpacker;
addFragmentAttrs = false;
break;
case FLOWFILE_STREAM_FORMAT_V3:
-unpacker = new FlowFileStreamUnpacker(new
FlowFileUnpackagerV3());
+unpacker = flowFileStreamV3Unpacker;
addFragmentAttrs = false;
break;
case FLOWFILE_TAR_FORMAT:
-unpacker = new FlowFileStreamUn
[GitHub] nifi issue #543: NIFI-1834 Create PutTCP Processor
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/543 In my view the demarcator is a good start. A header and footer could be added later or even just a header could be added and the demarcator could be considered the footer. Something like that. However, what I do think needs to be considered now is adding a way for the character set of the demarcator to be expressed explicitly. Right now it calls demarcator.getBytes() and that will result in different behavior on different platforms and such string to byte cases always cause issues when not explicit. Easy change for sure. Could also allow the user to supply a file containing bytes as an another option as well. Anyway - would do the simplest approach now which gets this processor in circulation then let usage dictate next steps. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #475: - Add Maven profile to compile nifi-hadoop-libraries-nar us...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/475 @trixpan @mattyb149 We don't need to aim to cover any or all vendors. If someone wants to submit a patch for whatever vendor's specific distribution they want then that is fine provided all the typical licensing and notice considerations are adhered to. I haven't looked into MapRs licensing for the dependent libraries we need here but if it is compatible with the proper ASF/ASLv2 licensing then great. If not then it should not be included even as a profile. Technically speaking we do not official release binary builds at all. We do provide them as a convenience and we do have to ensure even those binary builds have full valid licensing. The thing we are officially releasing is the source release so yes we could probably 'get away with' leaving the profile in but it seems quite avoidable. If we want to allow someone to augment their hadoop dependent libraries with vendor specific things and we're worried about licensing we could simply have default pure ASF dependency profi le and then let users override the profile using their settings.xml which we can document how to do. But, if MapR or CDH or HDP libraries are valid then anyone can make those available. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #492: NIFI-1975 - Processor for parsing evtx files
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/492#discussion_r66277217 --- Diff: nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,36 @@ +nifi-evtx-nar +Copyright 2016 The Apache Software Foundation + +This includes derived works from the Apache Software License V2 library python-evtx (https://github.com/williballenthin/python-evtx) +Copyright 2012, 2013 Willi Ballenthin william.ballent...@mandiant.com +while at Mandiant http://www.mandiant.com +The derived work is adapted from Evtx/Evtx.py, Evtx/BinaryParser.py, Evtx/Nodes.py, Evtx/Views.py and can be found in the org.apache.nifi.processors.evtx.parser package. + --- End diff -- i am hardly authoritative but i did review this case and believe it to be correct. Some would argue it is more than necessary i'm sure but let's err on the side of doing more than we must. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #397: NIFI-1815
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/397 @olegz definitely was thinking a customValidate check which determined whether tesseract was ready to roll. If the check you mention does that then that sounds good. We should be careful about ever call to customValidate resulting in an exception so probably best to periodically check if tesseract can work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #458: NIFI-1829 - Create new DebugFlow processor.
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/458 The name isn't great but not sure what would be better. It is indeed a debugger for the framework and less so for a flow. It is perhaps good to call it 'CreateChaos'. That said i personally think this is a great idea to have in the build and part of the standard processors. It would only be used by power users or curious types. And for everyone else it is a noop. But it could make it easier for us to demonstrate the evil effects of constant exceptions in a flow and do interesting regression checks on various infrastructures. There is admittedly probably a better place for this but with the upcoming nar registry work we can tackle that then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #492: NIFI-1502 - Processor for parsing evtx files
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/492#discussion_r65771143 --- Diff: nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,35 @@ +nifi-evtx-nar +Copyright 2016 The Apache Software Foundation + +This product includes the following work from python-evtx (https://github.com/williballenthin/python-evtx): --- End diff -- i take that back. They are needed. But would rephrase to This contains includes derived works from the Apache Software License V2 library python-evtx (https://github.com/williballenthin/python-evtx) Copyright 2012, 2013 Willi Ballenthin william.ballent...@mandiant.com while at Mandiant http://www.mandiant.com The derived work is adapted from Evtx/Evtx.py, Evtx/BinaryParser.py,Evtx/Nodes.py and can be found in the org.apache.nifi.processors.evtx.parser package. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #492: NIFI-1502 - Processor for parsing evtx files
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/492 ok i reviewed the handling of the ported python evtx library. I provided my suggestions on what I believe would be correct. Given that it was apache licensed making derived works is fortunately pretty straightforward. While it did not contain any NOTICE and thus items to carry forward it did contain copyright entries on their source files themselves so the recommendations I've made honor those by giving them an honorable mention in the notice file of our source distribution. And actually now that I think of it you will want to take that rephrased NOTICE section i mention and also include it in the nifi-assembly/NOTICE file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #492: NIFI-1502 - Processor for parsing evtx files
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/492#discussion_r65770640 --- Diff: NOTICE --- @@ -7,3 +7,7 @@ The Apache Software Foundation (http://www.apache.org/). This product includes the following work from the Apache Hadoop project: BoundedByteArrayOutputStream.java adapted to SoftLimitBoundedByteArrayOutputStream.java + +This product includes the following work from python-evtx (https://github.com/williballenthin/python-evtx): --- End diff -- I would recommend rephrasing to This contains includes derived works from the Apache Software License V2 library python-evtx (https://github.com/williballenthin/python-evtx) Copyright 2012, 2013 Willi Ballenthin <william.ballent...@mandiant.com> while at Mandiant <http://www.mandiant.com> The derived work is adapted from Evtx/Evtx.py, Evtx/BinaryParser.py,Evtx/Nodes.py and can be found in the org.apache.nifi.processors.evtx.parser package. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #492: NIFI-1502 - Processor for parsing evtx files
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/492#discussion_r65769425 --- Diff: nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/ParseEvtx.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.evtx; + +import com.google.common.annotations.VisibleForTesting; +import com.google.common.net.MediaType; +import com.google.common.primitives.UnsignedLong; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processors.evtx.parser.ChunkHeader; +import org.apache.nifi.processors.evtx.parser.FileHeader; +import org.apache.nifi.processors.evtx.parser.FileHeaderFactory; +import org.apache.nifi.processors.evtx.parser.MalformedChunkException; +import org.apache.nifi.processors.evtx.parser.Record; +import org.apache.nifi.processors.evtx.parser.XmlBxmlNodeVisitor; +import org.apache.nifi.processors.evtx.parser.bxml.RootNode; + +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamException; +import javax.xml.stream.XMLStreamWriter; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; + +/** + * Created by brosander on 5/24/16. --- End diff -- Recommend removal of these 'Created by brosander' lines on all things. By contributing the software becomes of the Apache NiFi community. For provenance considerations necessary for discussion/etc.. we have git blame/etc.. but for all rights/attribution/etc.. we have the Apache License and NOTICE associated with the project overall. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #492: NIFI-1502 - Processor for parsing evtx files
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/492#discussion_r65769083 --- Diff: nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,35 @@ +nifi-evtx-nar +Copyright 2016 The Apache Software Foundation + +This product includes the following work from python-evtx (https://github.com/williballenthin/python-evtx): --- End diff -- lines 4-8 are not needed here. This is necessary for the NOTICE of the source release which is the top level nifi/NOTICE file but this NOTICE file within the nar is only for the binary distribution of the nar and that does not contain these sources. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #492: NIFI-1502 - Processor for parsing evtx files
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/492 regarding Java8 angle obviously that just means this can only go in 1.x line and not 0.x. That is ok though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments processor
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/483 Sounds quite good to me @trixpan . I would just not worry about the additional attributes - yet. You identified good standard ones and if we find the need for more then we can iterate. You agree? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #474: NIFI-1919 Add replaceFirst method to expression language
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/474 Please just make sure we don't create problems for old flows. Can change names if we have a smooth transition for existing EL statements. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1899 - Introduce ExtractEmailAttachments processor
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/483 I think the behavior you have now is perfect (it should auto decode from base64) as it honors the user's intent. As far as tracing to the parent you just need to connect the provenance trail which can be done automatically through using the appropriate api methods. @markap14 or @bbende could either of you help point in the right direction here? For splitting the processors up I *think* i like two different processors. First is an ExtractEmailAttributes or some such name and then is a ExtractEmailAttachments which inherits attributes from the parent flow file. I haven't dug into this deep enough though to think my opinion is worth much though :-) Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1899 - Introduce ExtractEmailAttachments processor
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/483#discussion_r65358110
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/ExtractEmailAttachments.java
---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors;;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import javax.activation.DataSource;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "attachment.original.filename ",
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "filename ", description = "The
filename of the parent FlowFile"),
--- End diff --
we all are!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1899 - Introduce ExtractEmailAttachments processor
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/483#discussion_r65352609
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/ExtractEmailAttachments.java
---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors;;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import javax.activation.DataSource;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "attachment.original.filename ",
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "filename ", description = "The
filename of the parent FlowFile"),
--- End diff --
you probably mean filename of the attachment itself here right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1899 - Introduce ExtractEmailAttachments processor
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/483 Cool! Seems like a great start. Left a couple comments and have a couple question/comments. Might be a good idea to have more flow file attributes on the attachment such as any other information like timing/etc.. associated with the attachment if there are any (if not nevermind). Perhaps there is value in knowing a given attachment is the 2nd attachment out of 4 for example. Also, the attachments could be base64 encoded (i think). If that is right do you know if that mail api you're using will automatically convert them? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1899 - Introduce ExtractEmailAttachments processor
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/483#discussion_r65352657
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/ExtractEmailAttachments.java
---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors;;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import javax.activation.DataSource;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "attachment.original.filename ",
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "filename ", description = "The
filename of the parent FlowFile"),
+@WritesAttribute(attribute = "mime.type", description = "The
mime.type of the FlowFile which will be text/plain for Syslog messages.")})
--- End diff --
you probably mean from the attachment's stated mime type here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1767 AWS IoT processors
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/349 @JPercivall you by chance have bandwidth to keep progressing this with @KayLerch ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1280 Create FilterCSVColumns Processor.
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/420#discussion_r64578812 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -235,6 +235,11 @@ language governing permissions and limitations under the License. --> json-utils 0.0.20 + +org.apache.calcite +calcite-example-csv --- End diff -- "We don't want multiple CSV libraries, do we?" I agree it is ideal to avoid multiple libraries that 'do the same thing'. However... What we really favor is much more than just that. Bringing in something like Calcite's libraries means we're adopting their design decisions which they made out of good faith and effort just as we do with internally developed things. These cases cause natural tension on transitive dependencies. This is a great example of why we've provided the classloader isolation model we have in NiFi. https://en.wiktionary.org/wiki/let_a_thousand_flowers_bloom --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1907 Moving lazy init of SSLContext...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/457#issuecomment-220988156 @bbende i think the move makes sense to improve the lifecycle of when such issues are detected. However, it does seem like throwing RuntimeExceptions for those could be improved. Could they instead be considered IllegalStateExceptions and be a documented part of that interface? Also for the last runtime exception should we include the underlying exception when we create a new exception or do we intentionally bury it here? That said these questions/concerns are not 'because of this change' and i do agree the context should be constructed during the building phase of the client so I'm a +1 anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Nifi 539c
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/453#issuecomment-220190814 For what it is worth @olegz I too prefer to see folks avoid using '-1' because it is a bit of a blunt instrument in communities like this and there are good alternatives to keep the discussion going. However, my read on this does align with @mosermw in that this does look like another SFTP implementation so I too feel like this needs more work if we want to support SCP which offers subtle differences that may be preferred in some cases over SFTP. Knowing Mike I knew to take his -1 with nothing but noble intent and his follow-on comments support that as well. Jsch is likely a perfectly fine library for this but as mike points out there are different api calls that are likely to be used for SCP. @eorgad Would be curious to hear your thoughts on some good next steps to take with this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764, NIFI-1837, NIF...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/366#issuecomment-219842672 i certainly agree. this thing has to be one of the most reviewed PRs of all time. +1 please do merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Eorgad patch nifi 539
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/416#discussion_r63533145
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/SCPTransfer.java
---
@@ -0,0 +1,682 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.LoggerFactory;
+
+import com.jcraft.jsch.ChannelSftp;
+import com.jcraft.jsch.ChannelSftp.LsEntry;
+import com.jcraft.jsch.ChannelSftp.LsEntrySelector;
+import com.jcraft.jsch.JSch;
+import com.jcraft.jsch.JSchException;
+import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpException;
+
+
+public class SCPTransfer implements FileTransfer {
+
+public static final PropertyDescriptor PRIVATE_KEY_PATH = new
PropertyDescriptor.Builder()
+.name("Private Key Path")
+.description("The fully qualified path to the Private Key
file")
+.required(false)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.build();
+public static final PropertyDescriptor PRIVATE_KEY_PASSPHRASE = new
PropertyDescriptor.Builder()
+.name("Private Key Passphrase")
+.description("Password for the private key")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+public static final PropertyDescriptor HOST_KEY_FILE = new
PropertyDescriptor.Builder()
+.name("Host Key File")
+.description("If supplied, the given file will be used as the
Host Key; otherwise, no use host key file will be used")
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.required(false)
+.build();
+public static final PropertyDescriptor STRICT_HOST_KEY_CHECKING = new
PropertyDescriptor.Builder()
+.name("Strict Host Key Checking")
+.description("Indicates whether or not strict enforcement of
hosts keys should be applied")
+.allowableValues("true", "false")
+.defaultValue("false")
+.required(true)
+.build();
+public static final PropertyDescriptor PORT = new
PropertyDescriptor.Builder()
+.name("Port")
+.description("The port that the remote system is listening on
for file transfers")
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.required(true)
+.defaultValue("22")
+.build();
+public static final PropertyDescriptor USE_KEEPALIVE_ON_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("Send Keep Alive On Timeout")
+.description("Indicates whether or not to send a single Keep
Alive message
[GitHub] nifi pull request: Eorgad patch nifi 539
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/416#discussion_r63527657
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/SCPTransfer.java
---
@@ -0,0 +1,682 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.LoggerFactory;
+
+import com.jcraft.jsch.ChannelSftp;
+import com.jcraft.jsch.ChannelSftp.LsEntry;
+import com.jcraft.jsch.ChannelSftp.LsEntrySelector;
+import com.jcraft.jsch.JSch;
+import com.jcraft.jsch.JSchException;
+import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpException;
+
+
+public class SCPTransfer implements FileTransfer {
+
+public static final PropertyDescriptor PRIVATE_KEY_PATH = new
PropertyDescriptor.Builder()
+.name("Private Key Path")
+.description("The fully qualified path to the Private Key
file")
+.required(false)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.build();
+public static final PropertyDescriptor PRIVATE_KEY_PASSPHRASE = new
PropertyDescriptor.Builder()
+.name("Private Key Passphrase")
+.description("Password for the private key")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+public static final PropertyDescriptor HOST_KEY_FILE = new
PropertyDescriptor.Builder()
+.name("Host Key File")
+.description("If supplied, the given file will be used as the
Host Key; otherwise, no use host key file will be used")
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.required(false)
+.build();
+public static final PropertyDescriptor STRICT_HOST_KEY_CHECKING = new
PropertyDescriptor.Builder()
+.name("Strict Host Key Checking")
+.description("Indicates whether or not strict enforcement of
hosts keys should be applied")
+.allowableValues("true", "false")
+.defaultValue("false")
+.required(true)
+.build();
+public static final PropertyDescriptor PORT = new
PropertyDescriptor.Builder()
+.name("Port")
+.description("The port that the remote system is listening on
for file transfers")
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.required(true)
+.defaultValue("22")
+.build();
+public static final PropertyDescriptor USE_KEEPALIVE_ON_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("Send Keep Alive On Timeout")
+.description("Indicates whether or not to send a single Keep
Alive message
[GitHub] nifi pull request: Eorgad patch nifi 539
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/416#discussion_r63524926
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/SCPTransfer.java
---
@@ -0,0 +1,682 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.LoggerFactory;
+
+import com.jcraft.jsch.ChannelSftp;
+import com.jcraft.jsch.ChannelSftp.LsEntry;
+import com.jcraft.jsch.ChannelSftp.LsEntrySelector;
+import com.jcraft.jsch.JSch;
+import com.jcraft.jsch.JSchException;
+import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpException;
+
+
+public class SCPTransfer implements FileTransfer {
+
+public static final PropertyDescriptor PRIVATE_KEY_PATH = new
PropertyDescriptor.Builder()
+.name("Private Key Path")
+.description("The fully qualified path to the Private Key
file")
+.required(false)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.build();
+public static final PropertyDescriptor PRIVATE_KEY_PASSPHRASE = new
PropertyDescriptor.Builder()
+.name("Private Key Passphrase")
+.description("Password for the private key")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+public static final PropertyDescriptor HOST_KEY_FILE = new
PropertyDescriptor.Builder()
+.name("Host Key File")
+.description("If supplied, the given file will be used as the
Host Key; otherwise, no use host key file will be used")
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.required(false)
+.build();
+public static final PropertyDescriptor STRICT_HOST_KEY_CHECKING = new
PropertyDescriptor.Builder()
+.name("Strict Host Key Checking")
+.description("Indicates whether or not strict enforcement of
hosts keys should be applied")
+.allowableValues("true", "false")
+.defaultValue("false")
+.required(true)
+.build();
+public static final PropertyDescriptor PORT = new
PropertyDescriptor.Builder()
+.name("Port")
+.description("The port that the remote system is listening on
for file transfers")
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.required(true)
+.defaultValue("22")
+.build();
+public static final PropertyDescriptor USE_KEEPALIVE_ON_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("Send Keep Alive On Timeout")
+.description("Indicates whether or not to send a single Keep
Alive message
[GitHub] nifi pull request: Added PGP key to KEYS file.
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/450#issuecomment-219610061 Appreciate the spirit but this is not something you need RTC for. +1 in any event. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63410760
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
---
@@ -2527,6 +2554,43 @@ public String toString() {
return "StandardProcessSession[id=" + sessionId + "]";
}
+@Override
+public String getUnacknowledgedFlowfileInfo() {
--- End diff --
yeah i was just thinking that whenever rollback on a session is called it
would write some details about the state of that session at that time to the
log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: Eorgad patch nifi 539
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/416#issuecomment-219503574 @olegz i am not aware of any discussion or decision to avoid adding things to standard processors. I think for this one it is a perfectly valid place. For things with complicated dependencies and such I can see avoiding it but this one seems easy enough --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63296926
--- Diff:
nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java ---
@@ -109,6 +109,19 @@
void rollback(boolean penalize);
/**
+ * Produces a listing of pertinent information about the session's
+ * unacknowledged flowfiles that can be used for logging purposes. If
the
+ * session fails for any reason this can help identify problem files
with
+ * minimal system impact.
+ *
+ * Because of testing dependencies and ot
+ *
+ * @return {@link String} listing pertinent information about the
session's
+ * unacknowledged flowfiles, primarily intended for logging purposes.
+ */
+String getUnacknowledgedFlowfileInfo();
--- End diff --
this is good discusssion - taking this over to the JIRA to follow-up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63296904
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
---
@@ -175,6 +184,10 @@ public StandardProcessSession(final ProcessContext
context) {
this.sessionId = idGenerator.getAndIncrement();
this.connectableDescription = description;
+this.isRollbackCountEnabled =
NiFiProperties.getInstance().isRollbackCountEnabled();
+this.isRollbackLogUnackFFEnabled =
NiFiProperties.getInstance().isRollbackLogUnackFFEnabled();
+this.rollbackLogUnackFFMax =
NiFiProperties.getInstance().getRollbackLogUnackFFMax();
+
--- End diff --
Yeah...It just feels like attributes aren't the right place for this. Will
reply more thoughtfully on the JIRA itself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63242199
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FlowDebugger.java
---
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@EventDriven()
+@Tags({"test", "debug", "processor", "utility", "flow", "flowfile"})
+@CapabilityDescription("This processor aids in the testing and debugging
of the flowfile framework by allowing "
++ "a developer or flow manager to force various responses to a
flowfile. In response to a receiving a flowfile "
++ "it can route to the success or failure relationships, rollback,
rollback and yield, rollback with penalty, "
++ "or throw an exception. In addition, if using a timer based
scheduling strategy, upon being triggered without "
++ "a flowfile, it can be configured to throw an exception, when
triggered without a flowfile.\n"
++ "\n"
++ "The 'iterations' properties, such as \"Success iterations\",
configure how many times each response should occur "
++ "in succession before moving on to the next response within the
group of flowfile responses or no flowfile"
++ "responses.\n"
++ "\n"
++ "The order of responses when a flow file is received are:"
++ " 1. transfer flowfile to success relationship.\n"
++ " 2. transfer flowfile to failure relationship.\n"
++ " 3. rollback the flowfile without penalty.\n"
++ " 4. rollback the flowfile and yield the context.\n"
++ " 5. rollback the flowfile with penalty.\n"
++ " 6. throw an exception.\n"
++ "\n"
++ "The order of responses when no flow file is received are:"
++ " 1. yield the context.\n"
++ " 2. throw an exception.\n"
++ " 3. do nothing and return.\n"
++ "\n"
++ "By default, the processor is configured to perform each
response one time. After processing the list of "
++ "responses it will resume from the top of the list.\n"
++ "\n"
++ "To suppress any response, it's value can be set to zero (0) and
no responses of that type will occur during "
++ "processing.")
+public class FlowDebugger extends AbstractProcessor {
--- End diff --
this processor is an interesting idea. Does not belong in this particular
PR/JIRA though. Should be on its own.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63241990
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestStandardProcessSession.java
---
@@ -243,6 +243,16 @@ public void process(OutputStream out) throws
IOException {
flowFile = session.get();
assertNotNull(flowFile);
+
assertNotNull(flowFile.getAttribute(StandardProcessSession.ROLLBACK_COUNT_ATTR_NAME));
+assertEquals("1",
flowFile.getAttribute(StandardProcessSession.ROLLBACK_COUNT_ATTR_NAME));
+session.remove(flowFile);
+session.rollback();
+assertEquals(1, contentRepo.getExistingClaims().size());
+
+flowFile = session.get();
+assertNotNull(flowFile);
+
assertNotNull(flowFile.getAttribute(StandardProcessSession.ROLLBACK_COUNT_ATTR_NAME));
--- End diff --
not specific to this line but we should probably have details about a flow
files's rollback status/count/etc.. as part of the FlowFileRecord object rather
than the FlowFile class attributes. The idea I was hinting about was more
about making it possible when viewing information about a queue to see flow
files involved in a rollback recently to aid debugging. Putting it as
attributes, and ones which would not be persisted as is the case here (by
design in a rollback), heads in a different direction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63241497
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
---
@@ -2527,6 +2554,43 @@ public String toString() {
return "StandardProcessSession[id=" + sessionId + "]";
}
+@Override
+public String getUnacknowledgedFlowfileInfo() {
--- End diff --
I would make this private and have it called by rollback itself for the
sole purpose of writing to the log. If you want to return something to a
client/caller then it should definitely offer some sort of object model (set of
flowfiles) type return rather than a preformatted string.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63239852
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
---
@@ -175,6 +184,10 @@ public StandardProcessSession(final ProcessContext
context) {
this.sessionId = idGenerator.getAndIncrement();
this.connectableDescription = description;
+this.isRollbackCountEnabled =
NiFiProperties.getInstance().isRollbackCountEnabled();
+this.isRollbackLogUnackFFEnabled =
NiFiProperties.getInstance().isRollbackLogUnackFFEnabled();
+this.rollbackLogUnackFFMax =
NiFiProperties.getInstance().getRollbackLogUnackFFMax();
+
--- End diff --
not about this line specifically but it doesn't look like we'd be resetting
the flow file attributes after a given flow file was able to get passed a given
processor's commit call. Thus we could have rollback counts on attributes that
don't make sense any more since it was about a different processor. In either
case, we won't have the attributes retained anyway because unless a commit
happens there are no flow file attribute changes occurring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63239617
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
---
@@ -909,13 +922,27 @@ private void rollback(final boolean penalize, final
boolean rollbackCheckpoint)
if (record.getOriginal() != null) {
final FlowFileQueue originalQueue =
record.getOriginalQueue();
if (originalQueue != null) {
+FlowFileRecord fileRecord;
if (penalize) {
final long expirationEpochMillis =
System.currentTimeMillis() +
context.getConnectable().getPenalizationPeriod(TimeUnit.MILLISECONDS);
-final FlowFileRecord newFile = new
StandardFlowFileRecord.Builder().fromFlowFile(record.getOriginal()).penaltyExpirationTime(expirationEpochMillis).build();
-originalQueue.put(newFile);
+fileRecord = new
StandardFlowFileRecord.Builder().fromFlowFile(record.getOriginal()).penaltyExpirationTime(expirationEpochMillis).build();
} else {
-originalQueue.put(record.getOriginal());
+fileRecord = record.getOriginal();
}
+final Map<String, String> attributes = new HashMap<>();
+if (this.isRollbackCountEnabled) {
+try {
+Long rollbackCount = 0L;
+if
(record.getCurrent().getAttributes().containsKey(ROLLBACK_COUNT_ATTR_NAME)) {
+rollbackCount =
Long.parseLong(record.getOriginalAttributes().get(ROLLBACK_COUNT_ATTR_NAME));
--- End diff --
Very important to always avoid using Long or Int parsing to determine if
something is Long or Int in core high-rate framework code sections. Exception
handling for flow control in such cases can be an absolute performance killer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63238261
--- Diff:
nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java ---
@@ -109,6 +109,19 @@
void rollback(boolean penalize);
/**
+ * Produces a listing of pertinent information about the session's
+ * unacknowledged flowfiles that can be used for logging purposes. If
the
+ * session fails for any reason this can help identify problem files
with
+ * minimal system impact.
+ *
+ * Because of testing dependencies and ot
+ *
+ * @return {@link String} listing pertinent information about the
session's
+ * unacknowledged flowfiles, primarily intended for logging purposes.
+ */
+String getUnacknowledgedFlowfileInfo();
--- End diff --
also meant to say that returning a 'String' eliminates the caller/client
from having any control on formatting and details of output. It is fine to
return the flow file objects *i think* since they're immutable by intent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/430#discussion_r6323 --- Diff: nifi-assembly/pom.xml --- @@ -322,6 +322,10 @@ language governing permissions and limitations under the License. --> 30 sec 10 millis + false + false + 5 --- End diff -- I do think it would be great if we can find a nice way to signal flowfiles recently involved in a rollback. But I think whatever the solution is it should not require additional configuration/thought on the admin's part. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63237500
--- Diff:
nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java ---
@@ -109,6 +109,19 @@
void rollback(boolean penalize);
/**
+ * Produces a listing of pertinent information about the session's
+ * unacknowledged flowfiles that can be used for logging purposes. If
the
+ * session fails for any reason this can help identify problem files
with
+ * minimal system impact.
+ *
+ * Because of testing dependencies and ot
+ *
+ * @return {@link String} listing pertinent information about the
session's
+ * unacknowledged flowfiles, primarily intended for logging purposes.
+ */
+String getUnacknowledgedFlowfileInfo();
--- End diff --
When a session gets rolled back everything not committed (even things
already transferred but not committed) are part of the list. Also once a
session is rolled back or committed it should generally be able to be used
more. Therefore the information about the session you want to capture needs to
come back as a return of the rollback call itself most likely.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/430#discussion_r63233681
--- Diff: nifi-api/pom.xml ---
@@ -21,5 +21,12 @@
0.7.0-SNAPSHOT
nifi-api
-jar
+jar
+
+
+org.apache.nifi
+nifi-properties
+${project.version}
--- End diff --
we have avoided usage of ${project.version} references as it has caused
problems with the release plugin/process. Using explicit version if necessary
is sufficient as the plugin automatically handles this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764, NIFI-1837, NIF...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/366#issuecomment-219099819 Am able to build now after rebasing this on top of master. However, the ignored tests weren't just long running they were also unreliable as shown in the previous OutOfMemoryError and the preceding lifecycletest issue. They should be made faster/more discrete/reliable. Lots of System.out calls. Those should use the proper logger or be removed. There is an ignored test 'for sanity'. Fix or remove. This is closer but needs those items addressed before merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1872: Ignore failing unit test for now unt...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/437#issuecomment-218803112 i am certainly a +1 on ignoring this given that it provides a false-positive test failure making the build unstable. I think this is both 1.0 and 0.x thing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764, NIFI-1837, NIF...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/366#issuecomment-218793849 Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 441.183 sec - in org.apache.nifi.processors.kafka.pubsub.ConsumeKafkaTest Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764, NIFI-1837, NIF...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/366#issuecomment-218787020 [INFO] reporting-task.css (2512b) -> reporting-task.css (1264b)[50%] -> reporting-task.css.gz (488b)[19%] Tests run: 5, Failures: 2, Errors: 0, Skipped: 1, Time elapsed: 81.054 sec <<< FAILURE! - in org.apache.nifi.processors.kafka.pubsub.AbstractKafkaProcessorLifecycelTest validateConcurrencyWithAllFailures(org.apache.nifi.processors.kafka.pubsub.AbstractKafkaProcessorLifecycelTest) Time elapsed: 29.154 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.nifi.processors.kafka.pubsub.AbstractKafkaProcessorLifecycelTest.validateConcurrencyWithAllFailures(AbstractKafkaProcessorLifecycelTest.java:369) validateConcurrencyWithAllSuccesses(org.apache.nifi.processors.kafka.pubsub.AbstractKafkaProcessorLifecycelTest) Time elapsed: 33.284 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.nifi.processors.kafka.pubsub.AbstractKafkaProcessorLifecycelTest.validateConcurrencyWithAllSuccesses(AbstractKafkaProcessorLifecycelTest.java:291) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764, NIFI-1837, NIF...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/366#issuecomment-218763305 applying the patch version of this fails. merging this branch fails. uto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/test/resources/zookeeper.properties CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/test/resources/zookeeper.properties Auto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/test/resources/server.properties CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/test/resources/server.properties Auto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/test/resources/log4j.properties CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/test/resources/log4j.properties Auto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/resources/docs/org.apache.nifi.processors.kafka.pubsub.PublishKafka/additionalDetails.html CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/resources/docs/org.apache.nifi.processors.kafka.pubsub.PublishKafka/additionalDetails.html Auto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/resources/docs/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka/additionalDetails.html CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/resources/docs/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka/additionalDetails.html Auto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml Auto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-nar/src/main/resources/META-INF/NOTICE CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-nar/src/main/resources/META-INF/NOTICE Auto-merging nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-nar/src/main/resources/META-INF/LICENSE CONFLICT (add/add): Merge conflict in nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-nar/src/main/resources/META-INF/LICENSE Automatic merge failed; fix conflicts and then commit the result. The logs for travis-ci show an unused import (probably fixed by now). Please update patch to resolve merge conflicts and I'll try again. Also, likely worthwhile to have an 0.x and a master version of this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1858 Adding SiteToSiteProvenanceReportingT...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/419#issuecomment-217663105 It is expected. The ReportingTask isn't acting on behalf of the user that started it. It is acting on behalf of NiFi itself and so does have access. This sort of 'authorization gap' is no different than a DFM that cannot click on a particular piece of content for instance and yet they could redirect the live dataflow. Probably makes sense to think of it as someone with DFM access having both DFM and Provenance access. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Nifi AWS IoT processors
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/349#issuecomment-214778406 Errr..i messed that response up. I meant I agree with the point @JPercivall was making and understand what @KayLerch was thinking. I think the names should be GetAWSIoTMqtt PutAWSIoTMqtt GetAWSIoTShadow PutAWSIoTShadow The names have a lot going on but these are pretty purpose built for the AWS IoT platform. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Nifi 1540 - AWS Kinesis Get and Put Processors
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/239#issuecomment-214591612 certainly @mans2singh - I went ahead and removed the fix version from the JIRA. Totally understand and thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764 implemented new...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/366#discussion_r60837830
--- Diff:
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/PublishKafka.java
---
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.kafka.pubsub;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import
org.apache.nifi.processors.kafka.pubsub.KafkaPublisher.KafkaPublisherResult;
+import
org.apache.nifi.processors.kafka.pubsub.Partitioners.RoundRobinPartitioner;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({ "Apache", "Kafka", "Put", "Send", "Message", "PubSub" })
+@CapabilityDescription("Sends the contents of a FlowFile as a message to
Apache Kafka. The messages to send may be individual FlowFiles or may be
delimited, using a "
++ "user-specified delimiter, such as a new-line.")
+@DynamicProperty(name = "The name of a Kafka configuration property.",
value = "The value of a given Kafka configuration property.",
+ description = "These properties will be added on the
Kafka configuration after loading any provided configuration properties."
++ " In the event a dynamic property represents a property that was
already set as part of the static properties, its value wil be"
++ " overriden with warning message describing the override."
++ " For the list of available Kafka properties please refer to:
http://kafka.apache.org/documentation.html#configuration.;)
+public class PublishKafka extends AbstractKafkaProcessor {
+
+protected static final String FAILED_PROC_ID_ATTR = "failed.proc.id";
+
+protected static final String FAILED_LAST_ACK_IDX = "failed.last.idx";
+
+protected static final String FAILED_TOPIC_ATTR = "failed.topic";
+
+protected static final String FAILED_KEY_ATTR = "failed.key";
+
+protected static final String FAILED_DELIMITER_ATTR =
"failed.delimiter";
+
+static final AllowableValue DELIVERY_REPLICATED = new
AllowableValue("all", "Guarantee Replicated Delivery",
+"FlowFile will be routed to failure unless the message is
replicated to the appropriate "
++ "number of Kafka Nodes according to the Topic
configuration");
+static final AllowableValue DELIVERY_ONE_NODE = ne
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764 implemented new...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/366#discussion_r60797930
--- Diff:
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/test/java/org/apache/nifi/processors/kafka/pubsub/AbstractKafkaProcessorLifecycelTest.java
---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.kafka.pubsub;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.ByteArrayInputStream;
+import java.io.Closeable;
+import java.lang.reflect.Field;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.MockProcessSession;
+import org.apache.nifi.util.MockSessionFactory;
+import org.apache.nifi.util.SharedSessionState;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class AbstractKafkaProcessorLifecycelTest {
+
+@Test
+public void validateBaseProperties() throws Exception {
+TestRunner runner =
TestRunners.newTestRunner(DummyProcessor.class);
+runner.setProperty(AbstractKafkaProcessor.BOOTSTRAP_SERVERS, "");
+runner.setProperty(AbstractKafkaProcessor.TOPIC, "foo");
+runner.setProperty(ConsumeKafka.CLIENT_ID, "foo");
+
+try {
+runner.assertValid();
+fail();
+} catch (AssertionError e) {
+assertTrue(e.getMessage().contains("must contain at least one
character that is not white space"));
+}
+
+runner.setProperty(ConsumeKafka.BOOTSTRAP_SERVERS, "foo");
+try {
+runner.assertValid();
+fail();
+} catch (AssertionError e) {
+assertTrue(e.getMessage().contains("'bootstrap.servers'
validated against 'foo' is invalid"));
+}
+runner.setProperty(ConsumeKafka.BOOTSTRAP_SERVERS, "foo:1234");
+
+runner.removeProperty(ConsumeKafka.TOPIC);
+try {
+runner.assertValid();
+fail();
+} catch (AssertionError e) {
+assertTrue(e.getMessage().contains("'topic' is invalid because
topic is required"));
+}
+
+runner.setProperty(ConsumeKafka.TOPIC, "");
+try {
+runner.assertValid();
+fail();
+} catch (AssertionError e) {
+assertTrue(e.getMessage().contains("m
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764 implemented new...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/366#discussion_r60792649
--- Diff:
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/KafkaPublisher.java
---
@@ -78,152 +81,182 @@
}
/**
- *
- */
-void setProcessLog(ProcessorLog processLog) {
-this.processLog = processLog;
-}
-
-/**
- * Publishes messages to Kafka topic. It supports three publishing
- * mechanisms.
+ * Publishes messages to Kafka topic. It uses {@link StreamDemarcator}
to
+ * determine how many messages to Kafka will be sent from a provided
+ * {@link InputStream} (see {@link
PublishingContext#getContentStream()}).
+ * It supports two publishing modes:
*
- * Sending the entire content stream as a single Kafka
message.
- * Splitting the incoming content stream into chunks and sending
- * individual chunks as separate Kafka messages.
- * Splitting the incoming content stream into chunks and sending
only
- * the chunks that have failed previously @see
- * {@link SplittableMessageContext#getFailedSegments()}.
+ * Sending all messages constructed from
+ * {@link StreamDemarcator#nextToken()} operation.
+ * Sending only unacknowledged messages constructed from
+ * {@link StreamDemarcator#nextToken()} operation.
*
+ * The unacknowledged messages are determined from the value of
+ * {@link PublishingContext#getLastAckedMessageIndex()}.
+ *
* This method assumes content stream affinity where it is expected
that the
* content stream that represents the same Kafka message(s) will
remain the
* same across possible retries. This is required specifically for
cases
* where delimiter is used and a single content stream may represent
- * multiple Kafka messages. The failed segment list will keep the
index of
- * of each content stream segment that had failed to be sent to Kafka,
so
- * upon retry only the failed segments are sent.
+ * multiple Kafka messages. The
+ * {@link PublishingContext#getLastAckedMessageIndex()} will provide
the
+ * index of the last ACKed message, so upon retry only messages with
the
+ * higher index are sent.
*
- * @param messageContext
- *instance of {@link SplittableMessageContext} which hold
- *context information about the message to be sent
- * @param contentStream
- *instance of open {@link InputStream} carrying the
content of
- *the message(s) to be send to Kafka
- * @param partitionKey
- *the value of the partition key. Only relevant is user
wishes
- *to provide a custom partition key instead of relying on
- *variety of provided {@link Partitioner}(s)
- * @param maxBufferSize maximum message size
- * @return The set containing the failed segment indexes for messages
that
- * failed to be sent to Kafka.
+ * @param publishingContext
+ *instance of {@link PublishingContext} which hold context
+ *information about the message(s) to be sent.
+ * @return The index of the last successful offset.
*/
-BitSet publish(SplittableMessageContext messageContext, InputStream
contentStream, Integer partitionKey,
-int maxBufferSize) {
-List<Future> sendFutures =
this.split(messageContext, contentStream, partitionKey, maxBufferSize);
-return this.publish(sendFutures);
+KafkaPublisherResult publish(PublishingContext publishingContext) {
+StreamDemarcator streamTokenizer = new
StreamDemarcator(publishingContext.getContentStream(),
+publishingContext.getDelimiterBytes(),
publishingContext.getMaxRequestSize());
+
+int prevLastAckedMessageIndex =
publishingContext.getLastAckedMessageIndex();
+List<Future> resultFutures = new ArrayList<>();
+
+byte[] messageBytes;
+int tokenCounter = 0;
+for (; (messageBytes = streamTokenizer.nextToken()) != null;
tokenCounter++) {
+if (prevLastAckedMessageIndex < tokenCounter) {
+Integer partitionId = publishingContext.getPartitionId();
+if (partitionId == null && publishingContext.getKeyBytes()
!= null) {
+partitionId =
this.getPartition(publishingContext.getKeyBytes(),
publishingContext.getTopic());
+}
+ProducerRecord<byte[], byte[]> message
[GitHub] nifi pull request: NIFI-1296, NIFI-1680, NIFI-1764 implemented new...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/366#discussion_r60766260
--- Diff:
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/AbstractKafkaProcessor.java
---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.kafka;
+
+import java.io.Closeable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.exception.ProcessException;
+
+/**
+ * Base class for {@link Processor}s to publish and consume messages from
Kafka
+ *
+ * @see PutKafka
+ */
+abstract class AbstractKafkaProcessor extends
AbstractSessionFactoryProcessor {
+
+
+private volatile boolean acceptTask = true;
+
+private final AtomicInteger taskCounter = new AtomicInteger();
+
+
+/**
+ * @see KafkaPublisher
+ */
+volatile T kafkaResource;
+
+/**
+ *
+ */
+@Override
+public final void onTrigger(final ProcessContext context, final
ProcessSessionFactory sessionFactory) throws ProcessException {
+final ProcessSession session = sessionFactory.createSession();
+if (this.acceptTask) {
+try {
+this.taskCounter.incrementAndGet();
+if (this.kafkaResource == null) {
+synchronized (this) {
+if (this.kafkaResource == null) {
+this.kafkaResource =
this.buildKafkaResource(context, session);
+}
+}
+}
+
+this.rendezvousWithKafka(context, session);
+session.commit();
+} catch (Throwable e) {
+this.acceptTask = false;
+this.getLogger().error("{} failed to process due to {};
rolling back session", new Object[] { this, e });
+session.rollback(true);
+} finally {
+this.resetProcessorIfNecessary();
+}
+} else {
+context.yield();
+}
+}
+/**
+ * Resets the processor to initial state if necessary. The necessary
state
+ * means there are no active task which could only happen if currently
+ * executing tasks are given a chance to finish while no new tasks are
+ * accepted (see {@link #acceptTask}
+ */
+private boolean resetProcessorIfNecessary() {
+boolean reset = this.taskCounter.decrementAndGet() == 0 &&
!this.acceptTask;
+if (reset) {
+this.close();
+this.acceptTask = true;
+}
+return reset;
+}
+
+/**
+ * Will call {@link Closeable#close()} on the target resource after
which
+ * the target resource will be set to null
+ *
+ * @see KafkaPublisher
+ */
+@OnStopped
+public void close() {
--- End diff --
@olegz could we reduce the scope here? The concern I have is that
technically this is a check-then-modify scenario and it could be called by both
internal processor threads executing resetProcessorIfNecessary and whenever the
framework is calling onStopped. Now, by the way the framework works onStopped
should only be called once there are no threads left so technically this should
be legit. Just would not want someone to also call this from other places in
the cod
[GitHub] nifi pull request: NIFI-1764 fixed NPE in PutKafka
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/350#issuecomment-212598568 thanks Oleg. I'd like to review this as part of the larger #366 effort. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add supp...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/280#issuecomment-211956655 thanks @markobean and @jskora . What do you think about making ignore newlines only be honored/supported when not using the new features you're planning to include or only in very specific configurations? I ask because this, admittedly mistaken, feature is used a lot. Ultimately if that seems to unwieldy we can punt that feature in 1.0, add your new capabilities, and support end of line removal on ReplaceText instead. We just need to remember to document this in the migration guide for 1.0 as this could cause some pretty funky behavior changes for folks. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add supp...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/280#issuecomment-211726456 @jskora do you think this PR can be closed now given the updates made to fix the underlying defects found? A new PR could be submitted which adds the proposed features or goes into ReplaceText or a new processor. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/354#issuecomment-211726075 I am less concerned about thread safety of the transforms themselves now. Jolt's docs do refer to this at the readme level https://github.com/bazaarvoice/jolt/tree/1271e1919693e63ce1efdf534dfee0e0579d0b2f#-performance. So that leaves the update to NOTICE(s) concern :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/354#issuecomment-211724253 Oleg off-list pointed out the finer concerns with the member variable. While it seems pretty far out it also seems like it is accurate. In reading https://docs.oracle.com/javase/tutorial/essential/concurrency/atomic.html and https://docs.oracle.com/javase/tutorial/essential/concurrency/memconsist.html And indeed I cannot think of anything the framework is doing with thread management or how those lifecycle calls work that would enforce/guarantee a happens-before relationship for these processor member variables. @markap14 can you? That said the licensing and thread safety issue of the transform itself are still needing to be addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1614 File Identity Provider implementation
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/267#issuecomment-211716593 @jvwing @alopresto Have you had a chance to re-engage on this? It seems like a reasonable easy-path option for folks just wanting to use some simple/local username and password based setup. My responses to the questions James posed: What is required to make this viable? - This discussion appears on track Is there a better medium than bcrypt that combines widespread tool support with decent encryption. - Sounds like you and Andy both see it as a good option. Are we open to including a command-line user admin tool? - In my opinion we should be consistent that administrative actions occur by editing files on the command line in the less optimal case and interacting through a designed/intentional UX in the best case. We should strive to move away from config file based options and move fully towards service/REST API driven approaches. These will serve us better in clustered/cloud type environments as well. Are we open to including a sample credentials file? Where would you recommend it go? - Absolutely. In conf directory like the others of its type. I think an argument could be made to have this username/password driven mode be the default. Are we open to documenting this identity provider on the front-page of the Admin Guide alongside X.509 and LDAP? Where else should I do so? - We must do so. We should fully embrace this as an option and document what it is good for and not good for. Our current default of having no authentication at all is what we should be working to eliminate. I think this offers us a good first step to do that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-615 - Create a processor to extract WAV fi...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/252#issuecomment-211712568 @jskora I recall there was some discussion where you were considering supporting this as part of something a bit broader. Is that right or is this one you think good to go as is? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Nifi 1540 - AWS Kinesis Get and Put Processors
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/239#issuecomment-211711720 Hello @mans2singh . Do you think you'll have a chance to incorporate these review feedback items? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-840: Create ListS3 processor
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/238#issuecomment-211711025 Anyone in a good position to test/validate this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/354#issuecomment-211708711 @YolandaMDavis @olegz some feedback. 1. The com.bazaar.jolt... dependency does not appear to be accounted for in the LICENSE/NOTICE updates. Their LICENSE does indicate they would like reference made to 'Copyright 2013-2014 Bazaarvoice, Inc.' This is found in https://github.com/bazaarvoice/jolt/blob/1271e1919693e63ce1efdf534dfee0e0579d0b2f/LICENSE So we should propogate that into the NOTICE file as well. Also we need to reference JOLT in our top level nifi-assembly/NOTICE as well if it is new and I'd assume at least for Bazaar it would be. 2. I don't believe I follow the recommendation to change the member variable 'transform' to volatile or to set it to null on stopped. The variable appears to be set at the proper lifecycle point in NiFi and thus would not be reassigned in some non-thread safe way. The only question then is could its use itself be non Thread safe and that does appear questionable to me. In that case marking the variable as volatile won't help at all anyway. If indeed it is not thread safe you will want to consider marking the class as executing serially or you could have a pool of transforms or you could transform batches at a time. Lots of ways to tackle it. Of course if that is thread safe already then you are fine and in any event the volatile should be unnecessary. Finally, I am not aware of any cause for setting it to null during onStopped or on unScheduled as if the processor was being removed from the flow then it would get cleaned up anyway. If there was a need to call a close meth od or something then yes that is good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1337: Add Riemann Reporting Task
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/188#issuecomment-211702845 @rickysaltzer Looks like there might be a lack of time to focus on this at this time. How do you feel about closing this PR down for now and revisiting later? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NiFi-924: Camel integration
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/219#issuecomment-211701587 Commented on NIFI-924 to reference https://issues.apache.org/jira/browse/NIFI-1571 and consider this as resolved by that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Adjust batch.size property description and vali...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/335#issuecomment-207181902 Hello - thanks for contributing. Some feedback: - The commit message and the PR will need to associate to some open JIRA. If the JIRA were say NIFI-XYZ1 then the commit message should have "NIFI-XYZ1 ...the message" - The batch size does documentation does need to be changed. The change to a new default number of 16KB is fine but changing the name of the property is not ideal as it would break compatibility. You can change the display name safely but it is already appropriate. - The description you gave is good. - I like your change to using the data size validator but the problem is this would make all existing configurations invalid as they'd need to add the unit to their value. I am +0 on changing that personally as we do have provisions to deal with it and can document it in a migration guide. However, thinking further on it I'm told on older versions of Kafka that batch size really was about the number of messages so changing it may not be ideal. You could choose to simply fix the description of the property and leave the rest for now. There is an actively under work PR/JIRA to move to supporting the 0.9 Kafka producer and consumer API and that will give us a chance to tighten some of these behaviors up as well. It will be two new processors and these will still be around for some time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Adding requirejs, angularjs and its dependencie...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/331#issuecomment-206589245 The author email appears tied to a local system name. Probably not what you wanted. There doesn't appear to be updates to the LICENSE/NOTICE of NiFi which will be needed given this is a source dependency. It appears to include a lot of cruft like README files and gitignore files and so on which seem avoidable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1583 Added ConvertJSONtoCSV processor
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/254#issuecomment-206581942 I am not sure we can use org.json:json dependency. Its license as found in the readme of https://github.com/stleary/JSON-java/blob/master/README says "The license includes this restriction: "The software shall be used for good, not evil." If your conscience cannot live with that, then choose a different package."" I'm not sure this is acceptable in that it doesn't claim to be public domain or be part of any approved ASF friendly license. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1690 Changed MonitorMemory to use allowabl...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/328#discussion_r58706787
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-reporting-tasks/src/main/java/org/apache/nifi/controller/MonitorMemory.java
---
@@ -91,12 +92,22 @@
+ " that the memory pool is exceeding this threshold.")
public class MonitorMemory extends AbstractReportingTask {
+private static final AllowableValue[] memPoolAllowableValues;
+
+static {
+List memoryPoolBeans =
ManagementFactory.getMemoryPoolMXBeans();
+memPoolAllowableValues = new
AllowableValue[memoryPoolBeans.size()];
+for (int i = 0; i < memPoolAllowableValues.length; i++) {
+memPoolAllowableValues[i] = new
AllowableValue(memoryPoolBeans.get(i).getName());
+}
+}
+
public static final PropertyDescriptor MEMORY_POOL_PROPERTY = new
PropertyDescriptor.Builder()
.name("Memory Pool")
.description("The name of the JVM Memory Pool to monitor")
.required(true)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.defaultValue(null)
+.allowableValues(memPoolAllowableValues)
+.defaultValue(memPoolAllowableValues.length == 0 ? null :
memPoolAllowableValues[0].getValue())
--- End diff --
I think we're better off simply not having a default value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1728
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/325#issuecomment-205522127 @apiri probably best to just be explicit on all the tests about the charset. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1620 Allow empty Content-Type in InvokeHTT...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/272#issuecomment-199086549 @pvillard31 If you stay focused on the goal of NIFI-1620 here and avoid the body manipulation this seems like a good step. Clearly you found a case worthy of support (null body means unset or empty strong content type). But, i do share Adam's view that it is best to avoid the content manipulation here. Do you agree or do you feel like for the use case(s) you envisioned this would be problematic? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1578: Create PutSlack processor
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/256#issuecomment-198996892 @adamonduty Saw your tweet. https://twitter.com/microbatch/status/711618210886340608 I've not worked with Slack myself but have heard very positive things. We should definitely work this into the next feature release. Would be cool to hear a bit about your use case ideas for this or how you are using it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1651 unit tests work on windows. removed c...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/291#issuecomment-198995171 hmm good find @pvillard31 We need to make the code more locale tolerant apparently. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1337: Add Riemann Reporting Task
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/188#issuecomment-198151066 @rickysaltzer hey there. I just realized that this does have a commit against it post review commentary. My apologies for not engaging. Do you happen to be using this consistently/etc.. and are getting good results/like how it is working? I ask because the value of my own review will be fairly superficial and just want to make sure we have someone with good experience using it that is liking it (even if that is the author of it :)) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1488 Added hbase kerb auth with ugi
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/253#issuecomment-198146996 Hello @lordjc This ended up being merged and incorporated but the 'This closes #253' was not added to the commit log so the close hook didn't fire. Can you please close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/285#discussion_r56420919
--- Diff:
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext
context, final ProcessSessionFa
onTrigger(context, session);
session.commit();
} catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --
Right. So i think the short version is in the UI now. But you want the
stacktrace too. We should make that available in the logs. And this class is
what controls that I believe
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-logging-utils/src/main/java/org/apache/nifi/logging/NiFiLog.java
You'll see it has some ifDebugEnabled guards. We should probably just toss
those out. @markap14 what say you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1575: Add QueryDatabaseTable processor
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/261#issuecomment-198147766 @mattyb149 believe this PR is closed but apparently the commit message didn't reflect it. Can you please close it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/285#discussion_r56420228
--- Diff:
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext
context, final ProcessSessionFa
onTrigger(context, session);
session.commit();
} catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --
that is actually the current behavior/intent. I think the change needed is
to simply get rid of the 'if debug enabled' stuff we did in the logger and
instead just always log the stacktrace when it is there. This is what Adam
Taft had advocated for previously as I recall. I now see why he was saying it.
So yeah in the UI it should be a short and ideally meaningful message (not
always possible) and if a throwable shows up we should put the whole stack
trace in the logs.
The current idea that a user will go in an do debug enabledjust has
proven to be a failed experiment in my view and as someone who advocated for
that I now think i was wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-615 - Create a processor to extract WAV fi...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/252#issuecomment-198149231 @jskora do the attribute names that come from the media bundles have anything special to them in terms of how tika handles them or are they purely as found in the metadata of the raw entities? Just want to make sure there isn't some special mapping/normalization to worry about as versions of tika evolves. Also, i've not built this yet but do you know how large those parsers end up being when pulled in for the nar? I recall for some reason they can be quite huge which is why we have avoided them so far. Thinking being the are perfectly fine once we have the registry. Might be fine now too but curious. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/285#discussion_r56419500
--- Diff:
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext
context, final ProcessSessionFa
onTrigger(context, session);
session.commit();
} catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --
I would say generally no. They are uncommon in general but when they
happen they happen in bursts. As a general rule I think we should strive to
make what the user sees be something a lot more friendly than a wild stack
trace. But, the logs should have them. We did discuss this on dev list a
while back and there were some great inputs from various folks. We def need to
do something here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-615 - Create a processor to extract WAV fi...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/252#issuecomment-198411900 size = cool should be just fine then. attributes = interesting if they are named by tika. rather than being as they're found in the source media. Probably makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-614 Added initial support for new style JM...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/222#issuecomment-196858986 i am on the same page of this being a good start oleg provided the JIRA is created to enable ssl context to provide info for providers that want it. We don't have support for tibco ssl in apache now anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/271#discussion_r56103524
--- Diff:
nifi-nar-bundles/nifi-spring-bundle/nifi-spring-processors/src/main/java/org/apache/nifi/spring/SpringContextProcessor.java
---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.spring;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.spring.SpringDataExchanger.SpringResponse;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.messaging.Message;
+import org.springframework.messaging.MessageChannel;
+import org.springframework.messaging.PollableChannel;
+
+/**
+ * Implementation of {@link Processor} capable of sending and receiving
data
+ * from application defined in Spring Application context. It does so via
+ * predefined in/out {@link MessageChannel}s (see spring-messaging module
of
+ * Spring). Once such channels are defined user is free to implement the
rest of
+ * the application any way they wish (e.g., custom code and/or using
frameworks
+ * such as Spring Integration or Camel).
+ *
+ * The requirement and expectations for channel types are:
+ *
+ * Input channel must be of type {@link MessageChannel} and named
"fromNiFi"
+ * (see {@link SpringNiFiConstants#FROM_NIFI})
+ * Output channel must be of type {@link PollableChannel} and named
"toNiFi"
+ * (see {@link SpringNiFiConstants#TO_NIFI})
+ *
+ *
+ * Below is the example of sample configuration:
+ *
+ *
+ * ?xml version="1.0" encoding="UTF-8"?
+ * beans xmlns="http://www.springframework.org/schema/beans;
+ * xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ * xmlns:int="http://www.springframework.org/schema/integration;
+ * xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
+ * http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration-4.2.xsd";
+ *
+ * int:channel id="fromNiFi"/
+ *
+ * . . . . .
+ *
+ * int:channel id="toNiFi"
+ * int:queue/
+ * /int:channel
+ *
+ * /beans
+ *
+ *
+ * Defining {@link MessageChannel} is optional. That's why this processor
+ * s
[GitHub] nifi pull request: NIFI-1464 life-cycle refactoring part-2
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/275#discussion_r56073702
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/SchedulingAgentCallback.java
---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.Future;
+
+public interface SchedulingAgentCallback {
--- End diff --
would prefer that these api methods be documented.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1620 Allow empty Content-Type in InvokeHTT...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/272#issuecomment-196284703 Thank you @pvillard31 . That does make sense and in the case that there is no request body being sent over I am supportive of the notion of not sending the content type header. This also seems in line with Adam's rfc reference. You want to tweak this PR then to retain existing content header logic but rather have a 'send body' property or something which when false will not send the body and will not set that header? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1620 Allow empty Content-Type in InvokeHTT...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/272#issuecomment-196089886 i was reviewing this earlier today and frankly had a similar concern to this as Adam. I didn't reply because I hadn't really figured out what to think. First, I agree that a service which rejects that header is arguably broken. Second, as the patch is right now I am curious how it works when the value is empty string because there is a static call to MediaType which seems like it would have trouble (still need to verify the logic there though). However, having said this Pierre can you clarify if the intent is only for the case where there is no entity body or is it also for when there is an entity body in the request? If the idea is that this is only necessary when there is no entity body we should tighten the code for that case and if it is for either scenario then I think i'm of similar mind to Adam here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1511 now only sets groovy eclipse compiler...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/270#issuecomment-195434033 operation squash that jazz be complete --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1511 now only sets groovy eclipse compiler...
GitHub user joewitt opened a pull request: https://github.com/apache/nifi/pull/270 NIFI-1511 now only sets groovy eclipse compiler for test compiles NIFI-1511 now only sets groovy eclipse compiler for test compiles You can merge this pull request into a Git repository by running: $ git pull https://github.com/joewitt/incubator-nifi NIFI-1511 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/270.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #270 commit aaee620dc858d83e92a32145e15548728c3c61c3 Author: joewitt <joew...@apache.org> Date: 2016-03-11T15:33:46Z NIFI-1511 now only sets groovy eclipse compiler for test compiles --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
