[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69245092
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/test/resources/attachment-only.eml
 ---
@@ -0,0 +1,156 @@
+Date: Fri, 29 Jul 2011 09:41:11 +0200 (CET)
+From: Siegfried GOESCHL 
+Subject: Kunde 100029   Auftrag   3600
+To: 
+Reply-To: Siegfried GOESCHL 
+Message-ID: 
+MIME-Version: 1.0
+Importance: Normal
+X-Priority: 3 (Normal)
+X-Mailer: SAP Web Application Server 7.00
+Content-Type: application/pdf;
+ name="Kunde 100029   Auftrag   3600.pdf"
+Content-Transfer-Encoding: base64
+Content-Description: Kunde 100029   Auftrag   3600
--- End diff --

Yes, I noticed that, yet still concerned about that. @joewitt @bbende Would 
you give us a guidance on this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69244471
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailHeaders.java
 ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+import javax.mail.Address;
+import javax.mail.Header;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Using the flowfile content as source of data, 
extract header from an RFC  compliant  email file adding the relevant 
attributes to the flowfile. " +
+"This processor does not perform extensive RFC validation but 
still requires a bare minimum compliance with RFC 2822")
+@WritesAttributes({
+@WritesAttribute(attribute = "email.headers.bcc.*", description = 
"Each individual BCC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.cc.*", description = 
"Each individual CC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.from.*", description = 
"Each individual mailbox contained in the From  of the Email (array as per 
RFC-2822)"),
+@WritesAttribute(attribute = "email.headers.message-id", 
description = "The value of the Message-ID header (if available)"),
+@WritesAttribute(attribute = "email.headers.received_date", 
description = "The Received-Date of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.sent_date", 
description = "Date the message was sent"),
+@WritesAttribute(attribute = "email.headers.subject", description 
= "Subject of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.to.*", description = 
"Each individual TO recipient (if available)"),
+@WritesAttribute(attribute = "email.attachment_count", description 
= "Number of attachments of the message" )})
+
+public class ExtractEmailHeaders extends AbstractProcessor {
+public static final 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69244062
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailHeaders.java
 ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+import javax.mail.Address;
+import javax.mail.Header;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Using the flowfile content as source of data, 
extract header from an RFC  compliant  email file adding the relevant 
attributes to the flowfile. " +
+"This processor does not perform extensive RFC validation but 
still requires a bare minimum compliance with RFC 2822")
+@WritesAttributes({
+@WritesAttribute(attribute = "email.headers.bcc.*", description = 
"Each individual BCC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.cc.*", description = 
"Each individual CC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.from.*", description = 
"Each individual mailbox contained in the From  of the Email (array as per 
RFC-2822)"),
+@WritesAttribute(attribute = "email.headers.message-id", 
description = "The value of the Message-ID header (if available)"),
+@WritesAttribute(attribute = "email.headers.received_date", 
description = "The Received-Date of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.sent_date", 
description = "Date the message was sent"),
+@WritesAttribute(attribute = "email.headers.subject", description 
= "Subject of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.to.*", description = 
"Each individual TO recipient (if available)"),
+@WritesAttribute(attribute = "email.attachment_count", description 
= "Number of attachments of the message" )})
+
+public class ExtractEmailHeaders extends AbstractProcessor {
+public static final 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69243783
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
+public class ExtractEmailAttachments extends AbstractProcessor {
+public static final String ATTACHMENT_ORIGINAL_FILENAME = 
"email.attachment.parent.filename";
+public static final String ATTACHMENT_ORIGINAL_UUID = 
"email.attachment.parent.uuid";
+
+public static final Relationship REL_ATTACHMENTS = new 
Relationship.Builder()
+.name("attachments")
+.description("Each individual attachment will be routed to the 
attachments relationship")
+.build();
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder()
+.name("original")
+.description("The original file")
+.build();
+public static final Relationship REL_FAILURE = new 
Relationship.Builder()
+.name("failure")
+.description("Flowfiles that could not be parsed")
+.build();
+private Set relationships;
+private 

[GitHub] nifi-minifi pull request #22: MINIFI-42 Adjusting handling of the remote pro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/22


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69242757
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
+public class ExtractEmailAttachments extends AbstractProcessor {
+public static final String ATTACHMENT_ORIGINAL_FILENAME = 
"email.attachment.parent.filename";
+public static final String ATTACHMENT_ORIGINAL_UUID = 
"email.attachment.parent.uuid";
+
+public static final Relationship REL_ATTACHMENTS = new 
Relationship.Builder()
+.name("attachments")
+.description("Each individual attachment will be routed to the 
attachments relationship")
+.build();
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder()
+.name("original")
+.description("The original file")
+.build();
+public static final Relationship REL_FAILURE = new 
Relationship.Builder()
+.name("failure")
+.description("Flowfiles that could not be parsed")
+.build();
+private Set relationships;
+private List 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69241825
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/test/resources/attachment-only.eml
 ---
@@ -0,0 +1,156 @@
+Date: Fri, 29 Jul 2011 09:41:11 +0200 (CET)
+From: Siegfried GOESCHL 
+Subject: Kunde 100029   Auftrag   3600
+To: 
+Reply-To: Siegfried GOESCHL 
+Message-ID: 
+MIME-Version: 1.0
+Importance: Normal
+X-Priority: 3 (Normal)
+X-Mailer: SAP Web Application Server 7.00
+Content-Type: application/pdf;
+ name="Kunde 100029   Auftrag   3600.pdf"
+Content-Transfer-Encoding: base64
+Content-Description: Kunde 100029   Auftrag   3600
--- End diff --

As mentioned within the POM, the source file comes from Apache Commons test 
harness.

https://github.com/apache/commons-email/tree/trunk/src/test/resources/eml 

If that is still an issue, happy to address


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69241504
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,383 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to lister for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope)"),
+@WritesAttribute(attribute = "smtp.to", description = "The value 
used during RCPT TO (i.e. envelope)")})
+
+public class ListenSMTP extends AbstractProcessor {
+public static final String SMTP_HELO = "smtp.helo";
+public 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69241286
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,383 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to lister for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope)"),
+@WritesAttribute(attribute = "smtp.to", description = "The value 
used during RCPT TO (i.e. envelope)")})
+
+public class ListenSMTP extends AbstractProcessor {
+public static final String SMTP_HELO = "smtp.helo";
+public 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69241137
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailHeaders.java
 ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+import javax.mail.Address;
+import javax.mail.Header;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Using the flowfile content as source of data, 
extract header from an RFC  compliant  email file adding the relevant 
attributes to the flowfile. " +
+"This processor does not perform extensive RFC validation but 
still requires a bare minimum compliance with RFC 2822")
+@WritesAttributes({
+@WritesAttribute(attribute = "email.headers.bcc.*", description = 
"Each individual BCC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.cc.*", description = 
"Each individual CC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.from.*", description = 
"Each individual mailbox contained in the From  of the Email (array as per 
RFC-2822)"),
+@WritesAttribute(attribute = "email.headers.message-id", 
description = "The value of the Message-ID header (if available)"),
+@WritesAttribute(attribute = "email.headers.received_date", 
description = "The Received-Date of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.sent_date", 
description = "Date the message was sent"),
+@WritesAttribute(attribute = "email.headers.subject", description 
= "Subject of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.to.*", description = 
"Each individual TO recipient (if available)"),
+@WritesAttribute(attribute = "email.attachment_count", description 
= "Number of attachments of the message" )})
+
+public class ExtractEmailHeaders extends AbstractProcessor {
+public static final 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69241016
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,383 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to lister for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope)"),
+@WritesAttribute(attribute = "smtp.to", description = "The value 
used during RCPT TO (i.e. envelope)")})
+
+public class ListenSMTP extends AbstractProcessor {
+public static final String SMTP_HELO = "smtp.helo";
+public 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69240393
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,383 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to lister for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope)"),
+@WritesAttribute(attribute = "smtp.to", description = "The value 
used during RCPT TO (i.e. envelope)")})
+
+public class ListenSMTP extends AbstractProcessor {
+public static final String SMTP_HELO = "smtp.helo";
+public 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69240643
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/event/SmtpEvent.java
 ---
@@ -0,0 +1,93 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.nifi.processors.email.smtp.event;
+
+
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A Smtp event which adds the transaction number and command to the 
StandardEvent.
+ */
+
+public class SmtpEvent{
+
+private final String remoteIP;
+private final String helo;
+private final String from;
+private final String to;
+private final byte[] bodyData;
+private X509Certificate[] certificates;
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69240291
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,383 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to lister for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope)"),
+@WritesAttribute(attribute = "smtp.to", description = "The value 
used during RCPT TO (i.e. envelope)")})
+
+public class ListenSMTP extends AbstractProcessor {
+public static final String SMTP_HELO = "smtp.helo";
+public 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69239586
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,383 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to lister for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope)"),
+@WritesAttribute(attribute = "smtp.to", description = "The value 
used during RCPT TO (i.e. envelope)")})
+
+public class ListenSMTP extends AbstractProcessor {
+public static final String SMTP_HELO = "smtp.helo";
+public 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r69239406
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,383 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to lister for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope)"),
+@WritesAttribute(attribute = "smtp.to", description = "The value 
used during RCPT TO (i.e. envelope)")})
+
+public class ListenSMTP extends AbstractProcessor {
+public static final String SMTP_HELO = "smtp.helo";
+public 

Re: Question about Mock classes in ControllerServiceInitializer

[Oleg Zhurakousky]

Guys

FWIW, there are discussion points on the WIKI that may be relevant to 
understanding of this issue especially in relation to what Mark just stated
https://cwiki.apache.org/confluence/display/NIFI/Component+documentation+improvements

There are also some points in the Extension Registry (linked from the link 
above).
There is also an open JIRA with some more info: 
https://issues.apache.org/jira/browse/NIFI-1384

I think in reality if we declare (what is now obvious) a convention that 
PropertyDescriptors, Relationships etc. all have to be static variables then we 
would not need to create throw-away instances of class just to get their 
documentation.
Anyway, something to think about. . .

Cheers
Oleg

On Jun 30, 2016, at 1:59 PM, Mark Payne 
> wrote:

Joe,

I think the nifi-documentation module is using that to instantiate Processors, 
Controller Services, etc.
so that it can inspect their annotations & call their getPropertyDescriptors() 
methods, etc. when it generates
documentation for the component. Those should not be used for any component 
that is added to the flow.



On Jun 30, 2016, at 1:32 PM, Joe Skora 
> wrote:

Brandon and I have both run into log entries saying something along the
lines of "o.a.n.d.mock.MockProcessorLogger Shutting down server".

Checking the code
,
there are references to the MockProcessorLogger and MockConfigurationContext
in the org.apache.nifi.documentation.init.ControllerServiceInitizer,
ProcessorInitializer, and ReportingTaskingInitializer classes.

What are we missing?  Why are there Mock framework classes used in regular
classes?

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69218372
  
--- Diff: minifi-commons/minifi-commons-schema/pom.xml ---
@@ -0,0 +1,46 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.nifi.minifi
+minifi-commons
+0.0.1-SNAPSHOT
+
+
+minifi-commons-schema
+jar
+
+
+
+org.yaml
+snakeyaml
+1.17
+
+
+org.apache.nifi
+nifi-framework-core
+0.6.0
+
+
--- End diff --

(Accidentally commented on the wrong pom originally) This should probably 
be removed from this pom too since it will be used in both assemblies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #599: NIFI-2156: Add ListDatabaseTables processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/599#discussion_r69217243
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListDatabaseTables.java
 ---
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * A processor to retrieve a list of tables (and their metadata) from a 
database connection
+ */
+@TriggerSerially
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"sql", "list", "jdbc", "table", "database"})
+@CapabilityDescription("Generates a set of flow files, each containing 
attributes corresponding to metadata about a table from a database connection.")
+@WritesAttributes({
+@WritesAttribute(attribute = "db.table.name", description = 
"Contains the name of a database table from the connection"),
+@WritesAttribute(attribute = "db.table.catalog", description = 
"Contains the name of the catalog to which the table belongs (may be null)"),
+@WritesAttribute(attribute = "db.table.schema", description = 
"Contains the name of the schema to which the table belongs (may be null)"),
+@WritesAttribute(attribute = "db.table.fullname", description = 
"Contains the fully-qualifed table name (possibly including catalog, schema, 
etc.)"),
+@WritesAttribute(attribute = "db.table.type",
+description = "Contains the type of the database table 
from the connection. Typical types are \"TABLE\", \"VIEW\", \"SYSTEM TABLE\", "
++ "\"GLOBAL TEMPORARY\", \"LOCAL TEMPORARY\", 
\"ALIAS\", \"SYNONYM\""),
+@WritesAttribute(attribute = "db.table.remarks", description = 
"Contains the name of a database table from the connection"),
+@WritesAttribute(attribute = "db.table.count", description = 
"Contains the number of rows in the table")
+})
+@Stateful(scopes = {Scope.LOCAL}, description = "After performing a 
listing of tables, the timestamp of the query is stored. "
++ "This allows the Processor to not re-list tables the next time 
that the Processor is run. Changing any of the processor properties will "
++ "indicate that the processor should 

[GitHub] nifi pull request #599: NIFI-2156: Add ListDatabaseTables processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/599#discussion_r69216929
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListDatabaseTables.java
 ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.nifi.util.file.FileUtils;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.Statement;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Unit tests for ListDatabaseTables processor.
+ */
+public class TestListDatabaseTables {
--- End diff --

I am not familiar with Derby but would it be possible to also test 
catalog/schema aspects?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69214705
  
--- Diff: minifi-toolkit/minifi-toolkit-configuration/pom.xml ---
@@ -0,0 +1,42 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+minifi-toolkit
+org.apache.nifi.minifi
+0.0.1-SNAPSHOT
+
+minifi-toolkit-configuration
+jar
+
+
+
+org.apache.nifi.minifi
+minifi-commons-schema
+${project.version}
+
+
+org.apache.nifi
+nifi-framework-core
+0.6.0
--- End diff --

This should be added to the Dependency Management portion of the root pom 
so that it shares the nifi version set there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69210176
  
--- Diff: 
minifi-toolkit/minifi-toolkit-assembly/src/main/resources/config.sh ---
@@ -0,0 +1,133 @@
+#!/bin/sh
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 
2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+# chkconfig: 2345 20 80
+# description: Apache NiFi - MiNiFi
+#
+
+# Script structure inspired from Apache Karaf and other Apache projects 
with similar startup approaches
+
+SCRIPT_DIR=$(dirname "$0")
+SCRIPT_NAME=$(basename "$0")
+MINIFI_TOOLKIT_HOME=$(cd "${SCRIPT_DIR}" && cd .. && pwd)
+PROGNAME=$(basename "$0")
+
+
+warn() {
+echo "${PROGNAME}: $*"
+}
+
+die() {
+warn "$*"
+exit 1
+}
+
+detectOS() {
+# OS specific support (must be 'true' or 'false').
+cygwin=false;
+aix=false;
+os400=false;
+darwin=false;
+case "$(uname)" in
+CYGWIN*)
+cygwin=true
+;;
+AIX*)
+aix=true
+;;
+OS400*)
+os400=true
+;;
+Darwin)
+darwin=true
+;;
+esac
+# For AIX, set an environment variable
+if ${aix}; then
+ export LDR_CNTRL=MAXDATA=0xB000@DSA
+ echo ${LDR_CNTRL}
+fi
+}
+
+locateJava() {
+# Setup the Java Virtual Machine
+if $cygwin ; then
+[ -n "${JAVA}" ] && JAVA=$(cygpath --unix "${JAVA}")
+[ -n "${JAVA_HOME}" ] && JAVA_HOME=$(cygpath --unix "${JAVA_HOME}")
+fi
+
+if [ "x${JAVA}" = "x" ] && [ -r /etc/gentoo-release ] ; then
+JAVA_HOME=$(java-config --jre-home)
+fi
+if [ "x${JAVA}" = "x" ]; then
+if [ "x${JAVA_HOME}" != "x" ]; then
+if [ ! -d "${JAVA_HOME}" ]; then
+die "JAVA_HOME is not valid: ${JAVA_HOME}"
+fi
+JAVA="${JAVA_HOME}/bin/java"
+else
+warn "JAVA_HOME not set; results may vary"
+JAVA=$(type java)
+JAVA=$(expr "${JAVA}" : '.* \(/.*\)$')
+if [ "x${JAVA}" = "x" ]; then
+die "java command not found"
+fi
+fi
+fi
+# if command is env, attempt to add more to the classpath
+if [ "$1" = "env" ]; then
+[ "x${TOOLS_JAR}" =  "x" ] && [ -n "${JAVA_HOME}" ] && 
TOOLS_JAR=$(find -H "${JAVA_HOME}" -name "tools.jar")
+[ "x${TOOLS_JAR}" =  "x" ] && [ -n "${JAVA_HOME}" ] && 
TOOLS_JAR=$(find -H "${JAVA_HOME}" -name "classes.jar")
+if [ "x${TOOLS_JAR}" =  "x" ]; then
+ warn "Could not locate tools.jar or classes.jar. Please set 
manually to avail all command features."
+fi
+fi
--- End diff --

Is this block left over from the minifi.sh? I tried running it and got this:
Joseph-Percivall:bin jpercivall$ ./config.sh env

Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home
MiNiFi Toolkit home: 
/Users/jpercivall/projects/edge/nifi-minifi/minifi-toolkit/minifi-toolkit-assembly/target/minifi-0.0.1-SNAPSHOT-bin/minifi-toolkit-0.0.1-SNAPSHOT


Usage:

java org.apache.nifi.minifi.toolkit.configuration.ConfigMain  
options

Valid commands include:
transform: Transform template xml into MiNiFi config YAML
validate: Validate config YAML



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[brosander]

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69183488
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

Using map supplier to instantiate maps, toMap() is static now, as well as 
the helper methods that don't need the state of the schema to do their job.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69180850
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

Ah very true, ok I'm on board with not making them static.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Question about Mock classes in ControllerServiceInitializer

[Mark Payne]

Joe,

I think the nifi-documentation module is using that to instantiate Processors, 
Controller Services, etc.
so that it can inspect their annotations & call their getPropertyDescriptors() 
methods, etc. when it generates
documentation for the component. Those should not be used for any component 
that is added to the flow.



> On Jun 30, 2016, at 1:32 PM, Joe Skora  wrote:
> 
> Brandon and I have both run into log entries saying something along the
> lines of "o.a.n.d.mock.MockProcessorLogger Shutting down server".
> 
> Checking the code
> ,
> there are references to the MockProcessorLogger and MockConfigurationContext
> in the org.apache.nifi.documentation.init.ControllerServiceInitizer,
> ProcessorInitializer, and ReportingTaskingInitializer classes.
> 
> What are we missing?  Why are there Mock framework classes used in regular
> classes?

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69178841
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

Right the "Validation Issue helper methods" access the validation issues, 
which is something every schema will have, so it's good to keep them 
implemented how they are. 

The "Value Access/Interpretation helper methods" are all methods that do 
not necessarily rely on the internal representation of each schema but instead 
are helper methods for accessing/interpreting values. So they can be static.

For toMap I agree that it's good to encourage the use of LinkedHashMap. 
Maybe a protected method getMap for the schemas to use. Then toMap would be 
abstract.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[brosander]

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69177842
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

So those helper methods do access the internal state (validation issues).  
I can tweak them to take that as an argument but it seems like leaving them as 
instance methods makes sense for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[brosander]

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69166911
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java
 ---
@@ -15,13 +15,17 @@
  * limitations under the License.
  */
 
-package org.apache.nifi.minifi.bootstrap.util.schema.common;
+package org.apache.nifi.minifi.commons.schema;
 
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 public class BaseSchema {
--- End diff --

So I agree that the toMap method is a little light atm, the one thing I 
liked about defining it in the superclass was to centralize the decision to use 
a LinkedHashMap (to preserve insertion order and allow YAML to have attributes 
in the desired order).

I figured that it was possible we would want to add common attributes at 
the top level at some point in which case that would be a good place for it.

In terms of the helper methods, I don't have a problem with making them 
static.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[brosander]

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69165446
  
--- Diff: minifi-toolkit/minifi-toolkit-assembly/NOTICE ---
@@ -0,0 +1,21 @@
+Apache NiFi
+Copyright 2014-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===
+Apache Software License v2
+===
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Apache NiFi
+The following NOTICE information applies:
+  Apache NiFi
+  Copyright 2014-2016 The Apache Software Foundation
+
+  (ASLv2) Jetty
+The following NOTICE information applies:
+   Jetty Web Container
--- End diff --

Nope, removing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[brosander]

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69164484
  
--- Diff: minifi-commons/minifi-commons-schema/pom.xml ---
@@ -0,0 +1,46 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.nifi.minifi
+minifi-commons
+0.0.1-SNAPSHOT
+
+
+minifi-commons-schema
+jar
+
+
+
+org.yaml
+snakeyaml
+1.17
+
+
+org.apache.nifi
+nifi-framework-core
+0.6.0
+
+
--- End diff --

Good point, removed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69162642
  
--- Diff: minifi-toolkit/minifi-toolkit-assembly/NOTICE ---
@@ -0,0 +1,21 @@
+Apache NiFi
+Copyright 2014-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===
+Apache Software License v2
+===
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Apache NiFi
+The following NOTICE information applies:
+  Apache NiFi
+  Copyright 2014-2016 The Apache Software Foundation
+
+  (ASLv2) Jetty
+The following NOTICE information applies:
+   Jetty Web Container
--- End diff --

Is Jetty packaged in the toolkit-assembly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/21#discussion_r69159093
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/SchemaLoader.java
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.schema;
+
+import 
org.apache.nifi.minifi.commons.schema.exception.SchemaLoaderException;
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.error.YAMLException;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+public class SchemaLoader {
+public static Map loadYamlAsMap(InputStream 
sourceStream) throws IOException, SchemaLoaderException {
+try {
+Yaml yaml = new Yaml();
+
+// Parse the YAML file
+final Object loadedObject = yaml.load(sourceStream);
+
+// Verify the parsed object is a Map structure
+if (loadedObject instanceof Map) {
+return (Map) loadedObject;
+} else {
+throw new SchemaLoaderException("Provided YAML 
configuration is not a Map");
+}
+} catch (YAMLException e ) {
+throw new IOException(e);
+} finally {
+sourceStream.close();
+}
+}
+
+public static ConfigSchema loadConfigSchema(InputStream sourceStream) 
throws IOException, SchemaLoaderException {
--- End diff --

Since this is just a wrapper around loadYamlAsMap and has the requirement 
that the input is Yaml, I'd prefer to have "Yaml" in somewhere in the method 
name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #595: pull new code

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/595
  
This PR appears to be a collection of merge commits, were you trying to 
issue a PR with new code against the master branch? If so can you point me at 
the commit, I can help get the PR into the correct form. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---