[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
@olegz what about using the same strategy as PutEmail where the user can 
configure the socket on runtime?


https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutEmail.java#L122


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
@olegz agreed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72592085
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractEnrichProcessor extends AbstractProcessor {
+public static final PropertyDescriptor QUERY_INPUT = new 
PropertyDescriptor.Builder()
+.name("Format the query should be to be executed ")
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72592665
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractEnrichProcessor extends AbstractProcessor {
+public static final PropertyDescriptor QUERY_INPUT = new 
PropertyDescriptor.Builder()
+.name("Format the query should be to be executed ")
+.required(true)
+.description("The value that should be used to populate the 
query")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final AllowableValue SPLIT= new AllowableValue("Split", 
"Split",
+"String.splitUse a delimiter character or RegEx  to split the 
results into attributes");
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72592679
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractEnrichProcessor extends AbstractProcessor {
+public static final PropertyDescriptor QUERY_INPUT = new 
PropertyDescriptor.Builder()
+.name("Format the query should be to be executed ")
+.required(true)
+.description("The value that should be used to populate the 
query")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final AllowableValue SPLIT= new AllowableValue("Split", 
"Split",
+"String.splitUse a delimiter character or RegEx  to split the 
results into attributes");
+public static final AllowableValue REGEX = new AllowableValue("RegEx", 
"RegEx",
+"Use a regular expression to split the results into attributes 
");
+public static final AllowableValue NONE = new AllowableValue("None", 
"None",
+"Do not split results");
+
+public static final PropertyDescriptor QUERY_PARSER = new 
PropertyDescriptor.Builder()
+.name("Results Parser")
+.description("The method used to slice the results into 
attribute groups")
+.allowableValues(SPLIT, REGEX, NONE)
+.required(true)
+.defaultValue(NONE.getValue())
+.build();
+
+public static final PropertyDescriptor QUERY_PARSER_INPUT = new 
PropertyDescriptor.Builder()
+.name("Parser RegEx")
+.description("Choice between a splitter and regex matcher used 
to parse the results of the query into attribute groups")
+.expressionLanguageSupported(false)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+
+public static final Relationship REL_FOUND = new Relationship.Builder()
+.name("found")
+.description("Where to route flow files after successfully 
enriching attributes with data")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder()
+.name("not found")
+.description("Where to route flow if data enrichment query 
rendered no results")
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72593682
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractEnrichProcessor extends AbstractProcessor {
+public static final PropertyDescriptor QUERY_INPUT = new 
PropertyDescriptor.Builder()
+.name("Format the query should be to be executed ")
+.required(true)
+.description("The value that should be used to populate the 
query")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final AllowableValue SPLIT= new AllowableValue("Split", 
"Split",
+"String.splitUse a delimiter character or RegEx  to split the 
results into attributes");
+public static final AllowableValue REGEX = new AllowableValue("RegEx", 
"RegEx",
+"Use a regular expression to split the results into attributes 
");
+public static final AllowableValue NONE = new AllowableValue("None", 
"None",
+"Do not split results");
+
+public static final PropertyDescriptor QUERY_PARSER = new 
PropertyDescriptor.Builder()
+.name("Results Parser")
+.description("The method used to slice the results into 
attribute groups")
+.allowableValues(SPLIT, REGEX, NONE)
+.required(true)
+.defaultValue(NONE.getValue())
+.build();
+
+public static final PropertyDescriptor QUERY_PARSER_INPUT = new 
PropertyDescriptor.Builder()
+.name("Parser RegEx")
+.description("Choice between a splitter and regex matcher used 
to parse the results of the query into attribute groups")
+.expressionLanguageSupported(false)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+
+public static final Relationship REL_FOUND = new Relationship.Builder()
+.name("found")
+.description("Where to route flow files after successfully 
enriching attributes with data")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder()
+.name("not found")
+.description("Where to route flow if data enrichment query 
rendered no results")
+.build();
+
+
+@Override
+public List customValidate(ValidationContext 
validationContext) {
+final List results = new 
ArrayList<>(super.customValidate(validationContext));
+
+final String chosenQUERY_PARSER = 
validationContext.getProperty(QUERY_PARSER).getValue();
+final boolean QUERY_PARSER_INPUT_isSet = 
validationContext.getProperty(QUERY_PARSER_INPUT).isSet();
+
+if ((!chosenQUERY_PARSER.equals(NONE.getValue()) ) &a

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72596621
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+

[GitHub] nifi issue #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/496
  
@pvillard31 rebase complete and hopefully all issues were addressed. 

Let me know if I missed something


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72596717
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72596817
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72595518
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/test/java/org/apache/nifi/processors/enrich/TestQueryDNS.java
 ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+
+import javax.naming.Context;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttribute;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+
+import static org.junit.Assert.assertTrue;
+
+public class TestQueryDNS  {
+private QueryDNS queryDNS;
+private TestRunner queryDNSTestRunner;
+
+@Before
+public void setupTest() throws Exception {
+this.queryDNS =  new QueryDNS();
+this.queryDNSTestRunner = TestRunners.newTestRunner(queryDNS);
+
+Hashtable env = new Hashtable<String, String>();
+env.put(Context.INITIAL_CONTEXT_FACTORY, 
FakeDNSInitialDirContextFactory.class.getName());
+
+this.queryDNS.initializeContext(env);
+
+final DirContext mockContext = 
FakeDNSInitialDirContextFactory.getLatestMockContext();
+
+// Capture JNDI's getAttibutes method containing the (String) 
queryValue and (String[]) queryType
+Mockito.when( mockContext.getAttributes(Mockito.anyString(), 
Mockito.any(String[].class)))
+.thenAnswer(new Answer() {
+public Object answer(InvocationOnMock invocation) 
throws Throwable {
+// Craft a false DNS response
+// Note the DNS response will not make use of any 
of the mocked
+// query contents (all input is discarded and 
replies synthetically
+// generated
+return craftResponse(invocation);
+}
+});
+}
+
+@Test
+public void testVanillaQueryWithoutSplit()  {
+queryDNSTestRunner.setProperty(QueryDNS.DNS_QUERY_TYPE, "PTR");
+queryDNSTestRunner.setProperty(QueryDNS.DNS_RETRIES, "1");
+queryDNSTestRunner.setProperty(QueryDNS.DNS_TIMEOUT, "1000");
+queryDNSTestRunner.setProperty(QueryDNS.QUERY_INPUT, 
"${ip_address:getDelimitedField(4, '.'):trim()}" +
+".${ip_address:getDelimitedField(3, '.'):trim()}" +
+".${ip_address:getDelimitedField(2, '.'):trim()}" +
+".${ip_address:getDelimitedField(1, '.'):trim()}" +
+".in-addr.arpa");
+queryDNSTestRunner.setProperty(QueryDNS.QUERY_PARSER, 
QueryDNS.NONE.getValue());
+
+final Map<String, String> attributeMap = new HashMap<>();
+attributeMap.put("ip_address", "123.123.123.123");
+
+queryDNSTestRunner.enqueue(new byte[0], attributeMap);
+queryDNSTestRunner.enqueue("teste teste teste 
chocolate".getBytes());
+
+queryDNSTestRunner.run(1,true, false);
+
+List results = 
queryDNSTestRunner.getFlowFilesForRelationship(QueryDNS.REL_FOUND);
+assertTrue(results.size() == 1);
+String result = 
results.get(0).getAttribute("enrich.dns.record0.group0"

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72597016
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+

[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
@olegz, seems like an option. How about adding an additional help file 
withbthe basics of what you already found? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72882149
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72882132
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74418332
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -106,26 +69,17 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_HOSTNAME = new 
PropertyDescriptor.Builder()
-.name("SMTP_HOSTNAME")
-.displayName("SMTP hostname")
-.description("The hostname to be embedded into the banner 
displayed when an " +
-"SMTP client connects to the processor TCP port .")
-.required(true)
-.expressionLanguageSupported(false)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.build();
-
-protected static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = 
new PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = new 
PropertyDescriptor.Builder()
 .name("SMTP_MAXIMUM_CONNECTIONS")
 .displayName("Maximum number of SMTP connection")
 .description("The maximum number of simultaneous SMTP 
connections.")
 .required(true)
+.defaultValue("1")
 .expressionLanguageSupported(false)
 .addValidator(StandardValidators.INTEGER_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
--- End diff --

Is SMTP_TIMEOUT being enforced at the DATA stage?

The server seems to hang around the data stage with the connection open no 
matter how long the client idles for.

Typing . and quit do not work

```
$ time telnet 0 2525
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 localhost ESMTP Apache NiFi
helo .
250 localhost
mail from: x
250 Ok
rcpt to: a
250 Ok
data
354 End data with .
H

.
^]
telnet> quit
Connection closed.

real1m24.348s
user0m0.000s
sys 0m0.002s
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74414544
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -166,317 +108,158 @@
 .identifiesControllerService(SSLContextService.class)
 .build();
 
-public static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
 .name("CLIENT_AUTH")
 .displayName("Client Auth")
 .description("The client authentication policy to use for the 
SSL Context. Only used if an SSL Context Service is provided.")
 .required(false)
 .allowableValues(SSLContextService.ClientAuth.NONE.toString(), 
SSLContextService.ClientAuth.REQUIRED.toString())
 .build();
 
-@Override
-protected Collection customValidate(final 
ValidationContext validationContext) {
-final List results = new ArrayList<>();
+static final Relationship REL_SUCCESS = new Relationship.Builder()
+.name("success")
+.description("All new messages will be routed as FlowFiles to 
this relationship")
+.build();
 
-final String clientAuth = 
validationContext.getProperty(CLIENT_AUTH).getValue();
-final SSLContextService sslContextService = 
validationContext.getProperty(SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+private final static List propertyDescriptors;
 
-if (sslContextService != null && StringUtils.isBlank(clientAuth)) {
-results.add(new ValidationResult.Builder()
-.explanation("Client Auth must be provided when using 
TLS/SSL")
-.valid(false).subject("Client Auth").build());
-}
+private final static Set relationships;
 
-return results;
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.add(SMTP_PORT);
+_propertyDescriptors.add(SMTP_MAXIMUM_CONNECTIONS);
+_propertyDescriptors.add(SMTP_TIMEOUT);
+_propertyDescriptors.add(SMTP_MAXIMUM_MSG_SIZE);
+_propertyDescriptors.add(SSL_CONTEXT_SERVICE);
+_propertyDescriptors.add(CLIENT_AUTH);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
 
+Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+relationships = Collections.unmodifiableSet(_relationships);
 }
 
+private volatile SMTPServer smtp;
 
-public static final Relationship REL_SUCCESS = new 
Relationship.Builder()
-.name("success")
-.description("Extraction was successful")
-.build();
+private volatile SmtpConsumer smtpConsumer;
 
-private Set relationships;
-private List propertyDescriptors;
-private volatile LinkedBlockingQueue incomingMessages;
+/**
+ *
+ */
+@Override
+public void onTrigger(ProcessContext context, ProcessSessionFactory 
sessionFactory) throws ProcessException {
+ProcessSession processSession = sessionFactory.createSession();
+if (this.smtp == null) {
+this.setupSmtpIfNecessary(context, processSession);
+}
+
+if (this.smtpConsumer.hasMessage()) {
+try {
+/*
+ * Will consume incoming message directly from the wire 
and into
+ * FlowFile/Content repository before exiting. This 
essentially
+ * limits any potential data loss by allowing SMTPServer 
thread
+ * to actually commit NiFi session if all good. However in 
the
+ * event of exception, such exception will be propagated 
back to
+ * the email sender via "undeliverable message" allowing 
such
+ * user to re-send the message
+ */
+this.smtpConsumer.consumeUsing((inputDataStream) -> {
+FlowFile flowFile = processSession.create();
+AtomicInteger size = new AtomicInteger();
+flowFile = processSession.write(flowFile, new 
OutputStreamCallback() {
+@Override
+public void 

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74416389
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -135,29 +89,17 @@
 .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_MAXIMUM_MSG_SIZE = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_MSG_SIZE = new 
PropertyDescriptor.Builder()
--- End diff --

Is SMTP_MAXIMUM_SIZE being enforced? I've set it to 512 B and ended up with 
a message of 1KB?

Note that setMaxMessageSize has no effective role in dropping messages, 
instead all it does is to configure the EHLO response:

```
$ telnet 0 2525
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 localhost ESMTP Apache NiFi
EHLO .
250-localhost
250-8BITMIME
250-SIZE 512
250 Ok

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74421721
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -106,26 +69,17 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_HOSTNAME = new 
PropertyDescriptor.Builder()
-.name("SMTP_HOSTNAME")
-.displayName("SMTP hostname")
-.description("The hostname to be embedded into the banner 
displayed when an " +
-"SMTP client connects to the processor TCP port .")
-.required(true)
-.expressionLanguageSupported(false)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.build();
-
-protected static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = 
new PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = new 
PropertyDescriptor.Builder()
 .name("SMTP_MAXIMUM_CONNECTIONS")
 .displayName("Maximum number of SMTP connection")
 .description("The maximum number of simultaneous SMTP 
connections.")
 .required(true)
+.defaultValue("1")
 .expressionLanguageSupported(false)
 .addValidator(StandardValidators.INTEGER_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
--- End diff --

It will cause clients to hang around indefinitely 

Subethasmtp is a weird API. I don't truly understand the reason why it was 
coded like that but it transfers good chunks of the smtp conversation to the 
developer making use of it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74414087
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -13,89 +13,52 @@
  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
  *  See the License for the specific language governing permissions and
  *  limitations under the License.
-*/
+ */
 package org.apache.nifi.processors.email;
 
-import javax.net.ssl.SSLContext;
-import javax.net.ssl.SSLSocket;
-import javax.net.ssl.SSLSocketFactory;
 import java.io.IOException;
-import java.io.InputStream;
+import java.io.OutputStream;
 import java.net.InetSocketAddress;
 import java.net.Socket;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.Collections;
-import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
-import java.util.Map;
 import java.util.Set;
-import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-import org.apache.commons.lang3.StringUtils;
-
-import org.subethamail.smtp.server.SMTPServer;
-
-
-import org.apache.nifi.annotation.lifecycle.OnStopped;
-import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
-import org.apache.nifi.flowfile.attributes.CoreAttributes;
-import org.apache.nifi.processor.DataUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 
-import org.apache.nifi.annotation.lifecycle.OnScheduled;
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessorInitializationContext;
-import org.apache.nifi.processor.Relationship;
-import org.apache.nifi.processor.util.StandardValidators;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
 
+import org.apache.commons.io.IOUtils;
 import org.apache.nifi.annotation.behavior.InputRequirement;
-import org.apache.nifi.annotation.behavior.WritesAttribute;
-import org.apache.nifi.annotation.behavior.WritesAttributes;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.DataUnit;
 import org.apache.nifi.processor.ProcessContext;
 import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
-import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
 import org.apache.nifi.ssl.SSLContextService;
-
-import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
-import org.apache.nifi.processors.email.smtp.handler.SMTPResultCode;
-import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
 
 @Tags({"listen", "email", "smtp"})
 @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
-"allowing nifi to listen for incoming email. " +
-"" +
-"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
-"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
-@WritesAttributes({
-@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
-"certificates used by an TLS peer"),
-@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
-

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74420800
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -135,29 +89,17 @@
 .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_MAXIMUM_MSG_SIZE = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_MSG_SIZE = new 
PropertyDescriptor.Builder()
--- End diff --

I imagine you write straight to disk? If so we should enforce this somehow. 
It would be sort of concerning to have a client filling up your disk with a 
single message? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74425136
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -106,26 +69,17 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_HOSTNAME = new 
PropertyDescriptor.Builder()
-.name("SMTP_HOSTNAME")
-.displayName("SMTP hostname")
-.description("The hostname to be embedded into the banner 
displayed when an " +
-"SMTP client connects to the processor TCP port .")
-.required(true)
-.expressionLanguageSupported(false)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.build();
-
-protected static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = 
new PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = new 
PropertyDescriptor.Builder()
 .name("SMTP_MAXIMUM_CONNECTIONS")
 .displayName("Maximum number of SMTP connection")
 .description("The maximum number of simultaneous SMTP 
connections.")
 .required(true)
+.defaultValue("1")
 .expressionLanguageSupported(false)
 .addValidator(StandardValidators.INTEGER_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
--- End diff --

If you are intending to speak SMTP tgan you need to follow the relevant 
RFC, this means a connecting MTA should be actively informed of a time out 
otherwise it will indefinitely hang connected as you do not return a 4XX code 
nor an ok,  instead we just leave the  connection for dead. This means all 
clients will have to wait for their own timeouts before disconnecting 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74427203
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -106,26 +69,17 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_HOSTNAME = new 
PropertyDescriptor.Builder()
-.name("SMTP_HOSTNAME")
-.displayName("SMTP hostname")
-.description("The hostname to be embedded into the banner 
displayed when an " +
-"SMTP client connects to the processor TCP port .")
-.required(true)
-.expressionLanguageSupported(false)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.build();
-
-protected static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = 
new PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = new 
PropertyDescriptor.Builder()
 .name("SMTP_MAXIMUM_CONNECTIONS")
 .displayName("Maximum number of SMTP connection")
 .description("The maximum number of simultaneous SMTP 
connections.")
 .required(true)
+.defaultValue("1")
 .expressionLanguageSupported(false)
 .addValidator(StandardValidators.INTEGER_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_TIMEOUT = new 
PropertyDescriptor.Builder()
--- End diff --

Agree with that. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74423248
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -135,29 +89,17 @@
 .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_MAXIMUM_MSG_SIZE = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_MSG_SIZE = new 
PropertyDescriptor.Builder()
--- End diff --

@olegz I see where you come from but put file is controlled by the DFM, 
ListenSMTP is an exposed service

True, we don't do it everywhere but it is present on GetFile as well as 
other processors crossing administrative domains (DFM <> FS) and so it goes



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/827
  
@olegz You are clearly confusing envelope (the data exchanged within an 
SMTP session) with header information (the data added to the body of the 
message after the DATA command). 

They don't need to match. 

MAIL FROM is an envelope detail it is NOT added to a message. Please refer 
back to this comment:

https://github.com/apache/nifi/pull/827#discussion_r74362397

Note how although MAIL FROM was set to a...@aaa.com this information never 
makes into the resulting flowfile. 

The RCPT TO input makes to it but requires a multiline regex to parse it 
and that regex will be quite brittle as resolution will require a IPv4 / IPv6 
match, variable domains and so it goes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #817: NIFI-2380 - Introduce ExtractTNEFAttachments

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/817
  
@JPercivall - Given your review of ExtractEmailAttachments do you mind 
reviewing this PR? Cheers


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #817: NIFI-2380 - Introduce ExtractTNEFAttachments

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/817
  
@joewitt - 3rd party test files have been replaced with new files 
containing only NiFi & MiNiFi logos


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/827
  
Stopping the processor with an agent connected also seems to throw an 
exception (not sure if expected)

```
2016-08-12 01:42:06,593 INFO [StandardProcessScheduler Thread-7] 
org.subethamail.smtp.server.SMTPServer SMTP server *:2525 stopping
2016-08-12 01:42:06,594 INFO [org.subethamail.smtp.server.ServerThread 
*:2525] org.subethamail.smtp.server.ServerThread SMTP server *:2525 stopped
2016-08-12 01:42:06,596 ERROR 
[org.subethamail.smtp.server.Session-/127.0.0.1:38072] 
org.subethamail.smtp.server.Session Unexpected error in the SMTP handler thread
java.lang.IllegalStateException: NIFI Consumer was stopped before message 
was successfully consumed by NiFi
at 
org.apache.nifi.processors.email.SmtpConsumer.data(SmtpConsumer.java:125) 
~[na:na]
at 
org.subethamail.smtp.command.DataCommand.execute(DataCommand.java:64) ~[na:na]
at 
org.subethamail.smtp.server.RequireTLSCommandWrapper.execute(RequireTLSCommandWrapper.java:30)
 ~[na:na]
at 
org.subethamail.smtp.server.CommandHandler.handleCommand(CommandHandler.java:99)
 ~[na:na]
at 
org.subethamail.smtp.server.Session.runCommandLoop(Session.java:244) ~[na:na]
at org.subethamail.smtp.server.Session.run(Session.java:145) 
~[na:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
2016-08-12 01:42:06,597 ERROR [pool-33-thread-1] org.apache.nifi.NiFi An 
Unknown Error Occurred in Thread Thread[pool-33-thread-1,5,main]: 
java.lang.IllegalStateException: NIFI Consumer was stopped before message was 
successfully consumed by NiFi
2016-08-12 01:42:06,599 ERROR [pool-33-thread-1] org.apache.nifi.NiFi
java.lang.IllegalStateException: NIFI Consumer was stopped before message 
was successfully consumed by NiFi
at 
org.apache.nifi.processors.email.SmtpConsumer.data(SmtpConsumer.java:125) 
~[na:na]
at 
org.subethamail.smtp.command.DataCommand.execute(DataCommand.java:64) ~[na:na]
at 
org.subethamail.smtp.server.RequireTLSCommandWrapper.execute(RequireTLSCommandWrapper.java:30)
 ~[na:na]
at 
org.subethamail.smtp.server.CommandHandler.handleCommand(CommandHandler.java:99)
 ~[na:na]
at 
org.subethamail.smtp.server.Session.runCommandLoop(Session.java:244) ~[na:na]
at org.subethamail.smtp.server.Session.run(Session.java:145) 
~[na:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/827
  
@olegz have you pushed the notification recently? I got just an abrupt 
termination

```
$ telnet 0 2525
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 localhost ESMTP Apache NiFi
helo .
250 localhost
mail from: x
250 Ok
rcpt to: y
250 Ok
data
354 End data with .
waiting for server to get bored
Connection closed by foreign host.
```

Note the connection close as soon I stop the server


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/827
  
1. Start ListenSMTP port 2525 with 1 maximum connection, 10 seconds of 
timeout

```
$ telnet 0 2525
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 localhost ESMTP Apache NiFi
helo .
250 localhost
mail from: x
250 Ok
rcpt to: x
250 Ok
data
354 End data with .
```


wait for longer than timeout


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74359964
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -13,89 +13,52 @@
  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
  *  See the License for the specific language governing permissions and
  *  limitations under the License.
-*/
+ */
 package org.apache.nifi.processors.email;
 
-import javax.net.ssl.SSLContext;
-import javax.net.ssl.SSLSocket;
-import javax.net.ssl.SSLSocketFactory;
 import java.io.IOException;
-import java.io.InputStream;
+import java.io.OutputStream;
 import java.net.InetSocketAddress;
 import java.net.Socket;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.Collections;
-import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
-import java.util.Map;
 import java.util.Set;
-import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-import org.apache.commons.lang3.StringUtils;
-
-import org.subethamail.smtp.server.SMTPServer;
-
-
-import org.apache.nifi.annotation.lifecycle.OnStopped;
-import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
-import org.apache.nifi.flowfile.attributes.CoreAttributes;
-import org.apache.nifi.processor.DataUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 
-import org.apache.nifi.annotation.lifecycle.OnScheduled;
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessorInitializationContext;
-import org.apache.nifi.processor.Relationship;
-import org.apache.nifi.processor.util.StandardValidators;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
 
+import org.apache.commons.io.IOUtils;
 import org.apache.nifi.annotation.behavior.InputRequirement;
-import org.apache.nifi.annotation.behavior.WritesAttribute;
-import org.apache.nifi.annotation.behavior.WritesAttributes;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.DataUnit;
 import org.apache.nifi.processor.ProcessContext;
 import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
-import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
 import org.apache.nifi.ssl.SSLContextService;
-
-import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
-import org.apache.nifi.processors.email.smtp.handler.SMTPResultCode;
-import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
 
 @Tags({"listen", "email", "smtp"})
 @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
-"allowing nifi to listen for incoming email. " +
-"" +
-"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
-"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
-@WritesAttributes({
-@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
-"certificates used by an TLS peer"),
-@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
-

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74361191
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -106,26 +69,17 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_HOSTNAME = new 
PropertyDescriptor.Builder()
-.name("SMTP_HOSTNAME")
--- End diff --

Without this SubEtha seems to default to localhost

```
$ telnet 10.0.2.15 2525
Trying 10.0.2.15...
Connected to 10.0.2.15.
Escape character is '^]'.
220 localhost ESMTP Apache NiFi

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #817: NIFI-2380 - Introduce ExtractTNEFAttachments

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/817
  
@joewitt 

I will need some assistance with the L issue:

Note how POIs LICENSE makes mention to CurvesAPI
https://github.com/apache/poi/blob/trunk/legal/LICENSE#L459

Looking at CurvesAPI, the authors seem to require notice
https://github.com/virtuald/curvesapi/blob/master/license.txt#L8

However POIs NOTICE does not make mention to CurvesAPI
https://github.com/apache/poi/blob/trunk/legal/NOTICE#L5

Is NOTICE required (despite POI lacking it) or can we address that by 
appending nifi-assembly and the nar LICENSE files with: 

```
This project bundles 'CurvesAPI' which is available under a BSD license.

Copyright (c) 2005, Graph Builder
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

-Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.

-Redistributions in binary form must reproduce the above copyright 
notice,
this list of conditions and the following disclaimer in the 
documentation
and/or other materials provided with the distribution.

-Neither the name of Graph Builder nor the names of its contributors 
may be
used to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 
IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 
THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 
PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 
LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 
OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 
HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 
LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF 
THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74527077
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -106,26 +90,17 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_HOSTNAME = new 
PropertyDescriptor.Builder()
-.name("SMTP_HOSTNAME")
-.displayName("SMTP hostname")
-.description("The hostname to be embedded into the banner 
displayed when an " +
-"SMTP client connects to the processor TCP port .")
-.required(true)
-.expressionLanguageSupported(false)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.build();
-
-protected static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = 
new PropertyDescriptor.Builder()
+static final PropertyDescriptor SMTP_MAXIMUM_CONNECTIONS = new 
PropertyDescriptor.Builder()
--- End diff --

@olegz is SMTP_MAXIMUM_CONNECTIONS being enforced at all? 
```

$ date && telnet 0 2525
Fri Aug 12 09:02:31 AEST 2016
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 localhost ESMTP Apache NiFi
```

$ date && telnet 0 2525
Fri Aug 12 09:02:30 AEST 2016
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 localhost ESMTP Apache NiFi

I did this test and it doesn't seem to be:

```
$ date  && netstat -an| grep 2525 | pcregrep "2525\s+ES"
Fri Aug 12 09:02:33 AEST 2016
tcp0  0 0.0.0.0:25250.0.0.0:*   
LISTEN
tcp0  0 127.0.0.1:38840 127.0.0.1:2525  
ESTABLISHED
tcp0  0 127.0.0.1:38842 127.0.0.1:2525  
ESTABLISHED
```




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #785: NIFI-2341 - Introduce ParseCEF processor

[trixpan]

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/785

NIFI-2341 - Introduce ParseCEF processor

* Implements a processor to parse HPE's highly popular Common Event Format

* Please note this code relies on a Parser written outside the NiFi source 
tree. This was done with the intent of allowing code reuse, given that many 
Open Source projects tend to tackle the CEF format in a myriad of ways and some 
are still trying to implement a solid parser (e.g. METRON-157). The Parser is 
functional but feedback over its implementation is welcome as well.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-2341

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/785.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #785


commit be2c1e83f7f486b1b6a1e148bc828acf3865a8de
Author: Andre F de Miranda <trix...@users.noreply.github.com>
Date:   2016-08-04T13:30:37Z

NIFI-2341 - Introduce ParseCEF processor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #856: NIFI-2519 aligned processor to leverage the subethasmtp thr...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/856
  
:+1:

Tested manually and also forced some exceptions. ListenSMTP seems to handle 
errors very well, example:

Place breakpoint on 

`processSession.getProvenanceReporter().receive(flowFile, "smtp://" + host 
+ ":" + port + "/", watch.getDuration(TimeUnit.MILLISECONDS));
`

```
220 bermudatriangle ESMTP Apache NiFi SMTP
helo ,
250 bermudatriangle
mail from: x
250 Ok
rcpt to: y
250 Ok
data
354 End data with .
a
.
```

evaluate flowFile so that is null. Let process continue. Following message 
is returned.

```
421 4.3.0 Mail system failure, closing transmission channel
Connection closed by foreign host.
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #856: NIFI-2519 aligned processor to leverage the subethas...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/856#discussion_r74700141
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/SmtpConsumer.java
 ---
@@ -0,0 +1,161 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.nifi.processors.email.smtp;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.cert.Certificate;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.FlowFileAccessException;
+import org.apache.nifi.processors.email.ListenSMTP;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+/**
+ * A simple consumer that provides a bridge between 'push' message 
distribution
+ * provided by {@link SMTPServer} and NiFi polling scheduler mechanism.
+ */
+public class SmtpConsumer implements MessageHandler {
+
+private String from = null;
+private final List recipientList = new ArrayList<>();
+private final MessageContext context;
+private final ProcessSessionFactory sessionFactory;
+private final int port;
+private final int maxMessageSize;
+private final ComponentLog log;
+private final String host;
+
+public SmtpConsumer(
+final MessageContext context,
+final ProcessSessionFactory sessionFactory,
+final int port,
+final String host,
+final ComponentLog log,
+final int maxMessageSize
+) {
+this.context = context;
+this.sessionFactory = sessionFactory;
+this.port = port;
+if (host == null || host.trim().isEmpty()) {
+this.host = context.getSMTPServer().getHostName();
+} else {
+this.host = host;
+}
+this.log = log;
+this.maxMessageSize = maxMessageSize;
+}
+
+String getFrom() {
+return from;
+}
+
+List getRecipients() {
+return Collections.unmodifiableList(recipientList);
+}
+
+@Override
+public void data(final InputStream data) throws RejectException, 
TooMuchDataException, IOException {
+final ProcessSession processSession = 
sessionFactory.createSession();
+final StopWatch watch = new StopWatch();
+watch.start();
+try {
+FlowFile flowFile = processSession.create();
+final AtomicBoolean limitExceeded = new AtomicBoolean(false);
+flowFile = processSession.write(flowFile, (OutputStream out) 
-> {
+final LimitingInputStream lis = new 
LimitingInputStream(data, maxMessageSize);
+IOUtils.copy(lis, out);
+if (lis.hasReachedLimit()) {
+limitExceeded.set(true);
+}
+});
+if (limitExceeded.get()) {
+throw new TooMuchDataException("Maximum me

[GitHub] nifi issue #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/827
  
@olegz - Tested the processor and other than the view that attributes 
should not be removed (as they are useful for many auditing and downstream 
applications) and a few remarks on SubEthaSMTP intricacies  I am amazed by the 
refactoring. 

Truly awesome work in here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74361570
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -166,317 +108,158 @@
 .identifiesControllerService(SSLContextService.class)
 .build();
 
-public static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
 .name("CLIENT_AUTH")
 .displayName("Client Auth")
 .description("The client authentication policy to use for the 
SSL Context. Only used if an SSL Context Service is provided.")
 .required(false)
 .allowableValues(SSLContextService.ClientAuth.NONE.toString(), 
SSLContextService.ClientAuth.REQUIRED.toString())
 .build();
 
-@Override
--- End diff --

Although harmless, this should not be a valid configuration:


![image](https://cloud.githubusercontent.com/assets/3108527/17576962/e5e0414a-5fbb-11e6-9050-9234dc1fe7d1.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74362612
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -13,89 +13,52 @@
  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
  *  See the License for the specific language governing permissions and
  *  limitations under the License.
-*/
+ */
 package org.apache.nifi.processors.email;
 
-import javax.net.ssl.SSLContext;
-import javax.net.ssl.SSLSocket;
-import javax.net.ssl.SSLSocketFactory;
 import java.io.IOException;
-import java.io.InputStream;
+import java.io.OutputStream;
 import java.net.InetSocketAddress;
 import java.net.Socket;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.Collections;
-import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
-import java.util.Map;
 import java.util.Set;
-import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-import org.apache.commons.lang3.StringUtils;
-
-import org.subethamail.smtp.server.SMTPServer;
-
-
-import org.apache.nifi.annotation.lifecycle.OnStopped;
-import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
-import org.apache.nifi.flowfile.attributes.CoreAttributes;
-import org.apache.nifi.processor.DataUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 
-import org.apache.nifi.annotation.lifecycle.OnScheduled;
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessorInitializationContext;
-import org.apache.nifi.processor.Relationship;
-import org.apache.nifi.processor.util.StandardValidators;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
 
+import org.apache.commons.io.IOUtils;
 import org.apache.nifi.annotation.behavior.InputRequirement;
-import org.apache.nifi.annotation.behavior.WritesAttribute;
-import org.apache.nifi.annotation.behavior.WritesAttributes;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.DataUnit;
 import org.apache.nifi.processor.ProcessContext;
 import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
-import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
 import org.apache.nifi.ssl.SSLContextService;
-
-import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
-import org.apache.nifi.processors.email.smtp.handler.SMTPResultCode;
-import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
 
 @Tags({"listen", "email", "smtp"})
 @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
-"allowing nifi to listen for incoming email. " +
-"" +
-"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
-"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
-@WritesAttributes({
-@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
-"certificates used by an TLS peer"),
-@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
-

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74362680
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -106,26 +69,17 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
-protected static final PropertyDescriptor SMTP_HOSTNAME = new 
PropertyDescriptor.Builder()
-.name("SMTP_HOSTNAME")
--- End diff --

Because I may have a system using AWS's weird naming structures but look to 
display a fancy name like mx.trixpanmailservices.com




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74362949
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -13,89 +13,52 @@
  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
  *  See the License for the specific language governing permissions and
  *  limitations under the License.
-*/
+ */
 package org.apache.nifi.processors.email;
 
-import javax.net.ssl.SSLContext;
-import javax.net.ssl.SSLSocket;
-import javax.net.ssl.SSLSocketFactory;
 import java.io.IOException;
-import java.io.InputStream;
+import java.io.OutputStream;
 import java.net.InetSocketAddress;
 import java.net.Socket;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.Collections;
-import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
-import java.util.Map;
 import java.util.Set;
-import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-import org.apache.commons.lang3.StringUtils;
-
-import org.subethamail.smtp.server.SMTPServer;
-
-
-import org.apache.nifi.annotation.lifecycle.OnStopped;
-import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
-import org.apache.nifi.flowfile.attributes.CoreAttributes;
-import org.apache.nifi.processor.DataUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 
-import org.apache.nifi.annotation.lifecycle.OnScheduled;
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessorInitializationContext;
-import org.apache.nifi.processor.Relationship;
-import org.apache.nifi.processor.util.StandardValidators;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
 
+import org.apache.commons.io.IOUtils;
 import org.apache.nifi.annotation.behavior.InputRequirement;
-import org.apache.nifi.annotation.behavior.WritesAttribute;
-import org.apache.nifi.annotation.behavior.WritesAttributes;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.DataUnit;
 import org.apache.nifi.processor.ProcessContext;
 import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
-import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
 import org.apache.nifi.ssl.SSLContextService;
-
-import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
-import org.apache.nifi.processors.email.smtp.handler.SMTPResultCode;
-import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
 
 @Tags({"listen", "email", "smtp"})
 @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
-"allowing nifi to listen for incoming email. " +
-"" +
-"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
-"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
-@WritesAttributes({
-@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
-@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
-"certificates used by an TLS peer"),
-@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
-

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74363458
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -166,317 +108,158 @@
 .identifiesControllerService(SSLContextService.class)
 .build();
 
-public static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
+static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
 .name("CLIENT_AUTH")
 .displayName("Client Auth")
 .description("The client authentication policy to use for the 
SSL Context. Only used if an SSL Context Service is provided.")
 .required(false)
 .allowableValues(SSLContextService.ClientAuth.NONE.toString(), 
SSLContextService.ClientAuth.REQUIRED.toString())
 .build();
 
-@Override
-protected Collection customValidate(final 
ValidationContext validationContext) {
-final List results = new ArrayList<>();
+static final Relationship REL_SUCCESS = new Relationship.Builder()
+.name("success")
+.description("All new messages will be routed as FlowFiles to 
this relationship")
+.build();
 
-final String clientAuth = 
validationContext.getProperty(CLIENT_AUTH).getValue();
-final SSLContextService sslContextService = 
validationContext.getProperty(SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+private final static List propertyDescriptors;
 
-if (sslContextService != null && StringUtils.isBlank(clientAuth)) {
-results.add(new ValidationResult.Builder()
-.explanation("Client Auth must be provided when using 
TLS/SSL")
-.valid(false).subject("Client Auth").build());
-}
+private final static Set relationships;
 
-return results;
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.add(SMTP_PORT);
+_propertyDescriptors.add(SMTP_MAXIMUM_CONNECTIONS);
+_propertyDescriptors.add(SMTP_TIMEOUT);
+_propertyDescriptors.add(SMTP_MAXIMUM_MSG_SIZE);
+_propertyDescriptors.add(SSL_CONTEXT_SERVICE);
+_propertyDescriptors.add(CLIENT_AUTH);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
 
+Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+relationships = Collections.unmodifiableSet(_relationships);
 }
 
+private volatile SMTPServer smtp;
 
-public static final Relationship REL_SUCCESS = new 
Relationship.Builder()
-.name("success")
-.description("Extraction was successful")
-.build();
+private volatile SmtpConsumer smtpConsumer;
 
-private Set relationships;
-private List propertyDescriptors;
-private volatile LinkedBlockingQueue incomingMessages;
+/**
+ *
+ */
+@Override
+public void onTrigger(ProcessContext context, ProcessSessionFactory 
sessionFactory) throws ProcessException {
+ProcessSession processSession = sessionFactory.createSession();
+if (this.smtp == null) {
+this.setupSmtpIfNecessary(context, processSession);
+}
+
+if (this.smtpConsumer.hasMessage()) {
+try {
+/*
+ * Will consume incoming message directly from the wire 
and into
+ * FlowFile/Content repository before exiting. This 
essentially
+ * limits any potential data loss by allowing SMTPServer 
thread
+ * to actually commit NiFi session if all good. However in 
the
+ * event of exception, such exception will be propagated 
back to
+ * the email sender via "undeliverable message" allowing 
such
+ * user to re-send the message
+ */
+this.smtpConsumer.consumeUsing((inputDataStream) -> {
+FlowFile flowFile = processSession.create();
+AtomicInteger size = new AtomicInteger();
+flowFile = processSession.write(flowFile, new 
OutputStreamCallback() {
+@Override
+public void 

[GitHub] nifi pull request #827: NIFI-2519 Fixed and refactored ListenSMTP processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/827#discussion_r74364045
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/test/java/org/apache/nifi/processors/email/TestListenSMTP.java
 ---
@@ -13,307 +13,174 @@
  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
  *  See the License for the specific language governing permissions and
  *  limitations under the License.
-*/
+ */
 package org.apache.nifi.processors.email;
 
+import static org.junit.Assert.assertTrue;
+
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
 import org.apache.commons.mail.Email;
 import org.apache.commons.mail.EmailException;
 import org.apache.commons.mail.SimpleEmail;
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.remote.io.socket.NetworkUtils;
+import org.apache.nifi.ssl.SSLContextService;
 import org.apache.nifi.ssl.StandardSSLContextService;
-import org.apache.nifi.util.MockFlowFile;
 import org.apache.nifi.util.TestRunner;
 import org.apache.nifi.util.TestRunners;
-
-import org.apache.nifi.ssl.SSLContextService;
-
-import org.junit.Assert;
+import org.junit.After;
+import org.junit.Before;
 import org.junit.Test;
 
-import java.util.List;
-import java.util.concurrent.atomic.AtomicBoolean;
-
 public class TestListenSMTP {
 
-@Test(timeout=15000)
-public void ValidEmailTls() throws Exception {
-boolean[] failed = {false};
-ListenSMTP listenSmtp = new ListenSMTP();
-final TestRunner runner = TestRunners.newTestRunner(listenSmtp);
-
-runner.setProperty(ListenSMTP.SMTP_PORT, "0");
-runner.setProperty(ListenSMTP.SMTP_HOSTNAME, "bermudatriangle");
-runner.setProperty(ListenSMTP.SMTP_MAXIMUM_CONNECTIONS, "3");
-runner.setProperty(ListenSMTP.SMTP_TIMEOUT, "10 seconds");
-
-// Setup the SSL Context
-final SSLContextService sslContextService = new 
StandardSSLContextService();
-runner.addControllerService("ssl-context", sslContextService);
-runner.setProperty(sslContextService, 
StandardSSLContextService.TRUSTSTORE, "src/test/resources/localhost-ts.jks");
-runner.setProperty(sslContextService, 
StandardSSLContextService.TRUSTSTORE_PASSWORD, "localtest");
-runner.setProperty(sslContextService, 
StandardSSLContextService.TRUSTSTORE_TYPE, "JKS");
-runner.setProperty(sslContextService, 
StandardSSLContextService.KEYSTORE, "src/test/resources/localhost-ks.jks");
-runner.setProperty(sslContextService, 
StandardSSLContextService.KEYSTORE_PASSWORD, "localtest");
-runner.setProperty(sslContextService, 
StandardSSLContextService.KEYSTORE_TYPE, "JKS");
-runner.enableControllerService(sslContextService);
-
-// and add the SSL context to the runner
-runner.setProperty(ListenSMTP.SSL_CONTEXT_SERVICE, "ssl-context");
-runner.setProperty(ListenSMTP.CLIENT_AUTH, 
SSLContextService.ClientAuth.NONE.name());
-
+private ScheduledExecutorService executor;
 
+/**
+ *
+ */
+@Before
+public void before() {
+this.executor = Executors.newScheduledThreadPool(2);
+}
 
-final ProcessSessionFactory processSessionFactory = 
runner.getProcessSessionFactory();
-final ProcessContext context = runner.getProcessContext();
-
-// NOTE: This test routine uses  the same strategy used by 
TestListenAndPutSyslog
-// where listenSmtp method calls are used to allow the processor 
to be started using
-// port "0" without triggering a violation of PORT_VALIDATOR
-
-listenSmtp.onScheduled(context);
-listenSmtp.initializeSMTPServer(context);
-
-final int port = listenSmtp.getPort();
-
-try {
-final Thread clientThread = new Thread(new Runnable() {
-@Override
-public void run() {
-try {
-
-
-System.setProperty("mail.smtp.ssl.trust", "*");
-System.setProperty("javax.net.ssl.keyStore", 
"src/test/resources/localhost-ks.jks");
-
System.setProperty("javax.net.ssl.keyStorePassword", "localtest"

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70721418
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
+
+
+X509Certificate[] certificates = new X509Certificate[]{};
+
+String remoteIP = messageContext.getRemoteAddress().toString();
+String helo = messageContext.getHelo();
+if (messageContext.getTlsPeerCertificates() != null ){
+certificates = (X509Certificate[]) 
messageContext.getTlsPeerCertificates().clone();
+}
+
+SmtpEvent message = new SmtpEvent(remoteIP, helo, from, 
recipient, certificates, m

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70574245
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+private CountDownLatch latch;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+this.latch =  new CountDownLatch(1);
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+ByteArrayOutputStream messageData;
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(true);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+final long serverTimeout = 
TimeUnit.MILLISECONDS.convert(messageContext.getSMTPServer().getConnectionTimeout(),
 TimeUnit.MILLISECONDS);
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+
+int rd;
+
+try {
+while ((rd = inputStream.read(buffer, 0, buffer.length)) 
!= -1 ) {
+baos.write(buffer, 0, rd);
+if (baos.getBufferLength() > 
server.getMaxMessageSize() ) {
+throw new TooMuchDataException("Data exceeds the 
amount allowed.");
+}
+}
+baos.flush();
+} catch (IOException e) {
+throw new DropConnectionException(450, "Unexpected error 
processing your message. ");
+}
+
+this.messageData = baos;
+
+ 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70714764
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+ByteArrayOutputStream messageData;
+
+private CountDownLatch latch;
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+this.latch =  new CountDownLatch(1);
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(true);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+final long serverTimeout = 
TimeUnit.MILLISECONDS.convert(messageContext.getSMTPServer().getConnectionTimeout(),
 TimeUnit.MILLISECONDS);
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+
+int rd;
+
+try {
+while ((rd = inputStream.read(buffer, 0, buffer.length)) 
!= -1 ) {
+baos.write(buffer, 0, rd);
+if (baos.getBufferLength() > 
server.getMaxMessageSize() ) {
+throw new TooMuchDataException("Data exceeds the 
amount allowed.");
+}
+}
+baos.flush();
+} catch (IOException e) {
+throw new DropConnectionException(450, "Unexpected error 
processing your message. ");
+}
+
+this.messageData = b

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70714800
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+ByteArrayOutputStream messageData;
+
+private CountDownLatch latch;
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+this.latch =  new CountDownLatch(1);
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(true);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+final long serverTimeout = 
TimeUnit.MILLISECONDS.convert(messageContext.getSMTPServer().getConnectionTimeout(),
 TimeUnit.MILLISECONDS);
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+
+int rd;
+
+try {
+while ((rd = inputStream.read(buffer, 0, buffer.length)) 
!= -1 ) {
+baos.write(buffer, 0, rd);
+if (baos.getBufferLength() > 
server.getMaxMessageSize() ) {
+throw new TooMuchDataException("Data exceeds the 
amount allowed.");
+}
+}
+baos.flush();
+} catch (IOException e) {
+throw new DropConnectionException(450, "Unexpected error 
processing your message. ");
+}
+
+this.messageData = b

[GitHub] nifi pull request #475: NIFI-2026 - Add Maven profile to compile nifi-hadoop...

[trixpan]

Github user trixpan closed the pull request at:

https://github.com/apache/nifi/pull/475


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r71655778
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/test/java/org/apache/nifi/processors/TestQueryDNS.java
 ---
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors;
+
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.Test;
+
+
+
+import static org.junit.Assert.assertTrue;
+
+public class TestQueryDNS {
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments and ExtractEm...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/483
  
@JPercivall Beautiful code! Loved it. Interesting to see your programming 
thinking. Great lesson (specially because I tried to achieve a similar - though 
more covoluted - logic previously and failed. heheheh)

It makes total sense how you replaced the latch with a timed wait + 
notifyAll.

The only question that I have is how do you deal with a spurious wakeup in 
this section:

```
 final long serverTimeout = 
context.getProperty(SMTP_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS);
try {
message.wait(serverTimeout);
} catch (InterruptedException e) {
getLogger().info("Interrupted while waiting for Message 
Handler to acknowledge message.");
}
```

and this section:

```
try {
message.wait(serverTimeout - elapsed);
} catch (InterruptedException e) {
// Interrupted while waiting for the message to 
process. Will return error and request onTrigger to rollback
logger.trace("Interrupted while waiting for processor 
to process data. Returned error to SMTP client as precautionary measure");
```

Am I correct to assume you just preferred to ignore them and take a 
decision based on whatever information message object would have gathered by 
that time? Something like: 

if all completed by then, great, message gets acked, commited, client gets 
happy, but if message is partially processed (e.g. no returnCode when it hits 
the first wait above) test with resultCodeSetAndIsError and effectively treat 
it as success(commit) unless an error code is set?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments and ExtractEm...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/483
  
@JPercivall Fixing the rebase. Please hold


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70612760
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
+
+
+X509Certificate[] certificates = new X509Certificate[]{};
+
+String remoteIP = messageContext.getRemoteAddress().toString();
+String helo = messageContext.getHelo();
+if (messageContext.getTlsPeerCertificates() != null ){
+certificates = (X509Certificate[]) 
messageContext.getTlsPeerCertificates().clone();
+}
+
+SmtpEvent message = new SmtpEvent(remoteIP, helo, from, 
recipient, certificates, m

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70612776
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
+
+
+X509Certificate[] certificates = new X509Certificate[]{};
+
+String remoteIP = messageContext.getRemoteAddress().toString();
+String helo = messageContext.getHelo();
+if (messageContext.getTlsPeerCertificates() != null ){
+certificates = (X509Certificate[]) 
messageContext.getTlsPeerCertificates().clone();
+}
+
+SmtpEvent message = new SmtpEvent(remoteIP, helo, from, 
recipient, certificates, m

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70615355
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
--- End diff --

I understood that, however, from what I gather I cannot determine the 
amount of data an InputStream contains or will contain until I read it?

Since we must tell the client he exceeded the limit, I chose to perform the 
reading loop within the handler rather than sending it down the queue to the 
`onTrigger` thread for reading and then returning error from there.

Having said that, I am open to suggestions.


---
If your project is set up for it, you can reply to this email and have your
reply 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70366557
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
+
+
+X509Certificate[] certificates = new X509Certificate[]{};
+
+String remoteIP = messageContext.getRemoteAddress().toString();
+String helo = messageContext.getHelo();
+if (messageContext.getTlsPeerCertificates() != null ){
+certificates = (X509Certificate[]) 
messageContext.getTlsPeerCertificates().clone();
+}
+
+SmtpEvent message = new SmtpEvent(remoteIP, helo, from, 
recipient, certificates, m

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70157262
  
--- Diff: nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/pom.xml 
---
@@ -0,0 +1,75 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-email-bundle
+1.0.0-SNAPSHOT
+
+
+nifi-email-processors
+jar
+
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+javax.mail
+mail
+1.4.7
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70157601
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
+public class ExtractEmailAttachments extends AbstractProcessor {
+public static final String ATTACHMENT_ORIGINAL_FILENAME = 
"email.attachment.parent.filename";
+public static final String ATTACHMENT_ORIGINAL_UUID = 
"email.attachment.parent.uuid";
+
+public static final Relationship REL_ATTACHMENTS = new 
Relationship.Builder()
+.name("attachments")
+.description("Each individual attachment will be routed to the 
attachments relationship")
+.build();
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder()
+.name("original")
+.description("The original file")
+.build();
+public static final Relationship REL_FAILURE = new 
Relationship.Builde

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70157551
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
--- End diff --

I'm fine with the idea however I haven't considered any batching during the 
development of the processor... Would assume it is safer to leave as it is?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70157622
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailHeaders.java
 ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+import javax.mail.Address;
+import javax.mail.Header;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Using the flowfile content as source of data, 
extract header from an RFC  compliant  email file adding the relevant 
attributes to the flowfile. " +
+"This processor does not perform extensive RFC validation but 
still requires a bare minimum compliance with RFC 2822")
+@WritesAttributes({
+@WritesAttribute(attribute = "email.headers.bcc.*", description = 
"Each individual BCC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.cc.*", description = 
"Each individual CC recipient (if available)"),
+@WritesAttribute(attribute = "email.headers.from.*", description = 
"Each individual mailbox contained in the From  of the Email (array as per 
RFC-2822)"),
+@WritesAttribute(attribute = "email.headers.message-id", 
description = "The value of the Message-ID header (if available)"),
+@WritesAttribute(attribute = "email.headers.received_date", 
description = "The Received-Date of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.sent_date", 
description = "Date the message was sent"),
+@WritesAttribute(attribute = "email.headers.subject", description 
= "Subject of the message (if available)"),
+@WritesAttribute(attribute = "email.headers.to.*", description = 
"Each individual TO recipient (if available)"),
+ 

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70180440
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,385 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to listen for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70180458
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import org.apache.commons.io.IOUtils;
+
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue messages;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
messages) {
+this.messages = messages;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, messages);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+boolean failed;
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue messages){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+SMTPServer server = messageContext.getSMTPServer();
+
+if  (inputStream.available() > server.getMaxMessageSize()) {
+failed = true;
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+} else {
+this.messageBody = IOUtils.toByteArray(inputStream);
+}
+}
+
+@Override
+public void done() {
+
+X509Certificate[] certificates = new X509Certificate[]{};
+
+String remoteIP = messageContext.getRemoteAddress().toString();
+String helo = messageContext.getHelo();
+if (messageContext.getTlsPeerCertificates() != null ){
+  certificates = (X509Certificate[]) 
messageContext.getTlsPeerCertificates().clone();
+}
+if (!failed) {
+SmtpEvent message = new SmtpEvent(remoteIP, helo, from, 
recipient, certificates, messageBody);
+try {
+messages.put(message);
+} catch (InterruptedException e) {
+// Perhaps this should be logged?
--- End diff --

correct. I added log but also added a` throw new DropConnectionException` 
so that the client knows it failed and try to resend again later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrast

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70180468
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,385 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to listen for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70180484
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
--- End diff --

I didn't know that! Thanks for the insight.

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70180515
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
--- End diff --

@JPercivall - would that also apply to ExtractEmailHeaders as well? If not, 
minding explaining why not?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70180543
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
+public class ExtractEmailAttachments extends AbstractProcessor {
+public static final String ATTACHMENT_ORIGINAL_FILENAME = 
"email.attachment.parent.filename";
+public static final String ATTACHMENT_ORIGINAL_UUID = 
"email.attachment.parent.uuid";
+
+public static final Relationship REL_ATTACHMENTS = new 
Relationship.Builder()
+.name("attachments")
+.description("Each individual attachment will be routed to the 
attachments relationship")
+.build();
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder()
+.name("original")
+.description("The original file")
+.build();
+public static final Relationship REL_FAILURE = new 
Relationship.Builde

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70181608
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ListenSMTP.java
 ---
@@ -0,0 +1,385 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+*/
+package org.apache.nifi.processors.email;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+import 
org.apache.nifi.processors.email.smtp.handler.SMTPMessageHandlerFactory;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.ssl.SSLContextService;
+
+@Tags({"listen", "email", "smtp"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("This processor implements a lightweight SMTP 
server to an arbitrary port, " +
+"allowing nifi to listen for incoming email. " +
+"" +
+"Note this server does not perform any email validation. If direct 
exposure to the internet is sought," +
+"it may be a better idea to use the combination of NiFi and an 
industrial scale MTA (e.g. Postfix)")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.helo", description = "The value 
used during HELO"),
+@WritesAttribute(attribute = "smtp.certificates.*.serial", 
description = "The serial numbers for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.certificates.*.principal", 
description = "The principal for each of the " +
+"certificates used by an TLS peer"),
+@WritesAttribute(attribute = "smtp.from", description = "The value 
used during MAIL FROM (i.e. envelope

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70196911
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
+
+
+X509Certificate[] certificates = new X509Certificate[]{};
+
+String remoteIP = messageContext.getRemoteAddress().toString();
+String helo = messageContext.getHelo();
+if (messageContext.getTlsPeerCertificates() != null ){
+certificates = (X509Certificate[]) 
messageContext.getTlsPeerCertificates().clone();
+}
+
+SmtpEvent message = new SmtpEvent(remoteIP, helo, from, 
recipient, certificates, m

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments and ExtractEm...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/483
  
@ijokarumawak Still not clear to me how you are reaching the error.

Also, note the QUIT message is not a requirement, a client may chose to 
send more than a message per session, example:

```
   smtp_connection_reuse_count_limit (0)
  When  SMTP  connection  caching  is enabled, the number of 
times
  that an SMTP session may be reused before it is closed, or  
zero
  (no limit).
```
(Souce: http://www.postfix.org/lmtp.8.html)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments and ExtractEm...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/483
  
@ijokarumawak - thanks for looking at it. 

So just for me to confirm:

PutEmail connects and tries to use SSL. This fails. It holds the socket

PutEmail connects again with plain text. This connection fails.

?

If so, isn't that the expected behavior of a single connection limit? The 
client shouldn't hold for a server if it intends to do another connection?

On another note it seems the synchronized section introduced a serious bug 
around concurrency. Seems like I created a deadlock. :-)

Looking at it at the moment.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70287734
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
+
+
+X509Certificate[] certificates = new X509Certificate[]{};
+
+String remoteIP = messageContext.getRemoteAddress().toString();
+String helo = messageContext.getHelo();
+if (messageContext.getTlsPeerCertificates() != null ){
+certificates = (X509Certificate[]) 
messageContext.getTlsPeerCertificates().clone();
+}
+
+SmtpEvent message = new SmtpEvent(remoteIP, helo, from, 
recipient, certificates, m

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70288235
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/smtp/handler/SMTPMessageHandlerFactory.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.email.smtp.handler;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.util.StopWatch;
+import org.subethamail.smtp.DropConnectionException;
+import org.subethamail.smtp.MessageContext;
+import org.subethamail.smtp.MessageHandler;
+import org.subethamail.smtp.MessageHandlerFactory;
+import org.subethamail.smtp.RejectException;
+import org.subethamail.smtp.TooMuchDataException;
+import org.subethamail.smtp.server.SMTPServer;
+
+import org.apache.nifi.processors.email.smtp.event.SmtpEvent;
+
+
+public class SMTPMessageHandlerFactory implements MessageHandlerFactory {
+final LinkedBlockingQueue incomingMessages;
+final ComponentLog logger;
+
+public SMTPMessageHandlerFactory(LinkedBlockingQueue 
incomingMessages, ComponentLog logger) {
+this.incomingMessages = incomingMessages;
+this.logger = logger;
+}
+
+@Override
+public MessageHandler create(MessageContext messageContext) {
+return new Handler(messageContext, incomingMessages, logger);
+}
+
+class Handler implements MessageHandler {
+final MessageContext messageContext;
+String from;
+String recipient;
+byte [] messageBody;
+
+
+public Handler(MessageContext messageContext, 
LinkedBlockingQueue incomingMessages, ComponentLog logger){
+this.messageContext = messageContext;
+}
+
+@Override
+public void from(String from) throws RejectException {
+// TODO: possibly whitelist senders?
+this.from = from;
+}
+
+@Override
+public void recipient(String recipient) throws RejectException {
+// TODO: possibly whitelist receivers?
+this.recipient = recipient;
+}
+
+@Override
+public void data(InputStream inputStream) throws RejectException, 
TooMuchDataException, IOException {
+// Start counting the timer...
+
+StopWatch watch = new StopWatch(false);
+
+SMTPServer server = messageContext.getSMTPServer();
+
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+
+byte [] buffer = new byte[1024];
+int rd;
+
+while ((rd = inputStream.read(buffer, 0, buffer.length)) != 
-1) {
+baos.write(buffer, 0, rd);
+}
+if (baos.getBufferLength() > server.getMaxMessageSize()) {
+throw new TooMuchDataException("Data exceeds the amount 
allowed.");
+}
+
+baos.flush();
+this.messageBody = baos.toByteArray();
--- End diff --

sounds likes a great idea. will have a look at it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/483#discussion_r70286727
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
 ---
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Date;
+
+import javax.activation.DataSource;
+import javax.mail.Address;
+import javax.mail.MessagingException;
+import javax.mail.Session;
+import javax.mail.internet.MimeMessage;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.mail.util.MimeMessageParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.FlowFileHandlingException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+
+
+
+@SupportsBatching
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email 
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The 
filename of the attachment"),
+@WritesAttribute(attribute = "email.attachment.parent.filename ", 
description = "The filename of the parent FlowFile"),
+@WritesAttribute(attribute = "email.attachment.parent.uuid", 
description = "The UUID of the original FlowFile."),
+@WritesAttribute(attribute = "mime.type", description = "The mime 
type of the attachment.")})
+
+public class ExtractEmailAttachments extends AbstractProcessor {
+public static final String ATTACHMENT_ORIGINAL_FILENAME = 
"email.attachment.parent.filename";
+public static final String ATTACHMENT_ORIGINAL_UUID = 
"email.attachment.parent.uuid";
+
+public static final Relationship REL_ATTACHMENTS = new 
Relationship.Builder()
+.name("attachments")
+.description("Each individual attachment will be routed to the 
attachments relationship")
+.build();
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder()
+.name("original")
+

[GitHub] nifi issue #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/496
  
@pvillard31 let me know if you have any other comments as I am waiting for 
this to be merged to submit a PR covering NIFI-1971 as well 

Cheers


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #819: NIFI-1965 - Convert DNS_TIMEOUT property to TIME_PER...

[trixpan]

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/819

NIFI-1965 - Convert DNS_TIMEOUT property to TIME_PERIOD

@pvillard31 - for some reason this commit ended up on another branch. Mind 
having a look?

my apologies.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-1965p2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/819.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #819


commit bb0c9c3e9334fbb95a76930df7c600ae25aa7edb
Author: Andre F de Miranda <trix...@users.noreply.github.com>
Date:   2016-08-09T13:08:16Z

NIFI-1965 - COnvert DNS_TIMEOUT property to TIME_PERIOD




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r73065633
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r73066145
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r73065856
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
--- End diff --

completely missed this. my bad


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #712: NIFI-2374 and NIFI-2375 - Minor improve to documentation an...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/712
  
@olegz mind having a look on this? Simple PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/710#discussion_r71973547
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/AbstractEmailProcessor.java
 ---
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import javax.mail.Address;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.support.StaticListableBeanFactory;
+import org.springframework.integration.mail.AbstractMailReceiver;
+import org.springframework.util.Assert;
+import org.springframework.util.StreamUtils;
+
+/**
+ * Base processor for implementing processors to consume messages from 
Email
+ * servers using Spring Integration libraries.
+ *
+ * @param 
+ *the type of {@link AbstractMailReceiver}.
+ */
+abstract class AbstractEmailProcessor 
extends AbstractProcessor {
+
+public static final PropertyDescriptor HOST = new 
PropertyDescriptor.Builder()
+.name("Host Name")
+.description("Network address of Email server (e.g., 
pop.gmail.com, imap.gmail.com . . .)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PORT = new 
PropertyDescriptor.Builder()
+.name("Port")
+.description("Numeric value identifying Port of Email server 
(e.g., 993)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.build();
+public static final PropertyDescriptor USER = new 
PropertyDescriptor.Builder()
+.name("User Name")
+.description("User Name used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("Password")
+.description("Password used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+public static final PropertyDescriptor FOLDER = new 
PropertyDescriptor.Builder()
+.name("Folder")
+   

[GitHub] nifi pull request #712: NIFI-2374 and NIFI-2375 - Minor improve to documenta...

[trixpan]

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/712

NIFI-2374 and NIFI-2375 - Minor improve to documentation and version bump

NIFI-2374 - Today when I was about to raise an ISSUE I've noticed that 
although the IdentifyMimeType documentation provides a list of MIME-types that 
list is far from complete. This commit slightly changes wording to reflect this

NIFI-2375 - Bump Apache Tika's version used by IdentifyMimeType and 
ExtractMediaMetada processors

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-2374

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/712.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #712


commit d172899d315062dc1432af64c1f2ee4b0ceb07ad
Author: Andre F de Miranda <trix...@users.noreply.github.com>
Date:   2016-07-23T15:08:42Z

NIFI-2374 - Adjust documentation wording to clarify IdentifyMimeType
 is a non exhaustive list of mime type values

commit b918bd4d8d140543a46f2988b43eb2bd95999c8a
Author: Andre F de Miranda <trix...@users.noreply.github.com>
Date:   2016-07-23T15:10:39Z

NIFI-2375 - Bump Apache Tika dependency version to 1.13




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/710#discussion_r71973345
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/AbstractEmailProcessor.java
 ---
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import javax.mail.Address;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.support.StaticListableBeanFactory;
+import org.springframework.integration.mail.AbstractMailReceiver;
+import org.springframework.util.Assert;
+import org.springframework.util.StreamUtils;
+
+/**
+ * Base processor for implementing processors to consume messages from 
Email
+ * servers using Spring Integration libraries.
+ *
+ * @param 
+ *the type of {@link AbstractMailReceiver}.
+ */
+abstract class AbstractEmailProcessor 
extends AbstractProcessor {
+
+public static final PropertyDescriptor HOST = new 
PropertyDescriptor.Builder()
+.name("Host Name")
+.description("Network address of Email server (e.g., 
pop.gmail.com, imap.gmail.com . . .)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PORT = new 
PropertyDescriptor.Builder()
+.name("Port")
+.description("Numeric value identifying Port of Email server 
(e.g., 993)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.build();
+public static final PropertyDescriptor USER = new 
PropertyDescriptor.Builder()
+.name("User Name")
+.description("User Name used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("Password")
+.description("Password used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+public static final PropertyDescriptor FOLDER = new 
PropertyDescriptor.Builder()
+.name("Folder")
+   

[GitHub] nifi pull request #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/710#discussion_r71973519
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/AbstractEmailProcessor.java
 ---
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import javax.mail.Address;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.support.StaticListableBeanFactory;
+import org.springframework.integration.mail.AbstractMailReceiver;
+import org.springframework.util.Assert;
+import org.springframework.util.StreamUtils;
+
+/**
+ * Base processor for implementing processors to consume messages from 
Email
+ * servers using Spring Integration libraries.
+ *
+ * @param 
+ *the type of {@link AbstractMailReceiver}.
+ */
+abstract class AbstractEmailProcessor 
extends AbstractProcessor {
+
+public static final PropertyDescriptor HOST = new 
PropertyDescriptor.Builder()
+.name("Host Name")
+.description("Network address of Email server (e.g., 
pop.gmail.com, imap.gmail.com . . .)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PORT = new 
PropertyDescriptor.Builder()
+.name("Port")
+.description("Numeric value identifying Port of Email server 
(e.g., 993)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.build();
+public static final PropertyDescriptor USER = new 
PropertyDescriptor.Builder()
+.name("User Name")
+.description("User Name used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("Password")
+.description("Password used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+public static final PropertyDescriptor FOLDER = new 
PropertyDescriptor.Builder()
+.name("Folder")
+   

[GitHub] nifi pull request #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/710#discussion_r71973968
  
--- Diff: 
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/AbstractEmailProcessor.java
 ---
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import javax.mail.Address;
+import javax.mail.Message;
+import javax.mail.MessagingException;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.support.StaticListableBeanFactory;
+import org.springframework.integration.mail.AbstractMailReceiver;
+import org.springframework.util.Assert;
+import org.springframework.util.StreamUtils;
+
+/**
+ * Base processor for implementing processors to consume messages from 
Email
+ * servers using Spring Integration libraries.
+ *
+ * @param 
+ *the type of {@link AbstractMailReceiver}.
+ */
+abstract class AbstractEmailProcessor 
extends AbstractProcessor {
+
+public static final PropertyDescriptor HOST = new 
PropertyDescriptor.Builder()
+.name("Host Name")
+.description("Network address of Email server (e.g., 
pop.gmail.com, imap.gmail.com . . .)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PORT = new 
PropertyDescriptor.Builder()
+.name("Port")
+.description("Numeric value identifying Port of Email server 
(e.g., 993)")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.PORT_VALIDATOR)
+.build();
+public static final PropertyDescriptor USER = new 
PropertyDescriptor.Builder()
+.name("User Name")
+.description("User Name used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("Password")
+.description("Password used for authentication and 
authorization with Email server.")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+public static final PropertyDescriptor FOLDER = new 
PropertyDescriptor.Builder()
+.name("Folder")
+   

[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/858
  
# Note to reviewers: #

This is a logging pipeline focused processor that should be capable of 
reaching to a Whois server and grabbing information that can be used to enrich 
data.

The junit uses powermockito to prevent internet access dependency during 
build, however, real world testing can be done against the following Whois 
based APIs:

1. http://www.team-cymru.org/IP-ASN-mapping.html
2. https://www.shadowserver.org/wiki/pmwiki.php/Services/IP-BGP#toc4
3. http://www.pwhois.org/webquery.who

Please shout in case you have questions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor

[trixpan]

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/858

NIFI-1971 - Introduce QueryWhois processor 

supporting batching (i.e. netcat protocol) 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-1971

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/858.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #858


commit ba438e1e029e139848e83942dda722386ebee9b2
Author: Andre F de Miranda <trix...@users.noreply.github.com>
Date:   2016-08-09T13:01:53Z

NIFI-1971 - Introduce QueryWhois processor with batching (i.e. netcat 
protocol) support




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/858
  
A gist to help people going

https://gist.github.com/trixpan/9c663866d2b02322650bfddde3eefb6e


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/858
  
@pvillard31 given your experience reviewing QueryDNS, would you mind having 
a look on this one? Different interface, similar outcome.

Cheers


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #875: Nifi 2477

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/875#discussion_r75027919
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -169,6 +169,81 @@ Now that the User Interface has been secured, we can 
easily secure Site-to-Site
 accomplished by setting the `nifi.remote.input.secure` and 
`nifi.cluster.protocol.is.secure` properties, respectively, to `true`.
 
 
+TLS Generation Toolkit
+~~
+
+In order to facilitate the secure setup of NiFi, a tls-toolkit command 
line utility is available to automatically generate the required keystores, 
truststore, and relevant configuration files. This is especially useful for 
securing multiple NiFi nodes, which can be a tedious and error-prone process.
+
+The tls-toolkit has two primary modes of operation:
+
+1. Standalone -- generates the certificate authority, keystores, 
truststores, and nifi.properties files in one command.
+2. Client/Server mode -- uses a Certificate Authority Server that accepts 
Certificate Signing Requests from clients, signs them, and sends the resulting 
certificates back.  Both client and server validate the other’s identity 
through a shared secret.
+
+Standalone
+^^
+Standalone mode can be invoked by running “tls-toolkit.sh standalone 
-h” which will print the usage information along with descriptions of options 
that can be specified.
+
+The most common options to specify are:
+
+* -n (or --hostnames) a comma-separated list of hostnames that you’d 
like to generate certificates for.  It can be specified multiple times.  Range 
and instance patterns are supported. (see below for details)
+* -f (or --nifiPropertiesFile) a base nifi.properties file that the tool 
will update for each host
+* -o (or --outputDirectory) the directory to use for the resulting 
Certificate Authority files and NiFi configurations.  A subdirectory will be 
made for each host.
+
+Hostname Patterns:
+
+* Square brackets can be used in order to easily specify a range of 
hostnames. Example: [01-20]
+* Parentheses can be used in order to specify that more than one NiFi 
instance will run on the given host(s). Example: (5)
+
+Examples:
--- End diff --

+1 on @bbende comment


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/858#discussion_r75210911
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryWhois.java
 ---
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.net.whois.WhoisClient;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"whois", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful whois query processor primary designed 
to enrich DataFlows with whois based APIs " +
+"(e.g. ShadowServer's ASN lookup) but that can be also used to 
perform regular whois lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the Whois query response for each of the 
records received"),
+})
+public class QueryWhois extends AbstractEnrichProcessor {
+
+public static final AllowableValue BEGIN_END = new 
AllowableValue("Begin/End", "Begin/End",
+"The evaluated input of each flowfile is enclosed within begin 
and end tags. Each row contains a delimited set of fields");
+
+public static final AllowableValue BULK_NONE = new 
AllowableValue("None", "None",
+"Queries are made without any particular dialect");
+
+
+public static final PropertyDescriptor WHOIS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("WHOIS_QUERY_TYPE")
+.displayName("Whois Query Type")
+.description("The Whois query type to be used by the processor 
(if used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER = new 
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER")
+.displayName("Whois Server")
+.description("The Whois server to be used")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER_PORT = new 
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER_PORT")

[GitHub] nifi issue #1462: NIFI-819 - Extend GetHTTP to use dynamically produce HTTP ...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1462
  
@apiri A long time ago in a galaxy far far away you raised this JIRA 
ticket. 

Does this PR addresses what you had in mind?

Cheers


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically produc...

[trixpan]

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/1462

NIFI-819 - Extend GetHTTP to use dynamically produce HTTP headers upo…

…n a request

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [-] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [-] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [-] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [x] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-819

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1462.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1462


commit 853a83015fd13878d0cc52fc646b2221b15c1443
Author: Andre F de Miranda <trix...@users.noreply.github.com>
Date:   2017-02-01T23:23:23Z

NIFI-819 - Extend GetHTTP to use dynamically produce HTTP headers upon a 
request




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1468: NIFI-3430 adds support for date format property to parse i...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1468
  
Nick, thank you for the contribution. would you mind squashing the commits?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically add HT...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1462#discussion_r99038019
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
 ---
@@ -62,18 +62,12 @@
 import org.apache.http.impl.client.HttpClientBuilder;
 import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
 import org.apache.http.ssl.SSLContextBuilder;
-import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.*;
 import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
-import org.apache.nifi.annotation.behavior.Stateful;
-import org.apache.nifi.annotation.behavior.WritesAttribute;
-import org.apache.nifi.annotation.behavior.WritesAttributes;
 import org.apache.nifi.annotation.documentation.CapabilityDescription;
 import org.apache.nifi.annotation.documentation.Tags;
 import org.apache.nifi.annotation.lifecycle.OnScheduled;
-import org.apache.nifi.components.AllowableValue;
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.*;
--- End diff --

Damn you intellij!!! :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically add HT...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1462#discussion_r99038165
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
 ---
@@ -102,6 +96,11 @@
 + "once the content has been fetched from the given URL, it will not 
be fetched again until the content on the remote server changes. Note that due 
to limitations on state "
 + "management, stored \"last modified\" and etag fields never expire. 
If the URL in GetHttp uses Expression Language that is unbounded, there "
 + "is the potential for Out of Memory Errors to occur.")
+@DynamicProperties({
+@DynamicProperty(name = "header_XYZ", value = "The Expression Language 
to be used to populate the header value", description = "The additional headers 
to be sent by the processor " +
+"whenever making a new HTTP request. \n " +
+"Setting this the name to XYZ and value to ${attribute} will 
result on the header 'XYZ: attribute_value' being sent to the HTTP endpoint"),
--- End diff --

Oh ESL... :-)

Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically add HT...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1462#discussion_r99038013
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
 ---
@@ -62,18 +62,12 @@
 import org.apache.http.impl.client.HttpClientBuilder;
 import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
 import org.apache.http.ssl.SSLContextBuilder;
-import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.*;
--- End diff --

Damn you intellij!!! :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically add HT...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1462#discussion_r99038126
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
 ---
@@ -102,6 +96,11 @@
 + "once the content has been fetched from the given URL, it will not 
be fetched again until the content on the remote server changes. Note that due 
to limitations on state "
 + "management, stored \"last modified\" and etag fields never expire. 
If the URL in GetHttp uses Expression Language that is unbounded, there "
 + "is the potential for Out of Memory Errors to occur.")
+@DynamicProperties({
+@DynamicProperty(name = "header_XYZ", value = "The Expression Language 
to be used to populate the header value", description = "The additional headers 
to be sent by the processor " +
--- End diff --

I agree but couldn't think of a better way of conveying the message. Happy 
to give it another go.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically add HT...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1462#discussion_r99039524
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
 ---
@@ -62,18 +62,12 @@
 import org.apache.http.impl.client.HttpClientBuilder;
 import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
 import org.apache.http.ssl.SSLContextBuilder;
-import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.*;
--- End diff --

@alopresto I saw that one before... Challenge is trying not to forget doing 
it upon setting up a new dev environment. :-) 

Damn you memory for forgetting you would damn intellij for optimising 
imports. :-D


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically add HT...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1462#discussion_r99039594
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
 ---
@@ -102,6 +96,11 @@
 + "once the content has been fetched from the given URL, it will not 
be fetched again until the content on the remote server changes. Note that due 
to limitations on state "
 + "management, stored \"last modified\" and etag fields never expire. 
If the URL in GetHttp uses Expression Language that is unbounded, there "
 + "is the potential for Out of Memory Errors to occur.")
+@DynamicProperties({
+@DynamicProperty(name = "header_XYZ", value = "The Expression Language 
to be used to populate the header value", description = "The additional headers 
to be sent by the processor " +
--- End diff --

fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1462: NIFI-819 - Extend GetHTTP to use dynamically add HT...

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1462#discussion_r99038321
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
 ---
@@ -467,6 +486,18 @@ public void onTrigger(final ProcessContext context, 
final ProcessSessionFactory
 if (accept != null) {
 get.addHeader(HEADER_ACCEPT, accept);
 }
+
+// Add dynamic headers
+
+PropertyValue customHeaderValue;
+for (PropertyDescriptor customProperty : customHeaders) {
+customHeaderValue = 
context.getProperty(customProperty).evaluateAttributeExpressions();
--- End diff --

I considered that but refrained from it due to:


https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java#L99


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---