[GitHub] nifi issue #1481: NIFI-3055 StandardRecordWriter Can Throw UTFDataFormatExce...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1481 @markap14 I suspect the reason is @mosermw added the closure after the signature and closure should come above sign-off-by? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1326: NIFI-2779 - Add processor to GetEmail Supporting Exchange ...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1326 @patricker merge after build, reviewed code and ran against an exchange server. Seems to be working fine. thank you for the contribution! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1505: NIFI-2565 - Add Elastic copyright to notice due to the use...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1505 Epic. Will adjust --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #972: NIFI-2711 - Making top-level nifi-assemblies directory with...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/972 @brosander is the PR still wanted? If yes, mind rebasing? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1326: NIFI-2779 - Add processor to GetEmail Supporting Exchange ...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1326 @patricker can you please rebase as changes to master are causing conflicts ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #453: Nifi 539c
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/453 @eorgad , this PR is nearly 1 year old do you still plan to address some of the feedback? Would you mind closing the PR for now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1335: NIFI-3213: ListFile - Do not skip obviously old files.
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1335 @ijokarumawak my bad. I was just running a extra set of compilations and I noticed that under certain conditions there seems to be a race condition affecting ``` Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.66 sec <<< FAILURE! - in org.apache.nifi.processors.standard.TestListFile testFilterAge(org.apache.nifi.processors.standard.TestListFile) Time elapsed: 1.212 sec <<< FAILURE! org.junit.ComparisonFailure: expected:<age[2].txt> but was:<age[1].txt> at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.nifi.processors.standard.TestListFile.testFilterAge(TestListFile.java:223) ``` Do you want to have a look at it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1016 That was a merge, not a rebase :-( --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1016 Try git checkout master -b nifi-2724v2 git cherry-pick git push origin nifi-2724 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1016 Try git checkout master -b nifi-2724v2 git cherry-pick your commit hash git push - f origin nifi-2724 This will create a temporary branch Allow you to copy a commit from another branch (cherry pick) and the force git to push to your original branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1016 No no. Cherry pick your commit number. You will likely to fix conflicts in the process but git will flag those files with conflict so once you fix it should be ok at the end --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1402: NIFI-3288: Add Clojure support for ExecuteScript
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1402 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1108: NIFI-2565: add Grok parser
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1108#discussion_r96236367
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.thekraken.grok.api.Grok;
+import io.thekraken.grok.api.Match;
+import io.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+
+@Tags({"Grok Processor", "grok", "log", "text", "parse", "delimit",
"extract"})
--- End diff --
"Grok Processor" looks a bit out of place but should not prevent merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #420: NIFI-1280 Create FilterCSVColumns Processor.
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/420#discussion_r96251033 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -235,6 +235,11 @@ language governing permissions and limitations under the License. --> json-utils 0.0.20 + +org.apache.calcite +calcite-example-csv +1.7.0 --- End diff -- given months have passed, should we bump this to 1.11.0 ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1198: [NIFI-3015] Run NiFi service from nifi user instead of roo...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1198 @jfrazee - odd. Should the bootstrap.conf `run.as` be [populated](https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/bootstrap.conf) from the [RPM profile](https://github.com/apache/nifi/blame/92f17a995b7c6b763a55974fffda00780c56/nifi-assembly/pom.xml#L429) already? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1302: NIFI-2656: replace -k [password] with -K [passwordfile].
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1302 @alopresto ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #991: NIFI-2742 - Explicitly declare the kite hadoop depen...
Github user trixpan closed the pull request at: https://github.com/apache/nifi/pull/991 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #397: NIFI-1815
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/397 @jdye64 is the PR active or should we close it? Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1421: NIFI-3352 - Address issue where ListenLumberjack wa...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1421 NIFI-3352 - Address issue where ListenLumberjack was sending long num⦠â¦bers back to clients - instead of int Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-3352 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1421.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1421 commit 628c6673b2843117b28291acfafb86a049d62d10 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2017-01-14T11:57:33Z NIFI-3352 - Address issue where ListenLumberjack was sending long numbers back to clients - instead of int --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #709: NIFI-2369 - Added EL support to CSV properties
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/709 @simonellistonball @mattyb149 should we perhaps close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1326: NIFI-2779 - Add processor to GetEmail Supporting Ex...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1326#discussion_r96258831 --- Diff: nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java --- @@ -0,0 +1,518 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.email; + +import microsoft.exchange.webservices.data.autodiscover.IAutodiscoverRedirectionUrl; +import microsoft.exchange.webservices.data.core.ExchangeService; +import microsoft.exchange.webservices.data.core.PropertySet; +import microsoft.exchange.webservices.data.core.enumeration.misc.ExchangeVersion; +import microsoft.exchange.webservices.data.core.enumeration.property.BodyType; +import microsoft.exchange.webservices.data.core.enumeration.property.WellKnownFolderName; +import microsoft.exchange.webservices.data.core.enumeration.search.FolderTraversal; +import microsoft.exchange.webservices.data.core.enumeration.search.LogicalOperator; +import microsoft.exchange.webservices.data.core.enumeration.search.SortDirection; +import microsoft.exchange.webservices.data.core.enumeration.service.ConflictResolutionMode; +import microsoft.exchange.webservices.data.core.enumeration.service.DeleteMode; +import microsoft.exchange.webservices.data.core.service.folder.Folder; +import microsoft.exchange.webservices.data.core.service.item.EmailMessage; +import microsoft.exchange.webservices.data.core.service.item.Item; +import microsoft.exchange.webservices.data.core.service.schema.EmailMessageSchema; +import microsoft.exchange.webservices.data.core.service.schema.FolderSchema; +import microsoft.exchange.webservices.data.core.service.schema.ItemSchema; +import microsoft.exchange.webservices.data.credential.ExchangeCredentials; +import microsoft.exchange.webservices.data.credential.WebCredentials; +import microsoft.exchange.webservices.data.property.complex.FileAttachment; +import microsoft.exchange.webservices.data.search.FindFoldersResults; +import microsoft.exchange.webservices.data.search.FindItemsResults; +import microsoft.exchange.webservices.data.search.FolderView; +import microsoft.exchange.webservices.data.search.ItemView; +import microsoft.exchange.webservices.data.search.filter.SearchFilter; +import org.apache.commons.mail.EmailAttachment; +import org.apache.commons.mail.EmailException; +import org.apache.commons.mail.HtmlEmail; +import org.apache.commons.mail.MultiPartEmail; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.OutputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.mail.Address; +import javax.mail.Flags; +import javax.mail.Message; +import javax.mail.MessagingException; +import javax.mail.internet.MimeMessage; +import javax.mail.util.ByteArrayDataSource; +import java.io.IOException; +import java.io.OutputStream; +import java.net.URI; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.TimeUnit; +
[GitHub] nifi pull request #1326: NIFI-2779 - Add processor to GetEmail Supporting Ex...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1326#discussion_r96259066 --- Diff: nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java --- @@ -0,0 +1,518 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.email; + +import microsoft.exchange.webservices.data.autodiscover.IAutodiscoverRedirectionUrl; +import microsoft.exchange.webservices.data.core.ExchangeService; +import microsoft.exchange.webservices.data.core.PropertySet; +import microsoft.exchange.webservices.data.core.enumeration.misc.ExchangeVersion; +import microsoft.exchange.webservices.data.core.enumeration.property.BodyType; +import microsoft.exchange.webservices.data.core.enumeration.property.WellKnownFolderName; +import microsoft.exchange.webservices.data.core.enumeration.search.FolderTraversal; +import microsoft.exchange.webservices.data.core.enumeration.search.LogicalOperator; +import microsoft.exchange.webservices.data.core.enumeration.search.SortDirection; +import microsoft.exchange.webservices.data.core.enumeration.service.ConflictResolutionMode; +import microsoft.exchange.webservices.data.core.enumeration.service.DeleteMode; +import microsoft.exchange.webservices.data.core.service.folder.Folder; +import microsoft.exchange.webservices.data.core.service.item.EmailMessage; +import microsoft.exchange.webservices.data.core.service.item.Item; +import microsoft.exchange.webservices.data.core.service.schema.EmailMessageSchema; +import microsoft.exchange.webservices.data.core.service.schema.FolderSchema; +import microsoft.exchange.webservices.data.core.service.schema.ItemSchema; +import microsoft.exchange.webservices.data.credential.ExchangeCredentials; +import microsoft.exchange.webservices.data.credential.WebCredentials; +import microsoft.exchange.webservices.data.property.complex.FileAttachment; +import microsoft.exchange.webservices.data.search.FindFoldersResults; +import microsoft.exchange.webservices.data.search.FindItemsResults; +import microsoft.exchange.webservices.data.search.FolderView; +import microsoft.exchange.webservices.data.search.ItemView; +import microsoft.exchange.webservices.data.search.filter.SearchFilter; +import org.apache.commons.mail.EmailAttachment; +import org.apache.commons.mail.EmailException; +import org.apache.commons.mail.HtmlEmail; +import org.apache.commons.mail.MultiPartEmail; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.OutputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.mail.Address; +import javax.mail.Flags; +import javax.mail.Message; +import javax.mail.MessagingException; +import javax.mail.internet.MimeMessage; +import javax.mail.util.ByteArrayDataSource; +import java.io.IOException; +import java.io.OutputStream; +import java.net.URI; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.TimeUnit; +
[GitHub] nifi issue #1326: NIFI-2779 - Add processor to GetEmail Supporting Exchange ...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1326 @patricker Joe will have a more complete view on this but looking at the MSFT code there seem to be some dependencies with NOTICE. From top of my mind, joda-time, commons-io and others that must be called out both at the nar and assembly NOTICE files. I also noticed you have not added the MIT LICENSE provided by MSFT into nar and assembly. [This](https://github.com/patricker/nifi/blob/master/nifi-nar-bundles/nifi-flume-bundle/nifi-flume-nar/src/main/resources/META-INF/NOTICE) and [this](https://github.com/patricker/nifi/blob/master/nifi-nar-bundles/nifi-flume-bundle/nifi-flume-nar/src/main/resources/META-INF/LICENSE) should give you an idea what the files have to look like But as I said, @joewitt will be a more reliable source on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1422: NIFI-3015 - Corrects issue were RPM profile run.as ...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1422#discussion_r96312325 --- Diff: nifi-assembly/pom.xml --- @@ -19,6 +19,10 @@ language governing permissions and limitations under the License. --> nifi-assembly pom This is the assembly Apache NiFi + + + --- End diff -- @jfrazee yes it is expected as the pom file is shared across all packages. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1422: NIFI-3015 - Corrects issue were RPM profile run.as ...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1422#discussion_r96984003 --- Diff: nifi-assembly/pom.xml --- @@ -19,6 +19,10 @@ language governing permissions and limitations under the License. --> nifi-assembly pom This is the assembly Apache NiFi + + + --- End diff -- I find it overkill (the person would be voluntarily building the RPM) but doable. Will adjust the PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1422: NIFI-3015 - Corrects issue were RPM profile run.as ...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1422#discussion_r96322520 --- Diff: nifi-assembly/pom.xml --- @@ -19,6 +19,10 @@ language governing permissions and limitations under the License. --> nifi-assembly pom This is the assembly Apache NiFi + + + --- End diff -- I got what you are saying. The challenge is that the run.as user comes from a pom attribute before the RPM creation is started, meaning the the build-root used by the RPM packager already contains the correct run.as The sequence seems to be: If -Prpm is enable used the line you highlighted above, if not, use https://github.com/apache/nifi/blob/master/nifi-assembly/pom.xml#L429 The only other way I imagine being possible would be using sed to replace the `run.as=` with `run.as=nifi` prior to RPM the built-root folder? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1422: NIFI-3015 - Corrects issue were RPM profile run.as ...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1422 NIFI-3015 - Corrects issue were RPM profile run.as property was not b⦠â¦eing used This supersedes #1198 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [-] Have you written or updated unit tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [-] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [-] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-3015 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1422 commit 849c5cf26c4db516ce8499f4e9fa750ab8e1fe0c Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2017-01-16T16:44:55Z NIFI-3015 - Corrects issue were RPM profile run.as property was not being used This supersedes #1198 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1422: NIFI-3015 - Corrects issue were RPM profile run.as propert...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1422 @fidget83 and @jfrazee mind having a look and let me know if it addresses the issue? Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1422: NIFI-3015 - Corrects issue were RPM profile run.as propert...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1422 Reviewers: Key here is that bootstrap.conf with RPM profile should result on `run.as=nifi` while vanilla build should result in `run.as=` (naturally assuming nothing else breaks) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #871: NIFI-2519 - ListenSMTP addresses mismatch between sm...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/871 NIFI-2519 - ListenSMTP addresses mismatch between smtp.src attribute ⦠â¦documentation and attribute.put You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-2519 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/871.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #871 commit 47e4ca973c4d13be8452d09e67dd04d3493e4022 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2016-08-15T23:13:39Z NIFI-2519 - ListenSMTP addresses mismatch between smtp.src attribute documentation and attribute.put --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r75580268
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryWhois.java
---
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.net.whois.WhoisClient;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"whois", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful whois query processor primary designed
to enrich DataFlows with whois based APIs " +
+"(e.g. ShadowServer's ASN lookup) but that can be also used to
perform regular whois lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*",
description = "The captured fields of the Whois query response for each of the
records received"),
+})
+public class QueryWhois extends AbstractEnrichProcessor {
+
+public static final AllowableValue BEGIN_END = new
AllowableValue("Begin/End", "Begin/End",
+"The evaluated input of each flowfile is enclosed within begin
and end tags. Each row contains a delimited set of fields");
+
+public static final AllowableValue BULK_NONE = new
AllowableValue("None", "None",
+"Queries are made without any particular dialect");
+
+
+public static final PropertyDescriptor WHOIS_QUERY_TYPE = new
PropertyDescriptor.Builder()
+.name("WHOIS_QUERY_TYPE")
+.displayName("Whois Query Type")
+.description("The Whois query type to be used by the processor
(if used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER")
+.displayName("Whois Server")
+.description("The Whois server to be used")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER_PORT = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER_PORT")
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 The error is expected as the regex does not contain a capture group named KEY. It is a bug as I should detect KEY or whatever value was chosen. But your feedback highlights that instead of having named captures on regex vs numbered captures on split, we rather have a single approach (i.e. numbered captures) across both parsing strategies. I will fix it and also provide some better insight via validation messages --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/858#discussion_r75579997 --- Diff: nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/pom.xml --- @@ -63,5 +63,31 @@ nifi-mock test + +commons-net +commons-net +3.3 --- End diff -- addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r75580295
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryWhois.java
---
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.net.whois.WhoisClient;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"whois", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful whois query processor primary designed
to enrich DataFlows with whois based APIs " +
+"(e.g. ShadowServer's ASN lookup) but that can be also used to
perform regular whois lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*",
description = "The captured fields of the Whois query response for each of the
records received"),
+})
+public class QueryWhois extends AbstractEnrichProcessor {
+
+public static final AllowableValue BEGIN_END = new
AllowableValue("Begin/End", "Begin/End",
+"The evaluated input of each flowfile is enclosed within begin
and end tags. Each row contains a delimited set of fields");
+
+public static final AllowableValue BULK_NONE = new
AllowableValue("None", "None",
+"Queries are made without any particular dialect");
+
+
+public static final PropertyDescriptor WHOIS_QUERY_TYPE = new
PropertyDescriptor.Builder()
+.name("WHOIS_QUERY_TYPE")
+.displayName("Whois Query Type")
+.description("The Whois query type to be used by the processor
(if used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER")
+.displayName("Whois Server")
+.description("The Whois server to be used")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER_PORT = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER_PORT")
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/858#discussion_r75580308 --- Diff: nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/test/java/org/apache/nifi/processors/enrich/TestQueryWhois.java --- @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.enrich; + + +import org.apache.commons.net.whois.WhoisClient; +import org.apache.nifi.util.MockFlowFile; +import org.apache.nifi.util.TestRunner; +import org.apache.nifi.util.TestRunners; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.junit.runner.RunWith; +import org.mockito.Mockito; --- End diff -- addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/858#discussion_r75579988 --- Diff: nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/pom.xml --- @@ -63,5 +63,31 @@ nifi-mock test + +commons-net +commons-net +3.3 + + +com.google.guava +guava + + +org.powermock +powermock-module-junit4 +1.6.5 +test + + +org.powermock +powermock-api-mockito +1.6.5 +test + + +org.powermock +powermock-api-mockito-common +1.6.5 --- End diff -- addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 As you can see the KEY group is documented within NiFi code but the UI lacks such explanation https://github.com/apache/nifi/pull/858/files#diff-d13cbfd36645869513f9d09b76636240R183 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1418#discussion_r103645596 --- Diff: nifi-nar-bundles/nifi-beats-bundle/nifi-beats-nar/src/main/resources/META-INF/LICENSE --- @@ -0,0 +1,233 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have ma
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1418#discussion_r103647459 --- Diff: nifi-nar-bundles/nifi-beats-bundle/nifi-beats-processors/pom.xml --- @@ -0,0 +1,76 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-beats-bundle +1.2.0-SNAPSHOT + + +nifi-beats-processors +jar + + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-processor-utils + + +org.apache.nifi +nifi-mock +test + + +org.apache.nifi +nifi-socket-utils + + +org.apache.nifi +nifi-utils + + +org.apache.nifi +nifi-flowfile-packager + + +org.apache.nifi +nifi-ssl-context-service-api +provided + + +org.slf4j +slf4j-simple +test + + +junit +junit +4.11 +test + + +com.google.code.gson --- End diff -- fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1418#discussion_r103648185
--- Diff:
nifi-nar-bundles/nifi-lumberjack-bundle/nifi-lumberjack-processors/src/main/java/org/apache/nifi/processors/lumberjack/ListenLumberjack.java
---
@@ -62,12 +62,13 @@
import com.google.gson.Gson;
+@Deprecated
@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
@Tags({"listen", "lumberjack", "tcp", "logs"})
-@CapabilityDescription("Listens for Lumberjack messages being sent to a
given port over TCP. Each message will be " +
+@CapabilityDescription("This processor is deprecated and will be removed
in the near future. Listens for Lumberjack messages being sent to a given port
over TCP. Each message will be " +
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1418#discussion_r103648209 --- Diff: nifi-nar-bundles/nifi-lumberjack-bundle/nifi-lumberjack-processors/src/main/java/org/apache/nifi/processors/lumberjack/event/LumberjackEvent.java --- @@ -24,6 +24,7 @@ /** * A Lumberjack event which adds the transaction number and command to the StandardEvent. */ +@Deprecated --- End diff -- hopefully addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1418#discussion_r103649158
--- Diff:
nifi-nar-bundles/nifi-beats-bundle/nifi-beats-processors/src/main/java/org/apache/nifi/processors/beats/ListenBeats.java
---
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.beats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.BlockingQueue;
+
+import javax.net.ssl.SSLContext;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.attributes.FlowFileAttributeKey;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import
org.apache.nifi.processor.util.listen.AbstractListenEventBatchingProcessor;
+import
org.apache.nifi.processor.util.listen.dispatcher.AsyncChannelDispatcher;
+import org.apache.nifi.processor.util.listen.dispatcher.ChannelDispatcher;
+import
org.apache.nifi.processor.util.listen.dispatcher.SocketChannelDispatcher;
+import org.apache.nifi.processor.util.listen.event.EventFactory;
+import org.apache.nifi.processor.util.listen.handler.ChannelHandlerFactory;
+import org.apache.nifi.processor.util.listen.response.ChannelResponder;
+import org.apache.nifi.processor.util.listen.response.ChannelResponse;
+import org.apache.nifi.processors.beats.event.BeatsEvent;
+import org.apache.nifi.processors.beats.event.BeatsEventFactory;
+import org.apache.nifi.processors.beats.frame.BeatsEncoder;
+import
org.apache.nifi.processors.beats.handler.BeatsSocketChannelHandlerFactory;
+import org.apache.nifi.processors.beats.response.BeatsChannelResponse;
+import org.apache.nifi.processors.beats.response.BeatsResponse;
+import org.apache.nifi.ssl.SSLContextService;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"listen", "beats", "tcp", "logs"})
+@CapabilityDescription("Listens for messages sent by libbeat compatible
clients (e.g. filebeats, metricbeats, etc) being sent to a given port over TCP,
writing its contents in" +
+"JSON format to the content of the message to a FlowFile." +
+"This processor replaces the now deprecated ListenLumberjack")
+@WritesAttributes({
+@WritesAttribute(attribute = "beats.sender", description = "The
sending host of the messages."),
+@WritesAttribute(attribute = "beats.port", description = "The sending
port the messages were received over."),
+@WritesAttribute(attribute = "beats.sequencenumber", description =
"The sequence number of the message. Only included if is 1."),
+@WritesAttribute(attribute = "mime.type", description = "The mime.type
of the content which is application/json")
+})
+@SeeAlso(classNames = {"org.apache.nifi.processors.standard.ParseSyslog"})
+public class ListenBeats extends
AbstractListenEventBatchingProcessor {
+
+
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1418#discussion_r103650361
--- Diff:
nifi-nar-bundles/nifi-beats-bundle/nifi-beats-processors/src/main/java/org/apache/nifi/processors/beats/frame/BeatsDecoder.java
---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.beats.frame;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.zip.InflaterInputStream;
+
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Decodes a Beats frame by maintaining a state based on each byte that
has been processed. This class
+ * should not be shared by multiple threads.
+ */
+public class BeatsDecoder {
+
+
+static final Logger logger =
LoggerFactory.getLogger(BeatsDecoder.class);
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1418#discussion_r103651066
--- Diff:
nifi-nar-bundles/nifi-beats-bundle/nifi-beats-processors/src/main/java/org/apache/nifi/processors/beats/frame/BeatsDecoder.java
---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.beats.frame;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.zip.InflaterInputStream;
+
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Decodes a Beats frame by maintaining a state based on each byte that
has been processed. This class
+ * should not be shared by multiple threads.
+ */
+public class BeatsDecoder {
+
+
+static final Logger logger =
LoggerFactory.getLogger(BeatsDecoder.class);
+
+private BeatsFrame.Builder frameBuilder;
+private BeatsState currState = BeatsState.VERSION;
+private byte decodedFrameType;
+
+private byte[] unprocessedData;
+
+private final Charset charset;
+private final ByteArrayOutputStream currBytes;
+
+private long windowSize;
+
+static final int MIN_FRAME_HEADER_LENGTH = 2; // Version + Type
+static final int WINDOWSIZE_LENGTH = MIN_FRAME_HEADER_LENGTH + 4; //
32bit unsigned window size
+static final int COMPRESSED_MIN_LENGTH = MIN_FRAME_HEADER_LENGTH + 4;
// 32 bit unsigned + payload
+static final int JSON_MIN_LENGTH = MIN_FRAME_HEADER_LENGTH + 8; // 32
bit unsigned sequence number + 32 bit unsigned payload length
+
+public static final byte FRAME_WINDOWSIZE = 0x57, FRAME_DATA = 0x44,
FRAME_COMPRESSED = 0x43, FRAME_ACK = 0x41, FRAME_JSON = 0x4a;
+
+/**
+ * @param charset the charset to decode bytes from the frame
+ */
+public BeatsDecoder(final Charset charset) {
+this(charset, new ByteArrayOutputStream(4096));
--- End diff --
I suspect so. I tested with both smaller and larger values and they all
work ok. Happy to convert into a property tho so it can be adjusted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1418#discussion_r103652588
--- Diff:
nifi-nar-bundles/nifi-beats-bundle/nifi-beats-processors/src/main/java/org/apache/nifi/processors/beats/frame/BeatsDecoder.java
---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.beats.frame;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.zip.InflaterInputStream;
+
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Decodes a Beats frame by maintaining a state based on each byte that
has been processed. This class
+ * should not be shared by multiple threads.
+ */
+public class BeatsDecoder {
+
+
+static final Logger logger =
LoggerFactory.getLogger(BeatsDecoder.class);
+
+private BeatsFrame.Builder frameBuilder;
+private BeatsState currState = BeatsState.VERSION;
+private byte decodedFrameType;
+
+private byte[] unprocessedData;
+
+private final Charset charset;
+private final ByteArrayOutputStream currBytes;
+
+private long windowSize;
+
+static final int MIN_FRAME_HEADER_LENGTH = 2; // Version + Type
+static final int WINDOWSIZE_LENGTH = MIN_FRAME_HEADER_LENGTH + 4; //
32bit unsigned window size
+static final int COMPRESSED_MIN_LENGTH = MIN_FRAME_HEADER_LENGTH + 4;
// 32 bit unsigned + payload
+static final int JSON_MIN_LENGTH = MIN_FRAME_HEADER_LENGTH + 8; // 32
bit unsigned sequence number + 32 bit unsigned payload length
+
+public static final byte FRAME_WINDOWSIZE = 0x57, FRAME_DATA = 0x44,
FRAME_COMPRESSED = 0x43, FRAME_ACK = 0x41, FRAME_JSON = 0x4a;
+
+/**
+ * @param charset the charset to decode bytes from the frame
+ */
+public BeatsDecoder(final Charset charset) {
+this(charset, new ByteArrayOutputStream(4096));
--- End diff --
correction: passing as attribute will be painful. 4096 seems to be the
value of choice around NiFi. Small values also work so I would reckon except
for performance, no significant issue should be cause by the default
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1418#discussion_r103653787
--- Diff:
nifi-nar-bundles/nifi-beats-bundle/nifi-beats-processors/src/main/java/org/apache/nifi/processors/beats/ListenBeats.java
---
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.beats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.BlockingQueue;
+
+import javax.net.ssl.SSLContext;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.attributes.FlowFileAttributeKey;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import
org.apache.nifi.processor.util.listen.AbstractListenEventBatchingProcessor;
+import
org.apache.nifi.processor.util.listen.dispatcher.AsyncChannelDispatcher;
+import org.apache.nifi.processor.util.listen.dispatcher.ChannelDispatcher;
+import
org.apache.nifi.processor.util.listen.dispatcher.SocketChannelDispatcher;
+import org.apache.nifi.processor.util.listen.event.EventFactory;
+import org.apache.nifi.processor.util.listen.handler.ChannelHandlerFactory;
+import org.apache.nifi.processor.util.listen.response.ChannelResponder;
+import org.apache.nifi.processor.util.listen.response.ChannelResponse;
+import org.apache.nifi.processors.beats.event.BeatsEvent;
+import org.apache.nifi.processors.beats.event.BeatsEventFactory;
+import org.apache.nifi.processors.beats.frame.BeatsEncoder;
+import
org.apache.nifi.processors.beats.handler.BeatsSocketChannelHandlerFactory;
+import org.apache.nifi.processors.beats.response.BeatsChannelResponse;
+import org.apache.nifi.processors.beats.response.BeatsResponse;
+import org.apache.nifi.ssl.SSLContextService;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"listen", "beats", "tcp", "logs"})
+@CapabilityDescription("Listens for messages sent by libbeat compatible
clients (e.g. filebeats, metricbeats, etc) being sent to a given port over TCP,
writing its contents in" +
+"JSON format to the content of the message to a FlowFile." +
+"This processor replaces the now deprecated ListenLumberjack")
+@WritesAttributes({
+@WritesAttribute(attribute = "beats.sender", description = "The
sending host of the messages."),
+@WritesAttribute(attribute = "beats.port", description = "The sending
port the messages were received over."),
+@WritesAttribute(attribute = "beats.sequencenumber", description =
"The sequence number of the message. Only included if is 1."),
+@WritesAttribute(attribute = "mime.type", description = "The mime.type
of the content which is application/json")
+})
+@SeeAlso(classNames = {"org.apache.nifi.processors.standard.ParseSyslog"})
+public class ListenBeats extends
AbstractListenEventBatchingProcessor {
+
+
[GitHub] nifi issue #1541: NIFI-329 - Introduce IRC Client Services and ConsumeIRC pr...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1541 processor logic is theoretically complete. Now finishing the IRC over TLS logic --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1016#discussion_r103875353 --- Diff: nifi-nar-bundles/nifi-jmx-bundle/nifi-jmx-processors/pom.xml --- @@ -0,0 +1,67 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +org.apache.nifi +nifi-jmx-bundle +1.2.0-SNAPSHOT + +4.0.0 +nifi-jmx-processors +jar + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-processor-utils + + +org.apache.nifi +nifi-utils + + +org.glassfish +javax.json --- End diff -- Is there any chance you can use gson or any other gson JAR already used by other nifi processors? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1016 Looks good. Will review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1016#discussion_r103875643
--- Diff:
nifi-nar-bundles/nifi-jmx-bundle/nifi-jmx-processors/src/main/java/org/apache/nifi/processors/jmx/GetJMX.java
---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.jmx;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import javax.json.Json;
+import javax.json.JsonBuilderFactory;
+import javax.management.MBeanAttributeInfo;
+import javax.management.MBeanInfo;
+import javax.management.MBeanServerConnection;
+import javax.management.ObjectName;
+import javax.management.remote.JMXConnector;
+import javax.management.remote.JMXConnectorFactory;
+import javax.management.remote.JMXServiceURL;
+import javax.management.ReflectionException;
+import javax.management.InstanceNotFoundException;
+import javax.management.IntrospectionException;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.sql.Timestamp;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.Map;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.ListIterator;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+@Tags({"JMX"})
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@SeeAlso({})
+@CapabilityDescription(
+"Connects to the JMX RMI Url on the configured hostname and port.
"
++ "All domains are queried and can be filtered by
providing the full domain name "
++ "and optional MBean type as whitelist or blacklist
parameters.\n\n"
++ "Blacklist example to exclude all types that start with
'Memory' and GarbageCollector from the "
++ "java.lang domain and everything from the
java.util.logging domain:"
++ "\n\njava.lang:Memory.*
GarbageCollector,java.util.logging")
+@WritesAttributes({
+@WritesAttribute(attribute="hostname", description="The name of
the host that the object originates."),
+@WritesAttribute(attribute="port", description="The JMX connection
port that the object originates."),
+@WritesAttribute(attribute="timestamp", description="The timestamp
of when the object was emitted.") })
+
+public class GetJMX extends AbstractProcessor {
+
+public static final PropertyDescriptor HOSTNAME =
[GitHub] nifi pull request #1016: NIFI-2724 New JMX Processor
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1016#discussion_r103875149 --- Diff: nifi-nar-bundles/nifi-jmx-bundle/nifi-jmx-processors/pom.xml --- @@ -0,0 +1,67 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +org.apache.nifi +nifi-jmx-bundle +1.2.0-SNAPSHOT + +4.0.0 +nifi-jmx-processors +jar + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-processor-utils + + +org.apache.nifi +nifi-utils + + +org.glassfish +javax.json +1.0.4 + + +org.apache.nifi +nifi-mock +test + + +commons-io +commons-io +test + + +org.slf4j +slf4j-simple +test + + + + + +org.apache.rat +apache-rat-plugin --- End diff -- RAT should be inherited from the project pom --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1551: NIFI-3054 - Create default all default directions a...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1551 NIFI-3054 - Create default all default directions as part of maven-as⦠â¦sembly Update Admin Guide with recommended directory permissions Update some minor file permissions Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] ~~~Have you written or updated unit tests to verify your changes?~~~ - [X] ~~~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~~~ - [X] ~~~If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?~~~ - [X] ~~~If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?~~~ - [X] ~~~If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?~~~ ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-3054 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1551.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1551 commit ae16017443b7e59cf6804c1240ef63decd7e3aed Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2017-03-02T11:11:14Z NIFI-3054 - Create default all default directions as part of maven-assembly Update Admin Guide with recommended directory permissions Update some minor file permissions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1553 NIFI-1449 - Migrate PutEmail tests from Mock class to ListenSMTP Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-1449 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1553.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1553 commit 707b7aa7d8285f2e199ab5b0a03e4bbcc3340855 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2017-03-02T13:23:26Z NIFI-1449 - Migrate PutEmail tests from Mock class to ListenSMTP --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1552: NIFI-3544 - Fixes minor typo n ExtractEmailHeaders ...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1552 NIFI-3544 - Fixes minor typo n ExtractEmailHeaders log message Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] ~~~Have you written or updated unit tests to verify your changes?~~~ - [X] ~~~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~~~ - [X] ~~~If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?~~~ - [X] ~~~If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?~~~ - [X] ~~~If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?~~~ ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-3544 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1552.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1552 commit 5e901c3d9a33a528f106d8901460b990f4d523f8 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2017-03-02T13:25:36Z NIFI-3544 - Fixes minor typo n ExtractEmailHeaders log message --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1553: NIFI-1449 - Migrate PutEmail tests from Mock class to List...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1553 @joewitt - Perhaps you would you be able to give me some insight here. For some reason when I run the following lines of testOutgoingMessageAttachment ``` extractAttachmentsRunner.enqueue(message); extractAttachmentsRunner.run(); ``` I hit the following error message: ``` [pool-5-thread-1] ERROR org.apache.nifi.processors.email.ExtractEmailAttachments - ExtractEmailAttachments[id=a209646b-88f5-4a59-bafb-7aba063ed481] Could not parse the flowfile FlowFile[0,632793727136184.mockFlowFile,917B] as an email, treating as failure: java.lang.IllegalStateException: FlowFile[0,632793806568613.mockFlowFile,0B] already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed [pool-5-thread-1] ERROR org.apache.nifi.processors.email.ExtractEmailAttachments - ExtractEmailAttachments[id=a209646b-88f5-4a59-bafb-7aba063ed481] ExtractEmailAttachments[id=a209646b-88f5-4a59-bafb-7aba063ed481] failed to process due to org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,632793727136184.mockFlowFile,917B] is not the most recent version of this flow file within this session; rolling back session: org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,632793727136184.mockFlowFile,917B] is not the most recent version of this flow file within this session java.lang.AssertionError: org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,632793727136184.mockFlowFile,917B] is not the most recent version of this flow file within this session ``` Not sure if this is a bug on ExtractEmailHeaders / ExtractEmailAttachments or if it is a mistake within the test script itself? Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1418#discussion_r103599675
--- Diff:
nifi-nar-bundles/nifi-beats-bundle/nifi-beats-processors/src/main/java/org/apache/nifi/processors/beats/ListenBeats.java
---
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.beats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.BlockingQueue;
+
+import javax.net.ssl.SSLContext;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.attributes.FlowFileAttributeKey;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import
org.apache.nifi.processor.util.listen.AbstractListenEventBatchingProcessor;
+import
org.apache.nifi.processor.util.listen.dispatcher.AsyncChannelDispatcher;
+import org.apache.nifi.processor.util.listen.dispatcher.ChannelDispatcher;
+import
org.apache.nifi.processor.util.listen.dispatcher.SocketChannelDispatcher;
+import org.apache.nifi.processor.util.listen.event.EventFactory;
+import org.apache.nifi.processor.util.listen.handler.ChannelHandlerFactory;
+import org.apache.nifi.processor.util.listen.response.ChannelResponder;
+import org.apache.nifi.processor.util.listen.response.ChannelResponse;
+import org.apache.nifi.processors.beats.event.BeatsEvent;
+import org.apache.nifi.processors.beats.event.BeatsEventFactory;
+import org.apache.nifi.processors.beats.frame.BeatsEncoder;
+import
org.apache.nifi.processors.beats.handler.BeatsSocketChannelHandlerFactory;
+import org.apache.nifi.processors.beats.response.BeatsChannelResponse;
+import org.apache.nifi.processors.beats.response.BeatsResponse;
+import org.apache.nifi.ssl.SSLContextService;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"listen", "beats", "tcp", "logs"})
+@CapabilityDescription("Listens for messages sent by libbeat compatible
clients (e.g. filebeats, metricbeats, etc) being sent to a given port over TCP,
writing its contents in" +
+"JSON format to the content of the message to a FlowFile." +
+"This processor replaces the now deprecated ListenLumberjack")
+@WritesAttributes({
+@WritesAttribute(attribute = "beats.sender", description = "The
sending host of the messages."),
+@WritesAttribute(attribute = "beats.port", description = "The sending
port the messages were received over."),
+@WritesAttribute(attribute = "beats.sequencenumber", description =
"The sequence number of the message. Only included if is 1."),
+@WritesAttribute(attribute = "mime.type", description = "The mime.type
of the content which is application/json")
+})
+@SeeAlso(classNames = {"org.apache.nifi.processors.standard.ParseSyslog"})
+public class ListenBeats extends
AbstractListenEventBatchingProcessor {
+
+
[GitHub] nifi pull request #1418: NIFI-3238 - Introduce ListenBeats processor and dep...
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/1418#discussion_r103599499 --- Diff: nifi-nar-bundles/nifi-beats-bundle/nifi-beats-nar/src/main/resources/META-INF/LICENSE --- @@ -0,0 +1,233 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have ma
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r76220146
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryWhois.java
---
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.net.whois.WhoisClient;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"whois", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful whois query processor primary designed
to enrich DataFlows with whois based APIs " +
+"(e.g. ShadowServer's ASN lookup) but that can be also used to
perform regular whois lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*",
description = "The captured fields of the Whois query response for each of the
records received"),
+})
+public class QueryWhois extends AbstractEnrichProcessor {
+
+public static final AllowableValue BEGIN_END = new
AllowableValue("Begin/End", "Begin/End",
+"The evaluated input of each flowfile is enclosed within begin
and end tags. Each row contains a delimited set of fields");
+
+public static final AllowableValue BULK_NONE = new
AllowableValue("None", "None",
+"Queries are made without any particular dialect");
+
+
+public static final PropertyDescriptor WHOIS_QUERY_TYPE = new
PropertyDescriptor.Builder()
+.name("WHOIS_QUERY_TYPE")
+.displayName("Whois Query Type")
+.description("The Whois query type to be used by the processor
(if used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER")
+.displayName("Whois Server")
+.description("The Whois server to be used")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER_PORT = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER_PORT")
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r76225271
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryWhois.java
---
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.net.whois.WhoisClient;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"whois", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful whois query processor primary designed
to enrich DataFlows with whois based APIs " +
+"(e.g. ShadowServer's ASN lookup) but that can be also used to
perform regular whois lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*",
description = "The captured fields of the Whois query response for each of the
records received"),
+})
+public class QueryWhois extends AbstractEnrichProcessor {
+
+public static final AllowableValue BEGIN_END = new
AllowableValue("Begin/End", "Begin/End",
+"The evaluated input of each flowfile is enclosed within begin
and end tags. Each row contains a delimited set of fields");
+
+public static final AllowableValue BULK_NONE = new
AllowableValue("None", "None",
+"Queries are made without any particular dialect");
+
+
+public static final PropertyDescriptor WHOIS_QUERY_TYPE = new
PropertyDescriptor.Builder()
+.name("WHOIS_QUERY_TYPE")
+.displayName("Whois Query Type")
+.description("The Whois query type to be used by the processor
(if used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER")
+.displayName("Whois Server")
+.description("The Whois server to be used")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER_PORT = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER_PORT")
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r76225290
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryWhois.java
---
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.net.whois.WhoisClient;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"whois", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful whois query processor primary designed
to enrich DataFlows with whois based APIs " +
+"(e.g. ShadowServer's ASN lookup) but that can be also used to
perform regular whois lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*",
description = "The captured fields of the Whois query response for each of the
records received"),
+})
+public class QueryWhois extends AbstractEnrichProcessor {
+
+public static final AllowableValue BEGIN_END = new
AllowableValue("Begin/End", "Begin/End",
+"The evaluated input of each flowfile is enclosed within begin
and end tags. Each row contains a delimited set of fields");
+
+public static final AllowableValue BULK_NONE = new
AllowableValue("None", "None",
+"Queries are made without any particular dialect");
+
+
+public static final PropertyDescriptor WHOIS_QUERY_TYPE = new
PropertyDescriptor.Builder()
+.name("WHOIS_QUERY_TYPE")
+.displayName("Whois Query Type")
+.description("The Whois query type to be used by the processor
(if used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER")
+.displayName("Whois Server")
+.description("The Whois server to be used")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor WHOIS_SERVER_PORT = new
PropertyDescriptor.Builder()
+.name("WHOIS_SERVER_PORT")
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 - reworded validators, unified the regex / split parses to use numeric capture groups/columns --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #940: NIFI-2662 - Fix checkstyle violations introduced by ...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/940 NIFI-2662 - Fix checkstyle violations introduced by NIFI-2605 You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-2662 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/940.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #940 commit 57aa756206e75270685197face09eac860451747 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2016-08-25T12:19:25Z NIFI-2662 - Fix checkstyle violations introduced by NIFI-2605 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #940: NIFI-2662 - Fix checkstyle violations introduced by NIFI-26...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/940 previous title and commit message referred to incorrect JIRA number. Correct number according to Github's blame view is NIFI-2635 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #941: NIFI-2662 - Fix style violations previously missed
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/941 NIFI-2662 - Fix style violations previously missed Fix styles missed during PR940 You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-2662pt2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/941.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #941 commit d126b78b8b3eb8457f0de2ed4cde17a2cce67462 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2016-08-25T14:08:33Z NIFI-2662 - Fix style violations previously missed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #941: NIFI-2662 - Fix style violations previously missed
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/941 @mattyb149 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #785: NIFI-2341 - Introduce ParseCEF processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/785 @mattyb149 feedback addressed. My comment is to play particular attention on threat concurrency around Jackson's ObjectMapper (jackson say it is threat safe but thread safety is not my forté) and the fiddling with timezones (that has played tricks in the past). I will continue testing but overall I suspect it should be ok to review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #785: NIFI-2341 - Introduce ParseCEF processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/785 Rofl. Well at least I got 1 out 3 right. As you can see we security folks are a bit obsessed with threats and risks :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #985: NIFI-2728 - Attempt to fix travis-ci build woes
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/985 NIFI-2728 - Attempt to fix travis-ci build woes You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-2728 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/985.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #985 commit 18ff080bba7369b4e64a65913a617ba91e94964e Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2016-09-03T11:37:59Z NIFI-2728 - Attempt to fix travis-ci build woes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/985 @mcgilman can you please review this and if in agreement merge it to master? Current travis-ci settings seem to be unable to reliably build nifi. I fiddled with Xmx settings, enabled caching and added a license to the .travis.yml file and it seems to have alleviated the situation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/985 Seems like moving `mvn -T2.0C clean install` to travis install phase and `mvn checkstyle:check` to the script phase did the trick. around 25% reduction in build time and a far more reliable build process than previously, while preserving `-PContrib-check` single-threaded execution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/985 @apiri makes complete sense. afaik this can be achieved using `mvn apache-rat:check and checkstyle:check ` so I added the above command to the before_install stage of the travis job. This should "fast fail" jobs failing to comply with development guidelines and by consequence, reduce a bit the load on travis-ci infrastructure. Very good call. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #817: NIFI-2380 - Introduce ExtractTNEFAttachments
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/817#discussion_r77625743
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractTNEFAttachments.java
---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.FlowFileHandlingException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.poi.hmef.Attachment;
+import org.apache.poi.hmef.HMEFMessage;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+
+@SupportsBatching
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The
filename of the attachment"),
+@WritesAttribute(attribute =
"email.tnef.attachment.parent.filename ", description = "The filename of the
parent FlowFile"),
+@WritesAttribute(attribute = "email.tnef.attachment.parent.uuid",
description = "The UUID of the original FlowFile.")})
+
+public class ExtractTNEFAttachments extends AbstractProcessor {
+public static final String ATTACHMENT_ORIGINAL_FILENAME =
"email.tnef.attachment.parent.filename";
+public static final String ATTACHMENT_ORIGINAL_UUID =
"email.tnef.attachment.parent.uuid";
+
+public static final Relationship REL_ATTACHMENTS = new
Relationship.Builder()
+.name("attachments")
+.description("Each individual attachment will be routed to the
attachments relationship")
+.build();
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder()
+.name("original")
+.description("The original file")
+.build();
+public static final Relationship REL_FAILURE = new
Relationship.Builder()
+.name("failure")
+.description("Flowfiles that could not be parsed")
+.build();
+private Set relationships;
[GitHub] nifi pull request #817: NIFI-2380 - Introduce ExtractTNEFAttachments
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/817#discussion_r77625649
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractTNEFAttachments.java
---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.email;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.FlowFileHandlingException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.poi.hmef.Attachment;
+import org.apache.poi.hmef.HMEFMessage;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+
+@SupportsBatching
+@EventDriven
+@SideEffectFree
+@Tags({"split", "email"})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Extract attachments from a mime formatted email
file, splitting them into individual flowfiles.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename ", description = "The
filename of the attachment"),
+@WritesAttribute(attribute =
"email.tnef.attachment.parent.filename ", description = "The filename of the
parent FlowFile"),
+@WritesAttribute(attribute = "email.tnef.attachment.parent.uuid",
description = "The UUID of the original FlowFile.")})
+
+public class ExtractTNEFAttachments extends AbstractProcessor {
+public static final String ATTACHMENT_ORIGINAL_FILENAME =
"email.tnef.attachment.parent.filename";
+public static final String ATTACHMENT_ORIGINAL_UUID =
"email.tnef.attachment.parent.uuid";
+
+public static final Relationship REL_ATTACHMENTS = new
Relationship.Builder()
+.name("attachments")
+.description("Each individual attachment will be routed to the
attachments relationship")
+.build();
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder()
+.name("original")
+.description("The original file")
+.build();
+public static final Relationship REL_FAILURE = new
Relationship.Builder()
+.name("failure")
+.description("Flowfiles that could not be parsed")
+.build();
+private Set relationships;
[GitHub] nifi pull request #960: NIFI-2685 - Configure maven to use java headless mod...
Github user trixpan closed the pull request at: https://github.com/apache/nifi/pull/960 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #991: NIFI-2742 - Explicitly declare the kite hadoop dependencies...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/991 @rdblue would you have any feedback over this change? Not ideal but I couldn't find a better way. Open to suggestions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #991: NIFI-2742 - Explicitly declare the kite hadoop dependencies...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/991 @joewitt I will be happy to address that! Changing the master LICENSE and NOTICE files is one of my part time hobbies! :neutral_face: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #990: NIFI-2744 - turn hbase-client version into a pom.xml...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/990 NIFI-2744 - turn hbase-client version into a pom.xml property You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-2744 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/990.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #990 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #990: NIFI-2744 - turn hbase-client version into a pom.xml proper...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/990 @mattyb149 seems like this one is also required to address @xmlking concerns around MapR-DB --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/985 Matt, the profile is still being run but it run after compilation. First (install) travis performs a multi threaded mvn build. Immediately aftet this completes it runs a mvn checkstyle:check with a single thread. This allows the best of both worlds: speedy compilation with ongoing style and licensing enforcement --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/985 @mcgillman, what do you think about leaving the - T2.0C setting to speed up build and adding an extra post-compile step to run - Pcontrib-check single threaded (this is similar to spark's approach)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #817: NIFI-2380 - Introduce ExtractTNEFAttachments
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/817 @olegz Rebased. Please note that while I will be happy to keep rebasing as peer review is in progress, this PR makes additions to the nifi-assembly LICENSE and NOTICE files so it is expected it will continue experience merge conflicts, as other PRs try to change the same file. Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #990: NIFI-2744 - turn hbase-client version into a pom.xml...
Github user trixpan closed the pull request at: https://github.com/apache/nifi/pull/990 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #990: NIFI-2744 - turn hbase-client version into a pom.xml proper...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/990 No worries --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #980: NIFI-1170 - Improved TailFile processor to support multiple...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/980 Let me know when this is ready to test and I will give it a go --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #817: NIFI-2380 - Introduce ExtractTNEFAttachments
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/817 @olegz how is testing looking like? Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 any chance of completing this PR ? :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1003: NIFI-2755 - Fixes minor typo in Developers Guide
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1003 NIFI-2755 - Fixes minor typo in Developers Guide You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-2755 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1003.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1003 commit 0641bfd688cd1408b4963239a25140bc846763d1 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2016-09-11T15:04:11Z NIFI-2755 - Fixes minor typo in Developers Guide --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1004: NIFI-2756 - Add displayName to maven archetypes
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1004 NIFI-2756 - Add displayName to maven archetypes @alopresto Would you mind having a look on this PR? I am not familiar with maven archetypes but I noticed that the ones we have seem to result in code lacking some of the present practices such as having displayName, including * imports, unused classes, etc. This PR tries to address that. Cheer You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-2756 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1004.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1004 commit 7f4436286713dcbb303114eeb5f1fd8fc8ef9799 Author: Andre F de Miranda <trix...@users.noreply.github.com> Date: 2016-09-11T15:49:01Z NIFI-2756 - Add displayName to maven archetypes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 feedback addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #980: NIFI-1170 - Improved TailFile processor to support m...
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/980#discussion_r78770355
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java
---
@@ -117,31 +173,78 @@
.allowableValues(LOCATION_LOCAL, LOCATION_REMOTE)
.defaultValue(LOCATION_LOCAL.getValue())
.build();
+
static final PropertyDescriptor START_POSITION = new
PropertyDescriptor.Builder()
.name("Initial Start Position")
-.description("When the Processor first begins to tail data,
this property specifies where the Processor should begin reading data. Once
data has been ingested from the file, "
+.description("When the Processor first begins to tail data,
this property specifies where the Processor should begin reading data. Once
data has been ingested from a file, "
+ "the Processor will continue from the last point
from which it has received data.")
.allowableValues(START_BEGINNING_OF_TIME, START_CURRENT_FILE,
START_CURRENT_TIME)
.defaultValue(START_CURRENT_FILE.getValue())
.required(true)
.build();
+static final PropertyDescriptor RECURSIVE = new
PropertyDescriptor.Builder()
+.name("tailfile-recursive-lookup")
+.displayName("Recursive lookup")
+.description("When using Multiple files mode, this property
defines if files must be listed recursively or not"
++ " in the base directory.")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.build();
+
+static final PropertyDescriptor ROLLING_STRATEGY = new
PropertyDescriptor.Builder()
+.name("tailfile-rolling-strategy")
+.displayName("Rolling Strategy")
+.description("Specifies if the files to tail have a fixed name
or not.")
+.required(true)
+.allowableValues(FIXED_NAME, CHANGING_NAME)
+.defaultValue(FIXED_NAME.getValue())
+.build();
+
+static final PropertyDescriptor LOOKUP_FREQUENCY = new
PropertyDescriptor.Builder()
+.name("tailfile-lookup-frequency")
+.displayName("Lookup frequency")
+.description("Only used in Multiple files mode and Changing
name rolling strategy, it specifies the minimum "
++ "duration the processor will wait before listing
again the files to tail.")
+.required(false)
+.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.defaultValue("10 minutes")
--- End diff --
@joewitt
I think @pvillard31 has a point when he says the status of a file should
ALWAYS be tracked unless:
1. overwritten / reset by the user (causing data duplication).
2. too old to be relevant (removed automatically)
Under this arrangement, the two timers make sense:
1 - Maximum age of file - if file is older than this date it won't be
tailed. (_this happens to be very similar to Heka's approach as well_)
2 - How frequently to harvest for new files - self explanatory
2b - if new file is found tail. If file is pre-existent and is older than
max age remove status;
In addition, we could consider what flume-ng taildir called an idle
timeout,
`idleTimeout - Time (ms) to close inactive files. If the closed file is
appended new lines to, this source will automatically re-open it.`
These are files that are younger than maximum age, but largely stagnated.
We would keep their status (until expiry) but they would be closed and only
re-opened if the file size increased (or other tail conditions were be
triggered).
flume-ng tried to deal with resource waste by using and increasing delay to
poll the idle files. The higher the number of polls without new data, the
longer it would take before a new retry. Not sure if this is something we would
like to do but would also help.
Inevitably, most teams using date based naming conventions do that to
prevent performing truncation of a file when logrotate runs and I suspect we
should simply let the user know that having too many files in the same folder,
matching the same URL would cause impact to performance and that compressing
them so they don't match the file regex, or moving them to other directories in
order to minimise resource waste.
Hope this makes sense
---
I
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r78476147
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
---
@@ -65,7 +70,16 @@
.description("Choice between a splitter and regex matcher used
to parse the results of the query into attribute groups")
.expressionLanguageSupported(false)
.required(false)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor KEY_GROUP = new
PropertyDescriptor.Builder()
+.name("KEY_GROUP")
+.displayName("Key lookup group (multiline / batch)")
+.description("When performing a batched lookup, the following
RegEx named capture group or Column number will be used to match" +
+"the whois server response with the lookup field")
--- End diff --
good catch. addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r78475823
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
---
@@ -152,10 +166,68 @@
// Fails to NONE
default:
// NONE was chosen, just appending the record result as
group0 without further splitting
-results.put("enrich." + schema + ".record" +
String.valueOf(recordPosition) + ".group0", rawResult);
+results.put("enrich." + schema + ".record" +
recordPosition + ".group0", rawResult);
break;
}
return results;
}
+/**
+ * This method returns the parsed record string in the form of
+ * a map of two strings, consisting of a iteration aware attribute
+ * names and its values
+ *
+
+ * @param rawResult the raw query results to be parsed
+ * @param queryParser The parsing mechanism being used to parse the
data into groups
+ * @param queryRegex The regex to be used to split the query results
into groups. The regex MUST implement at least on named capture group "KEY" to
be used to populate the table rows
+ * @param lookupKey The regular expression named capture group or
number of the column of a split to be used for matching
+ * @return Table with attribute names and values where each Table row
uses the value of the KEY named capture group specified in @param queryRegex
+ */
+protected Table<String, String, String> parseBatchResponse(String
rawResult, String queryParser, String queryRegex, String lookupKey, String
schema) {
+// Note the hardcoded record0.
+// Since iteration is done within the parser and Multimap is
used, the record number here will always be 0.
+// Consequentially, 0 is hardcoded so that batched and non batched
attributes follow the same naming
+// conventions
+final String recordPosition = ".record0";
+
+final Table<String, String, String> results =
HashBasedTable.create();
+
+switch (queryParser) {
+case "Split":
+Scanner scanner = new Scanner(rawResult);
+while (scanner.hasNextLine()) {
+String line = scanner.nextLine();
+// Time to Split the results...
+String[] splitResult = line.split(queryRegex);
+
+for (int r = 0; r < splitResult.length; r++) {
+results.put(splitResult[
Integer.valueOf(lookupKey) - 1 ], "enrich." + schema + recordPosition +
".group" + String.valueOf(r), splitResult[r]);
+
+}
+}
+break;
+case "RegEx":
+// prepare the regex
+Pattern p;
+// Regex is multiline. Each line should include a KEY for
lookup
+p = Pattern.compile(queryRegex, Pattern.MULTILINE);
+
+Matcher matcher = p.matcher(rawResult);
+while (matcher.find()) {
+// Note that RegEx matches capture group 0 is usually
broad but starting with it anyway
+// for the sake of purity
+for (int r = 0; r <= matcher.groupCount(); r++) {
+if (!StringUtils.isEmpty(matcher.group("KEY"))) {
--- End diff --
Pierre,
This KEY reference should have been removed so I guess I stuffed up
something on my git... I rewrote the code to unify on regex capture group
numbers.
I also add code to catch IndexOutOfBoundsException (the equivalent to the
IllegalArgumentException you mentioned above)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/858#discussion_r78476100
--- Diff:
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
---
@@ -152,10 +166,68 @@
// Fails to NONE
default:
// NONE was chosen, just appending the record result as
group0 without further splitting
-results.put("enrich." + schema + ".record" +
String.valueOf(recordPosition) + ".group0", rawResult);
+results.put("enrich." + schema + ".record" +
recordPosition + ".group0", rawResult);
break;
}
return results;
}
+/**
+ * This method returns the parsed record string in the form of
+ * a map of two strings, consisting of a iteration aware attribute
+ * names and its values
+ *
+
+ * @param rawResult the raw query results to be parsed
+ * @param queryParser The parsing mechanism being used to parse the
data into groups
+ * @param queryRegex The regex to be used to split the query results
into groups. The regex MUST implement at least on named capture group "KEY" to
be used to populate the table rows
+ * @param lookupKey The regular expression named capture group or
number of the column of a split to be used for matching
+ * @return Table with attribute names and values where each Table row
uses the value of the KEY named capture group specified in @param queryRegex
+ */
+protected Table<String, String, String> parseBatchResponse(String
rawResult, String queryParser, String queryRegex, String lookupKey, String
schema) {
+// Note the hardcoded record0.
+// Since iteration is done within the parser and Multimap is
used, the record number here will always be 0.
+// Consequentially, 0 is hardcoded so that batched and non batched
attributes follow the same naming
+// conventions
+final String recordPosition = ".record0";
+
+final Table<String, String, String> results =
HashBasedTable.create();
+
+switch (queryParser) {
+case "Split":
+Scanner scanner = new Scanner(rawResult);
+while (scanner.hasNextLine()) {
+String line = scanner.nextLine();
+// Time to Split the results...
+String[] splitResult = line.split(queryRegex);
+
+for (int r = 0; r < splitResult.length; r++) {
+results.put(splitResult[
Integer.valueOf(lookupKey) - 1 ], "enrich." + schema + recordPosition +
".group" + String.valueOf(r), splitResult[r]);
+
+}
+}
+break;
+case "RegEx":
+// prepare the regex
+Pattern p;
+// Regex is multiline. Each line should include a KEY for
lookup
+p = Pattern.compile(queryRegex, Pattern.MULTILINE);
+
+Matcher matcher = p.matcher(rawResult);
+while (matcher.find()) {
+// Note that RegEx matches capture group 0 is usually
broad but starting with it anyway
+// for the sake of purity
+for (int r = 0; r <= matcher.groupCount(); r++) {
+if (!StringUtils.isEmpty(matcher.group("KEY"))) {
+results.put(matcher.group(lookupKey), "enrich." +
schema + recordPosition + ".group" + String.valueOf(r), matcher.group(r));
+} else {
+getLogger().warn("Could not find group {} while
processing result. Ignoring row", new Object[] {lookupKey});
--- End diff --
great idea. Addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 can you confirm if this was generated by Split or Regex capture? Note that the regex is a multiline match, therefore user can remove the newline via regular expression. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/985 @ijokarumawak --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1025: Trying to fix Travis build failure.
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1025 @ijokarumawak there's already a PR for that keen for you to look at https://github.com/apache/nifi/pull/985 and merge it if you don't mind --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on a diff in the pull request: https://github.com/apache/nifi/pull/985#discussion_r78266044 --- Diff: .travis.yml --- @@ -6,12 +21,22 @@ os: jdk: - oraclejdk8 +# Caches mvn repository in order to speed upbuilds +cache: + directories: + - $HOME/.m2 + - $HOME/.npm + # before_install aids in a couple workarounds for issues within the Travis-CI environment # 1. Workaround for buffer overflow issues with OpenJDK versions of java as per https://github.com/travis-ci/travis-ci/issues/5227#issuecomment-165135711 # 2. Workaround for non-existent Maven repository as per: https://github.com/travis-ci/travis-ci/issues/4629#issuecomment-131214465 before_install: - sudo hostname "$(hostname | cut -c1-63)" - sed -e "s/^\\(127\\.0\\.0\\.1.*\\)/\\1 $(hostname | cut -c1-63)/" /etc/hosts | sudo tee /etc/hosts - sed -i.bak -e 's|https://nexus.codehaus.org/snapshots/|https://oss.sonatype.org/content/repositories/codehaus-snapshots/|g' ~/.m2/settings.xml + - echo "MAVEN_OPTS='-Xmx2g -XX:+UseG1GC'" > ~/.mavenrc + - mvn -Pcontrib-check apache-rat:check checkstyle:check --- End diff -- @apiri Yeah... that was a bit weird but it was an attempted workaround run checkstyle:check before compilation time. Sadly it didn't work, when we run checkstyle check, it triggers some style errors that have been disabled in our style rules: `[WARNING] src/main/java/org/apache/nifi/reporting/ReportingTask.java[64:8] (javadoc) JavadocMethod: Unable to get class information for @throws tag 'org.apache.nifi.reporting.InitializationException'.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #985: NIFI-2728 - Attempt to fix travis-ci build woes
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/985 @apiri, @mcgilman I did a few new tweaks so that job sequence is: 1. rat check (before_install) 2. parallel compile with skipTests (install) 3. style check and test via mvn verify (script) As mentioned above previously I could not find a way of running `checkstyle:check` before compilation, while respecting our checkstyle rules so to prevent errors such as: `[WARNING] src/main/java/org/apache/nifi/reporting/ReportingTask.java[64:8] (javadoc) JavadocMethod: Unable to get class information for @throws tag 'org.apache.nifi.reporting.InitializationException'.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1002: NIFI-2684 - Change AbstractConfigurableComponent to use di...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1002 @joewitt this commit is quite small (intent is just aesthetics / UX) but ventures into waters I don't usually sail... Would you mind having a look? I thank you in advance --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1057: NIFI-2816 - Clean typos across the code
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1057 @pvillard31 note to myself: Never try to hunt typos after midnight ... :smiley: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
