[jira] [Commented] (NIFI-5166) Create deep learning classification and regression processor
[ https://issues.apache.org/jira/browse/NIFI-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518821#comment-16518821 ] ASF GitHub Bot commented on NIFI-5166: -- Github user mans2singh commented on the issue: https://github.com/apache/nifi/pull/2686 @ijokarumawak - Just following-up: 1. Regarding implementing record look up service - I believe that the processor and a record service lookup can be separate components useful for different use cases. As the file/rdbms based flows I've mentioned above, show - the processor can be used as a transformer. 2. Regarding providing more tools to prepare data - You are right, we can do that once we have the basics in place and there is a need for it. 3. You had mentioned the concern of writing the results (predictions) in the body of the flow file - if you/community think we should keep the observations in the body and put the output in an attribute, I'd be happy to change that. Thanks again for your thoughts and let me know if you have any more advice/recommendations. Mans > Create deep learning classification and regression processor > > > Key: NIFI-5166 > URL: https://issues.apache.org/jira/browse/NIFI-5166 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: Learning, classification,, deep, regression, > Original Estimate: 168h > Remaining Estimate: 168h > > We need a deep learning classification and regression processor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2686: NIFI-5166 - Deep learning classification and regression pr...
Github user mans2singh commented on the issue: https://github.com/apache/nifi/pull/2686 @ijokarumawak - Just following-up: 1. Regarding implementing record look up service - I believe that the processor and a record service lookup can be separate components useful for different use cases. As the file/rdbms based flows I've mentioned above, show - the processor can be used as a transformer. 2. Regarding providing more tools to prepare data - You are right, we can do that once we have the basics in place and there is a need for it. 3. You had mentioned the concern of writing the results (predictions) in the body of the flow file - if you/community think we should keep the observations in the body and put the output in an attribute, I'd be happy to change that. Thanks again for your thoughts and let me know if you have any more advice/recommendations. Mans ---
[GitHub] nifi pull request #2686: NIFI-5166 - Deep learning classification and regres...
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2686#discussion_r196994774
--- Diff:
nifi-nar-bundles/nifi-deeplearning4j-bundle/nifi-deeplearning4j-processors/src/main/java/org/apache/nifi/processors/deeplearning4j/DeepLearning4JMultiLayerPredictor.java
---
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.deeplearning4j;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
+import org.deeplearning4j.util.ModelSerializer;
+import org.nd4j.linalg.api.ndarray.INDArray;
+import org.nd4j.linalg.factory.Nd4j;
+import com.google.gson.Gson;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"deeplearning4j", "dl4j", "multilayer", "predict",
"classification", "regression", "deep", "learning", "neural", "network"})
+@CapabilityDescription("The DeepLearning4JMultiLayerPredictor predicts one
or more value(s) based on provided deeplearning4j
(https://github.com/deeplearning4j) model and the content of a FlowFile. "
++ "The processor supports both classification and regression by
extracting the record from the FlowFile body and applying the model. "
++ "The processor supports batch by allowing multiple records to be
passed in the FlowFile body with each record separated by the 'Record
Separator' property. "
++ "Each record can contain multiple fields with each field separated
by the 'Field Separator' property."
+)
+@WritesAttributes({
+@WritesAttribute(attribute =
AbstractDeepLearning4JProcessor.DEEPLEARNING4J_ERROR_MESSAGE, description =
"Deeplearning4J error message"),
+@WritesAttribute(attribute =
AbstractDeepLearning4JProcessor.DEEPLEARNING4J_OUTPUT_SHAPE, description =
"Deeplearning4J output shape"),
+})
+public class DeepLearning4JMultiLayerPredictor extends
AbstractDeepLearning4JProcessor {
+
+static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("Successful DeepLearning4j results are routed to
this relationship").build();
+
+static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("Failed DeepLearning4j results are routed to this
relationship").build();
+
+protected final Gson gson = new Gson();
+
+protected MultiLayerNetwork model = null;
+
+@OnStopped
+public void close() {
+getLogger().info("Closing");
+model = null;
+}
+
+private static final Set relationships;
+private static final List propertyDescriptors;
+static {
[jira] [Commented] (NIFI-5166) Create deep learning classification and regression processor
[
https://issues.apache.org/jira/browse/NIFI-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518809#comment-16518809
]
ASF GitHub Bot commented on NIFI-5166:
--
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2686#discussion_r196994774
--- Diff:
nifi-nar-bundles/nifi-deeplearning4j-bundle/nifi-deeplearning4j-processors/src/main/java/org/apache/nifi/processors/deeplearning4j/DeepLearning4JMultiLayerPredictor.java
---
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.deeplearning4j;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
+import org.deeplearning4j.util.ModelSerializer;
+import org.nd4j.linalg.api.ndarray.INDArray;
+import org.nd4j.linalg.factory.Nd4j;
+import com.google.gson.Gson;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"deeplearning4j", "dl4j", "multilayer", "predict",
"classification", "regression", "deep", "learning", "neural", "network"})
+@CapabilityDescription("The DeepLearning4JMultiLayerPredictor predicts one
or more value(s) based on provided deeplearning4j
(https://github.com/deeplearning4j) model and the content of a FlowFile. "
++ "The processor supports both classification and regression by
extracting the record from the FlowFile body and applying the model. "
++ "The processor supports batch by allowing multiple records to be
passed in the FlowFile body with each record separated by the 'Record
Separator' property. "
++ "Each record can contain multiple fields with each field separated
by the 'Field Separator' property."
+)
+@WritesAttributes({
+@WritesAttribute(attribute =
AbstractDeepLearning4JProcessor.DEEPLEARNING4J_ERROR_MESSAGE, description =
"Deeplearning4J error message"),
+@WritesAttribute(attribute =
AbstractDeepLearning4JProcessor.DEEPLEARNING4J_OUTPUT_SHAPE, description =
"Deeplearning4J output shape"),
+})
+public class DeepLearning4JMultiLayerPredictor extends
AbstractDeepLearning4JProcessor {
+
+static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("Successful DeepLearning4j results are routed to
this relationship").build();
+
+static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("Failed DeepLearning4j results are routed to this
relationship").build();
+
+protected final Gson gson = new Gson();
+
+protected MultiLayerNetwork model = null;
+
+
[jira] [Commented] (MINIFICPP-540) add link to issue tracker from README.md
[ https://issues.apache.org/jira/browse/MINIFICPP-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518782#comment-16518782 ] ASF GitHub Bot commented on MINIFICPP-540: -- GitHub user dtrodrigues opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/362 MINIFICPP-540 - add issue tracker link to README.md Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dtrodrigues/nifi-minifi-cpp MINIFICPP-540 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/362.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #362 commit 91f4607a0edb60869e6036450f7f7d046528adbe Author: Dustin Rodrigues Date: 2018-06-21T01:22:16Z MINIFICPP-540 - add issue tracker link to README.md > add link to issue tracker from README.md > > > Key: MINIFICPP-540 > URL: https://issues.apache.org/jira/browse/MINIFICPP-540 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Dustin Rodrigues >Assignee: Dustin Rodrigues >Priority: Minor > > Linking to the apache minifi-cpp issue tracker from the README would allow > users to view/submit issues since the Github issues tab is disabled -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #362: MINIFICPP-540 - add issue tracker link to...
GitHub user dtrodrigues opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/362 MINIFICPP-540 - add issue tracker link to README.md Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dtrodrigues/nifi-minifi-cpp MINIFICPP-540 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/362.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #362 commit 91f4607a0edb60869e6036450f7f7d046528adbe Author: Dustin Rodrigues Date: 2018-06-21T01:22:16Z MINIFICPP-540 - add issue tracker link to README.md ---
[jira] [Created] (MINIFICPP-540) add link to issue tracker from README.md
Dustin Rodrigues created MINIFICPP-540: -- Summary: add link to issue tracker from README.md Key: MINIFICPP-540 URL: https://issues.apache.org/jira/browse/MINIFICPP-540 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Dustin Rodrigues Assignee: Dustin Rodrigues Linking to the apache minifi-cpp issue tracker from the README would allow users to view/submit issues since the Github issues tab is disabled -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-539) Included rocksdb doesn't build on Fedora 28 (gcc 8.1.1)
Dustin Rodrigues created MINIFICPP-539: -- Summary: Included rocksdb doesn't build on Fedora 28 (gcc 8.1.1) Key: MINIFICPP-539 URL: https://issues.apache.org/jira/browse/MINIFICPP-539 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Dustin Rodrigues The version of RocksDB (5.8.6) currently included with minifi does not build on gcc 8.1. This is a known issue and [https://github.com/facebook/rocksdb/pull/3736] appears to fix it. It may also be worth including the version number in the included directory similar to librdkakfa-0.11.4 and rapidjson-1.1.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication
[ https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518668#comment-16518668 ] ASF GitHub Bot commented on NIFI-4838: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2448 Might be shorter to copy the code from AbstractProcessor into AbstractMongoProcessor, change AbstractMongoProcessor to extend AbstractSessionFactoryProcessor, and override onTrigger in GetMongo to call the one with the ProcessSessionFactory. > Make GetMongo support multiple commits and give some progress indication > > > Key: NIFI-4838 > URL: https://issues.apache.org/jira/browse/NIFI-4838 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > It shouldn't wait until the end to do a commit() call because the effect is > that GetMongo looks like it has hung to a user who is pulling a very large > data set. > It should also have an option for running a count query to get the current > approximate count of documents that would match the query and append an > attribute that indicates where a flowfile stands in the total result count. > Ex: > query.progress.point.start = 2500 > query.progress.point.end = 5000 > query.count.estimate = 17,568,231 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2448 Might be shorter to copy the code from AbstractProcessor into AbstractMongoProcessor, change AbstractMongoProcessor to extend AbstractSessionFactoryProcessor, and override onTrigger in GetMongo to call the one with the ProcessSessionFactory. ---
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518572#comment-16518572 ] ASF GitHub Bot commented on NIFI-5325: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2805 @bbende I think I addressed all the feedback except for the new msgbody idea > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518569#comment-16518569
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932469
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518571#comment-16518571
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932621
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume
project for RFC3164 logs.
+ * For 5424 we use simple-syslog-5424 since it parsers out structured data.
+ */
+public class StrictSyslog5424Parser {
+private Charset charset;
+private com.github.palindromicity.syslog.SyslogParser parser;
+KeyProvider keyProvider = new DefaultKeyProvider();
--- End diff --
done
> Need a Syslog Parser that fully supports the 5424 Spec
> --
>
> Key: NIFI-5325
> URL: https://issues.apache.org/jira/browse/NIFI-5325
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> from NIFI-5139
> "The Structured Data as described in
> [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi.
>
> I suggest the SD to be put in attributes, prefixed by the ID:
> [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"]
> to become 3 attributes
> * exampleSID@32473-iut = 3
> * exampleSID@32473-eventSource=Application
> * exampleSID@32473-eventID=1011
>
> This could be useful to preprocess RFC5424 message, prioritize or route them."
>
> Nifi should a have a parser to fully support the spec and structured data.
> Since integrating it, and special casing the existing parser would
> simnifically complicate that processor and move it from it's least common
> denominator goal, it should be a new parser
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5424 Str...
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2805 @bbende I think I addressed all the feedback except for the new msgbody idea ---
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518570#comment-16518570 ] ASF GitHub Bot commented on NIFI-5325: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196932588 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.syslog; + +import com.github.palindromicity.syslog.DefaultKeyProvider; +import com.github.palindromicity.syslog.KeyProvider; +import com.github.palindromicity.syslog.NilPolicy; +import com.github.palindromicity.syslog.SyslogParserBuilder; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.regex.Pattern; + +/** + * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance. + * + * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs. --- End diff -- sorry, leftover from my initial SyslogParser modifications > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196932588 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.syslog; + +import com.github.palindromicity.syslog.DefaultKeyProvider; +import com.github.palindromicity.syslog.KeyProvider; +import com.github.palindromicity.syslog.NilPolicy; +import com.github.palindromicity.syslog.SyslogParserBuilder; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.regex.Pattern; + +/** + * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance. + * + * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs. --- End diff -- sorry, leftover from my initial SyslogParser modifications ---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932469
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP address of the Syslog message."),
+@WritesAttribute(attribute = "syslog.appname", description = "The
appname of the Syslog message."),
+@WritesAttribute(attribute = "syslog.procid", description = "The
procid of the Syslog message."),
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932621
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume
project for RFC3164 logs.
+ * For 5424 we use simple-syslog-5424 since it parsers out structured data.
+ */
+public class StrictSyslog5424Parser {
+private Charset charset;
+private com.github.palindromicity.syslog.SyslogParser parser;
+KeyProvider keyProvider = new DefaultKeyProvider();
--- End diff --
done
---
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518564#comment-16518564 ] ASF GitHub Bot commented on NIFI-5325: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196932176 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE --- @@ -218,6 +218,14 @@ The following binary components are provided under the Apache Software License v Copyright 2006 - 2013 Pentaho Corporation. All rights reserved. Copyright 2000-2005, 2014-2016 Julian Hyde + (ASLv2) simple-syslog-5424 +The following NOTICE information applies: + +simple-syslog-5424 +https://github.com/palindromicity/simple-syslog-5424 + +Copyright 2018 simple-syslog-5424 authors. + --- End diff -- done > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518568#comment-16518568 ] ASF GitHub Bot commented on NIFI-5325: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196932374 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -327,6 +327,10 @@ 1.7.0-SNAPSHOT test + +com.github.palindromicity +simple-syslog-5424 + --- End diff -- I could only find references to this in the LICENSE files, not the NOTICE, so I made sure it was there where it was absent before ( assembly had it ) > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196932374 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -327,6 +327,10 @@ 1.7.0-SNAPSHOT test + +com.github.palindromicity +simple-syslog-5424 + --- End diff -- I could only find references to this in the LICENSE files, not the NOTICE, so I made sure it was there where it was absent before ( assembly had it ) ---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196932176 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE --- @@ -218,6 +218,14 @@ The following binary components are provided under the Apache Software License v Copyright 2006 - 2013 Pentaho Corporation. All rights reserved. Copyright 2000-2005, 2014-2016 Julian Hyde + (ASLv2) simple-syslog-5424 +The following NOTICE information applies: + +simple-syslog-5424 +https://github.com/palindromicity/simple-syslog-5424 + +Copyright 2018 simple-syslog-5424 authors. + --- End diff -- done ---
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518560#comment-16518560
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196931673
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseSyslog5424.java
---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+public class TestParseSyslog5424 {
+private static final String SYSLOG_LINE_ALL = "<14>1
2014-06-20T09:14:07+00:00 loggregator"
++ " d0602076-b14a-4c55-852a-981e7afeed38 DEA MSG-01"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+private static final String SYSLOG_LINE_NILS= "<14>1
2014-06-20T09:14:07+00:00 -"
++ " d0602076-b14a-4c55-852a-981e7afeed38 - -"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+
+@Test
+public void testValidMessage() {
+final TestRunner runner = TestRunners.newTestRunner(new
ParseSyslog5424());
+
runner.setProperty(ParseSyslog5424.NIL_POLICY,NilPolicy.DASH.name());
+runner.enqueue(SYSLOG_LINE_ALL.getBytes());
+runner.run();
+
runner.assertAllFlowFilesTransferred(ParseSyslog5424.REL_SUCCESS,1);
--- End diff --
Only version and priority must be present ( IE> cannot be the NIL '-'
character ), and therefore based on the NilPolicy by be missing etc.
I *could* check for those, but their absence would actually throw an
exception. I have a test for that specifically wrt priority
> Need a Syslog Parser that fully supports the 5424 Spec
> --
>
> Key: NIFI-5325
> URL: https://issues.apache.org/jira/browse/NIFI-5325
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> from NIFI-5139
> "The Structured Data as described in
> [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi.
>
> I suggest the SD to be put in attributes, prefixed by the ID:
> [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"]
> to become 3 attributes
> * exampleSID@32473-iut = 3
> * exampleSID@32473-eventSource=Application
> * exampleSID@32473-eventID=1011
>
> This could be useful to preprocess RFC5424 message, prioritize or route them."
>
> Nifi should a have a parser to fully support the spec and structured data.
> Since integrating it, and special casing the existing parser would
> simnifically complicate that processor and move it from it's least common
> denominator goal, it should be a new parser
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196931673
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseSyslog5424.java
---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+public class TestParseSyslog5424 {
+private static final String SYSLOG_LINE_ALL = "<14>1
2014-06-20T09:14:07+00:00 loggregator"
++ " d0602076-b14a-4c55-852a-981e7afeed38 DEA MSG-01"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+private static final String SYSLOG_LINE_NILS= "<14>1
2014-06-20T09:14:07+00:00 -"
++ " d0602076-b14a-4c55-852a-981e7afeed38 - -"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+
+@Test
+public void testValidMessage() {
+final TestRunner runner = TestRunners.newTestRunner(new
ParseSyslog5424());
+
runner.setProperty(ParseSyslog5424.NIL_POLICY,NilPolicy.DASH.name());
+runner.enqueue(SYSLOG_LINE_ALL.getBytes());
+runner.run();
+
runner.assertAllFlowFilesTransferred(ParseSyslog5424.REL_SUCCESS,1);
--- End diff --
Only version and priority must be present ( IE> cannot be the NIL '-'
character ), and therefore based on the NilPolicy by be missing etc.
I *could* check for those, but their absence would actually throw an
exception. I have a test for that specifically wrt priority
---
[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication
[ https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518558#comment-16518558 ] ASF GitHub Bot commented on NIFI-4838: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2448 I have a ticket open to transition everything in the Mongo space over to making a client service the source of configurations and connection pooling for Mongo. So the short term solution would be to copy pasta the abstract base class into one specifically for GetMongo that does this and just eliminate it later once all of the connection building is done through the client service (ie 90% of the base class will be moved over) > Make GetMongo support multiple commits and give some progress indication > > > Key: NIFI-4838 > URL: https://issues.apache.org/jira/browse/NIFI-4838 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > It shouldn't wait until the end to do a commit() call because the effect is > that GetMongo looks like it has hung to a user who is pulling a very large > data set. > It should also have an option for running a count query to get the current > approximate count of documents that would match the query and append an > attribute that indicates where a flowfile stands in the total result count. > Ex: > query.progress.point.start = 2500 > query.progress.point.end = 5000 > query.count.estimate = 17,568,231 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2448 I have a ticket open to transition everything in the Mongo space over to making a client service the source of configurations and connection pooling for Mongo. So the short term solution would be to copy pasta the abstract base class into one specifically for GetMongo that does this and just eliminate it later once all of the connection building is done through the client service (ie 90% of the base class will be moved over) ---
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518555#comment-16518555
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196929192
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196929192
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP address of the Syslog message."),
+@WritesAttribute(attribute = "syslog.appname", description = "The
appname of the Syslog message."),
+@WritesAttribute(attribute = "syslog.procid", description = "The
procid of the Syslog message."),
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518534#comment-16518534 ] ASF GitHub Bot commented on NIFI-5325: -- Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196906833 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -327,6 +327,10 @@ 1.7.0-SNAPSHOT test + +com.github.palindromicity +simple-syslog-5424 + --- End diff -- Since this transitively brings in antlr-runtime (I think) then we probably need to account for that in the NAR and assembly LICENSE/NOTICE. From a quick glance it looks like antlr is BSD 3-Clause so it probably goes in the LICENSE files with the verbiage like: "The binary distribution of this product bundles 'antlr-runtime' which is available under a "3-clause BSD" license." > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518539#comment-16518539 ] ASF GitHub Bot commented on NIFI-5325: -- Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196910966 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.syslog; + +import com.github.palindromicity.syslog.DefaultKeyProvider; +import com.github.palindromicity.syslog.KeyProvider; +import com.github.palindromicity.syslog.NilPolicy; +import com.github.palindromicity.syslog.SyslogParserBuilder; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.regex.Pattern; + +/** + * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance. + * + * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs. --- End diff -- Can probably remove the part about Flume and RFC3164 since its not part of this class right? > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518538#comment-16518538
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196912736
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume
project for RFC3164 logs.
+ * For 5424 we use simple-syslog-5424 since it parsers out structured data.
+ */
+public class StrictSyslog5424Parser {
+private Charset charset;
+private com.github.palindromicity.syslog.SyslogParser parser;
+KeyProvider keyProvider = new DefaultKeyProvider();
--- End diff --
Seems like this is unused
> Need a Syslog Parser that fully supports the 5424 Spec
> --
>
> Key: NIFI-5325
> URL: https://issues.apache.org/jira/browse/NIFI-5325
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> from NIFI-5139
> "The Structured Data as described in
> [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi.
>
> I suggest the SD to be put in attributes, prefixed by the ID:
> [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"]
> to become 3 attributes
> * exampleSID@32473-iut = 3
> * exampleSID@32473-eventSource=Application
> * exampleSID@32473-eventID=1011
>
> This could be useful to preprocess RFC5424 message, prioritize or route them."
>
> Nifi should a have a parser to fully support the spec and structured data.
> Since integrating it, and special casing the existing parser would
> simnifically complicate that processor and move it from it's least common
> denominator goal, it should be a new parser
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518540#comment-16518540
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196913302
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseSyslog5424.java
---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+public class TestParseSyslog5424 {
+private static final String SYSLOG_LINE_ALL = "<14>1
2014-06-20T09:14:07+00:00 loggregator"
++ " d0602076-b14a-4c55-852a-981e7afeed38 DEA MSG-01"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+private static final String SYSLOG_LINE_NILS= "<14>1
2014-06-20T09:14:07+00:00 -"
++ " d0602076-b14a-4c55-852a-981e7afeed38 - -"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+
+@Test
+public void testValidMessage() {
+final TestRunner runner = TestRunners.newTestRunner(new
ParseSyslog5424());
+
runner.setProperty(ParseSyslog5424.NIL_POLICY,NilPolicy.DASH.name());
+runner.enqueue(SYSLOG_LINE_ALL.getBytes());
+runner.run();
+
runner.assertAllFlowFilesTransferred(ParseSyslog5424.REL_SUCCESS,1);
--- End diff --
Not totally necessary, but may want to verify that the flow file
transferred to success has some of the attributes you would expect from the
parsed message
> Need a Syslog Parser that fully supports the 5424 Spec
> --
>
> Key: NIFI-5325
> URL: https://issues.apache.org/jira/browse/NIFI-5325
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> from NIFI-5139
> "The Structured Data as described in
> [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi.
>
> I suggest the SD to be put in attributes, prefixed by the ID:
> [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"]
> to become 3 attributes
> * exampleSID@32473-iut = 3
> * exampleSID@32473-eventSource=Application
> * exampleSID@32473-eventID=1011
>
> This could be useful to preprocess RFC5424 message, prioritize or route them."
>
> Nifi should a have a parser to fully support the spec and structured data.
> Since integrating it, and special casing the existing parser would
> simnifically complicate that processor and move it from it's least common
> denominator goal, it should be a new parser
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518536#comment-16518536 ] ASF GitHub Bot commented on NIFI-5325: -- Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196905162 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE --- @@ -218,6 +218,14 @@ The following binary components are provided under the Apache Software License v Copyright 2006 - 2013 Pentaho Corporation. All rights reserved. Copyright 2000-2005, 2014-2016 Julian Hyde + (ASLv2) simple-syslog-5424 +The following NOTICE information applies: + +simple-syslog-5424 +https://github.com/palindromicity/simple-syslog-5424 + +Copyright 2018 simple-syslog-5424 authors. + --- End diff -- Lets also add this to the NOTICE in the nifi-assembly > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518537#comment-16518537
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196908047
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP addres
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[
https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518535#comment-16518535
]
ASF GitHub Bot commented on NIFI-5325:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196907444
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP addres
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196906833 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -327,6 +327,10 @@ 1.7.0-SNAPSHOT test + +com.github.palindromicity +simple-syslog-5424 + --- End diff -- Since this transitively brings in antlr-runtime (I think) then we probably need to account for that in the NAR and assembly LICENSE/NOTICE. From a quick glance it looks like antlr is BSD 3-Clause so it probably goes in the LICENSE files with the verbiage like: "The binary distribution of this product bundles 'antlr-runtime' which is available under a "3-clause BSD" license." ---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196905162 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE --- @@ -218,6 +218,14 @@ The following binary components are provided under the Apache Software License v Copyright 2006 - 2013 Pentaho Corporation. All rights reserved. Copyright 2000-2005, 2014-2016 Julian Hyde + (ASLv2) simple-syslog-5424 +The following NOTICE information applies: + +simple-syslog-5424 +https://github.com/palindromicity/simple-syslog-5424 + +Copyright 2018 simple-syslog-5424 authors. + --- End diff -- Lets also add this to the NOTICE in the nifi-assembly ---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196907444
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP address of the Syslog message."),
+@WritesAttribute(attribute = "syslog.appname", description = "The
appname of the Syslog message."),
+@WritesAttribute(attribute = "syslog.procid", description = "The
procid of the Syslog message."),
+
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2805#discussion_r196910966 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.syslog; + +import com.github.palindromicity.syslog.DefaultKeyProvider; +import com.github.palindromicity.syslog.KeyProvider; +import com.github.palindromicity.syslog.NilPolicy; +import com.github.palindromicity.syslog.SyslogParserBuilder; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.regex.Pattern; + +/** + * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance. + * + * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs. --- End diff -- Can probably remove the part about Flume and RFC3164 since its not part of this class right? ---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196908047
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java
---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event",
"message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed
Syslog message in accordance to RFC5424 " +
+"format and adds attributes to the FlowFile for each of the parts
of the Syslog message, including Structured Data." +
+"Structured Data will be written to attributes as on attribute per
item id + parameter "+
+"see https://tools.ietf.org/html/rfc5424."; +
+"Note: ParseSyslog5424 follows the specification more closely than
ParseSyslog. If your Syslog producer " +
+"does not follow the spec closely, with regards to using '-' for
missing header entries for example, those logs " +
+"will fail with this parser, where they would not fail with
ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority",
description = "The priority of the Syslog message."),
+@WritesAttribute(attribute = "syslog.severity", description = "The
severity of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.facility", description = "The
facility of the Syslog message derived from the priority."),
+@WritesAttribute(attribute = "syslog.version", description = "The
optional version from the Syslog message."),
+@WritesAttribute(attribute = "syslog.timestamp", description = "The
timestamp of the Syslog message."),
+@WritesAttribute(attribute = "syslog.hostname", description = "The
hostname or IP address of the Syslog message."),
+@WritesAttribute(attribute = "syslog.appname", description = "The
appname of the Syslog message."),
+@WritesAttribute(attribute = "syslog.procid", description = "The
procid of the Syslog message."),
+
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196912736
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java
---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume
project for RFC3164 logs.
+ * For 5424 we use simple-syslog-5424 since it parsers out structured data.
+ */
+public class StrictSyslog5424Parser {
+private Charset charset;
+private com.github.palindromicity.syslog.SyslogParser parser;
+KeyProvider keyProvider = new DefaultKeyProvider();
--- End diff --
Seems like this is unused
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196913302
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseSyslog5424.java
---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+public class TestParseSyslog5424 {
+private static final String SYSLOG_LINE_ALL = "<14>1
2014-06-20T09:14:07+00:00 loggregator"
++ " d0602076-b14a-4c55-852a-981e7afeed38 DEA MSG-01"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+private static final String SYSLOG_LINE_NILS= "<14>1
2014-06-20T09:14:07+00:00 -"
++ " d0602076-b14a-4c55-852a-981e7afeed38 - -"
++ " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\"
eventID=\"1011\"]"
++ " [exampleSDID@32480 iut=\"4\" eventSource=\"Other
Application\" eventID=\"2022\"] Removing instance";
+
+@Test
+public void testValidMessage() {
+final TestRunner runner = TestRunners.newTestRunner(new
ParseSyslog5424());
+
runner.setProperty(ParseSyslog5424.NIL_POLICY,NilPolicy.DASH.name());
+runner.enqueue(SYSLOG_LINE_ALL.getBytes());
+runner.run();
+
runner.assertAllFlowFilesTransferred(ParseSyslog5424.REL_SUCCESS,1);
--- End diff --
Not totally necessary, but may want to verify that the flow file
transferred to success has some of the attributes you would expect from the
parsed message
---
[jira] [Created] (NIFI-5327) NetFlow Processors
Prashanth Venkatesan created NIFI-5327: -- Summary: NetFlow Processors Key: NIFI-5327 URL: https://issues.apache.org/jira/browse/NIFI-5327 Project: Apache NiFi Issue Type: New Feature Components: Core Framework Affects Versions: 1.6.0 Reporter: Prashanth Venkatesan As network traffic data scopes for the big data use case, would like NiFi to have processors to support parsing of those protocols. Netflow is a protocol introduced by Cisco that provides the ability to collect IP network traffic as it enters or exits an interface and is described in detail in here: [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] Currently, I have created the following processor: *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either NiFi flowfile attributes or as a JSON content. This also sends one-time-template. Further ahead, we can add many processor specific to network protocols in this nar bundle. I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5326) Create a controller service that returns random data from a JSON file
Mike Thomsen created NIFI-5326:
--
Summary: Create a controller service that returns random data from
a JSON file
Key: NIFI-5326
URL: https://issues.apache.org/jira/browse/NIFI-5326
Project: Apache NiFi
Issue Type: New Feature
Reporter: Mike Thomsen
Assignee: Mike Thomsen
The controller service should be able to load a JSON file and pull random
entries from the top level regardless of whether it is a JSON object or list.
Arrays of primitive values (ex. strings) should be handled with a common
structure like this:
{ "key": "XYZ" }
where XYZ is the value that was pulled from a list like this:
[ "ABC", "DEF",...,"XYZ" ]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5068) Create a script to automate some of RC verification
[ https://issues.apache.org/jira/browse/NIFI-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518496#comment-16518496 ] ASF GitHub Bot commented on NIFI-5068: -- GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2806 NIFI-5068 [WIP] Script to automate RC validation This is a work in process. Currently this script will download and validate the hashes and signing, as well as unpack the zip. Still need to: - validate the commit - run tests Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi rc-check Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2806.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2806 commit d7b41852f04d6d1d7027f95b11bf1eafb8d52282 Author: Otto Fowler Date: 2018-04-11T14:39:59Z start of utility, though there is no download to test against commit d0cd56cdba5ad3c8bf668068883a40193e51ffdc Author: Otto Fowler Date: 2018-06-11T15:39:58Z Merge branch 'master' of https://github.com/apache/nifi into rc-check commit efad5da1557fba3f88bb3a8c45d64d523b674924 Author: Otto Fowler Date: 2018-06-14T19:58:51Z some rework commit 41995d5c57405cf10fe17d242dd75a1014b92f31 Author: Otto Fowler Date: 2018-06-14T20:19:04Z works up until this point, but only with a release, need to have switch commit 5081c1536d29b956abb1245a47a44629f9f784ea Author: Otto Fowler Date: 2018-06-20T18:59:33Z working for 1.7.0 rc > Create a script to automate some of RC verification > --- > > Key: NIFI-5068 > URL: https://issues.apache.org/jira/browse/NIFI-5068 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Similar to other projects, the download, signing and checksums of the rc > materials can be verified by script and save everyone some trouble -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2806: NIFI-5068 [WIP] Script to automate RC validation
GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2806 NIFI-5068 [WIP] Script to automate RC validation This is a work in process. Currently this script will download and validate the hashes and signing, as well as unpack the zip. Still need to: - validate the commit - run tests Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi rc-check Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2806.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2806 commit d7b41852f04d6d1d7027f95b11bf1eafb8d52282 Author: Otto Fowler Date: 2018-04-11T14:39:59Z start of utility, though there is no download to test against commit d0cd56cdba5ad3c8bf668068883a40193e51ffdc Author: Otto Fowler Date: 2018-06-11T15:39:58Z Merge branch 'master' of https://github.com/apache/nifi into rc-check commit efad5da1557fba3f88bb3a8c45d64d523b674924 Author: Otto Fowler Date: 2018-06-14T19:58:51Z some rework commit 41995d5c57405cf10fe17d242dd75a1014b92f31 Author: Otto Fowler Date: 2018-06-14T20:19:04Z works up until this point, but only with a release, need to have switch commit 5081c1536d29b956abb1245a47a44629f9f784ea Author: Otto Fowler Date: 2018-06-20T18:59:33Z working for 1.7.0 rc ---
[jira] [Updated] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-5325: -- Status: Patch Available (was: In Progress) https://github.com/apache/nifi/pull/2805 > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
[ https://issues.apache.org/jira/browse/NIFI-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518455#comment-16518455 ] ASF GitHub Bot commented on NIFI-5325: -- GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2805 NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi syslog5424 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2805.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2805 commit 145582a9d3d7b6a00d14039bbb9e1f8d1a9ba5a1 Author: Otto Fowler Date: 2018-06-19T22:22:26Z NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data > Need a Syslog Parser that fully supports the 5424 Spec > -- > > Key: NIFI-5325 > URL: https://issues.apache.org/jira/browse/NIFI-5325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from NIFI-5139 > "The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them." > > Nifi should a have a parser to fully support the spec and structured data. > Since integrating it, and special casing the existing parser would > simnifically complicate that processor and move it from it's least common > denominator goal, it should be a new parser > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2805 NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi syslog5424 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2805.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2805 commit 145582a9d3d7b6a00d14039bbb9e1f8d1a9ba5a1 Author: Otto Fowler Date: 2018-06-19T22:22:26Z NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data ---
[jira] [Created] (NIFI-5325) Need a Syslog Parser that fully supports the 5424 Spec
Otto Fowler created NIFI-5325: - Summary: Need a Syslog Parser that fully supports the 5424 Spec Key: NIFI-5325 URL: https://issues.apache.org/jira/browse/NIFI-5325 Project: Apache NiFi Issue Type: New Feature Reporter: Otto Fowler Assignee: Otto Fowler from NIFI-5139 "The Structured Data as described in [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. I suggest the SD to be put in attributes, prefixed by the ID: [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] to become 3 attributes * exampleSID@32473-iut = 3 * exampleSID@32473-eventSource=Application * exampleSID@32473-eventID=1011 This could be useful to preprocess RFC5424 message, prioritize or route them." Nifi should a have a parser to fully support the spec and structured data. Since integrating it, and special casing the existing parser would simnifically complicate that processor and move it from it's least common denominator goal, it should be a new parser -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5320) Template instantiation failure when multiple component nar versions exist
[ https://issues.apache.org/jira/browse/NIFI-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518438#comment-16518438 ] Bryan Bende commented on NIFI-5320: --- The scenario is when there are more than two versions available, but none of the available versions are the one in the template, or flow.xml, or flow from registry. For example, template says processors from v 1.3.0, but the available choices are 1.5.0 and 1.6.0. In this case we can't know which one they wanted. If only 1.6.0 was available, then we just change the version from 1.3.0 to 1.6.0 (this is how upgrades happen transparently) or if 1.3.0 was one of the versions available then we have the versions that was asked for so it works. The challenges with the proposed solutions are that there could be tons of components that need their version selected so it could be quite cumbersome to step through many screens selecting versions, and for auto-selecting we don't really enforce a format for the version string so someone could make the new version "abc" and we don't really know that abc is newer than "1.6.0". The ultimate solution is when we have an extension registry and we detect a version that is not found, then we go out and download it automatically from the extension registry. > Template instantiation failure when multiple component nar versions exist > - > > Key: NIFI-5320 > URL: https://issues.apache.org/jira/browse/NIFI-5320 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0 >Reporter: Matthew Clarke >Priority: Minor > > NiFi allows users to load multiple versions of the same component nar. > For example: > nifi-scripting-nar:1.2.0 > nifi-scripting-nar:1.5.0 > - > Once I have done this, if i try to instantiate a template to the canvas that > has a processor from this nar bundle that is neither of these versions > (nifi-scripting-nar:1.3.0), it will fail to instantiate the entire template. > - > This error is displayed to user: > "Multiple versions of org.apache.nifi.processors.script.ExecuteScript exist. > No exact match for org.apache.nifi:nifi-scripting-nar:1.3.0" > - > When there are not multiple nar versions, template instantiation works fine > since NiFi will automatically select the current processor version. > - > NiFi should allow a user to instantiate a template in such a scenario by > either: > 1. Prompt user to select which of the available versions to use during > instantiation. > 2. Auto-select a new version (select most current version by default) > - -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-177) Add version info to the UI
Mark Bean created NIFIREG-177: - Summary: Add version info to the UI Key: NIFIREG-177 URL: https://issues.apache.org/jira/browse/NIFIREG-177 Project: NiFi Registry Issue Type: Improvement Affects Versions: 0.2.0 Reporter: Mark Bean Currently, the UI does not present the current version of NiFi Registry. This information should be available through an "about" menu item and included on the documentation pages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication
[ https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518385#comment-16518385 ] ASF GitHub Bot commented on NIFI-4838: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2448 From an awesome suggestion by @markap14, you could extend AbstractSessionFactoryProcessor (although this has implications since you're sharing a base class already) and use two sessions, one to get the incoming flow file, and one to create the child flow files. Then you can use session1.get() and save off the FlowFile, call session2.create(flowFile), and session2.commit() as many times as you want. Then at the end of processing you can do session1.transfer(flowFile, REL_ORIGINAL) and session1.commit(). That's a great solution IMO because it retains the "original" use case and behavior while still allowing incremental commits. We should consider this pattern when doing any source processor that allows incoming flow files and also wants to offer incremental commits. > Make GetMongo support multiple commits and give some progress indication > > > Key: NIFI-4838 > URL: https://issues.apache.org/jira/browse/NIFI-4838 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > It shouldn't wait until the end to do a commit() call because the effect is > that GetMongo looks like it has hung to a user who is pulling a very large > data set. > It should also have an option for running a count query to get the current > approximate count of documents that would match the query and append an > attribute that indicates where a flowfile stands in the total result count. > Ex: > query.progress.point.start = 2500 > query.progress.point.end = 5000 > query.count.estimate = 17,568,231 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2448 From an awesome suggestion by @markap14, you could extend AbstractSessionFactoryProcessor (although this has implications since you're sharing a base class already) and use two sessions, one to get the incoming flow file, and one to create the child flow files. Then you can use session1.get() and save off the FlowFile, call session2.create(flowFile), and session2.commit() as many times as you want. Then at the end of processing you can do session1.transfer(flowFile, REL_ORIGINAL) and session1.commit(). That's a great solution IMO because it retains the "original" use case and behavior while still allowing incremental commits. We should consider this pattern when doing any source processor that allows incoming flow files and also wants to offer incremental commits. ---
[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication
[ https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518368#comment-16518368 ] ASF GitHub Bot commented on NIFI-4838: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2448 You could call session.remove() instead of transferring it, but then I'm not sure you can still use that flow file in future operations when calling create(original). > Make GetMongo support multiple commits and give some progress indication > > > Key: NIFI-4838 > URL: https://issues.apache.org/jira/browse/NIFI-4838 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > It shouldn't wait until the end to do a commit() call because the effect is > that GetMongo looks like it has hung to a user who is pulling a very large > data set. > It should also have an option for running a count query to get the current > approximate count of documents that would match the query and append an > attribute that indicates where a flowfile stands in the total result count. > Ex: > query.progress.point.start = 2500 > query.progress.point.end = 5000 > query.count.estimate = 17,568,231 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2448 You could call session.remove() instead of transferring it, but then I'm not sure you can still use that flow file in future operations when calling create(original). ---
[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication
[ https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518355#comment-16518355 ] ASF GitHub Bot commented on NIFI-4838: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2448 With your second option, if you don't transfer the original input before you call `session.commit()` it will throw an exception because the original input will not be assigned to a relationship. So I am not sure if your second option is possible if I am understanding it correctly since it seems to rely on keeping the original flowfile around for each set of results and using `create(FlowFile)` to set a parent relationship. if incremental commits are used, would we want to dynamically remove the original relationship or keep it in case the user changes their mind so they don't have to reconfigure flows as they experiment? > Make GetMongo support multiple commits and give some progress indication > > > Key: NIFI-4838 > URL: https://issues.apache.org/jira/browse/NIFI-4838 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > It shouldn't wait until the end to do a commit() call because the effect is > that GetMongo looks like it has hung to a user who is pulling a very large > data set. > It should also have an option for running a count query to get the current > approximate count of documents that would match the query and append an > attribute that indicates where a flowfile stands in the total result count. > Ex: > query.progress.point.start = 2500 > query.progress.point.end = 5000 > query.count.estimate = 17,568,231 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2448 With your second option, if you don't transfer the original input before you call `session.commit()` it will throw an exception because the original input will not be assigned to a relationship. So I am not sure if your second option is possible if I am understanding it correctly since it seems to rely on keeping the original flowfile around for each set of results and using `create(FlowFile)` to set a parent relationship. if incremental commits are used, would we want to dynamically remove the original relationship or keep it in case the user changes their mind so they don't have to reconfigure flows as they experiment? ---
[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2448 That use case is what prompted the Jira for QueryDatabaseTable to support incremental commits. It didn't have an "original" relationship so that wasn't an issue, but the documentation explains that some attributes won't be populated if "Output Batch Size" is specified. Having a source processor (GetMongo) with an "original" relationship is where it got a little weird for me, as the pattern I'm aware of (that is getting increasingly popular) is to have source processors accept incoming flow files for the purpose of configuration, not content per se. That's why I wasn't sure where the "original" relationship belonged in the use case. Often it is used as a downstream trigger that all documents were processed, but that wouldn't apply if we were using incremental commits. I'd prefer to avoid a new processor if we can make it work in a user-friendly manner for GetMongo. My vote is for my option #1 above, with sufficient documentation to describe the behavior. If the use case dictates the need for the "original" flow file to be transferred, then they won't be able to use incremental commits. On the other hand (as @bbende suggested to me), they could always send the upstream flow to both GetMongo (with incremental commits) and to another flow. At that point it'd be similar to my option #2 where a different downstream flow would get the original flow file while it was also being worked on by GetMongo. By keeping the child flow files connected to the original as a parent (via provenance and the session), we still have the lineage intact. Thoughts? ---
[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication
[ https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518309#comment-16518309 ] ASF GitHub Bot commented on NIFI-4838: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2448 That use case is what prompted the Jira for QueryDatabaseTable to support incremental commits. It didn't have an "original" relationship so that wasn't an issue, but the documentation explains that some attributes won't be populated if "Output Batch Size" is specified. Having a source processor (GetMongo) with an "original" relationship is where it got a little weird for me, as the pattern I'm aware of (that is getting increasingly popular) is to have source processors accept incoming flow files for the purpose of configuration, not content per se. That's why I wasn't sure where the "original" relationship belonged in the use case. Often it is used as a downstream trigger that all documents were processed, but that wouldn't apply if we were using incremental commits. I'd prefer to avoid a new processor if we can make it work in a user-friendly manner for GetMongo. My vote is for my option #1 above, with sufficient documentation to describe the behavior. If the use case dictates the need for the "original" flow file to be transferred, then they won't be able to use incremental commits. On the other hand (as @bbende suggested to me), they could always send the upstream flow to both GetMongo (with incremental commits) and to another flow. At that point it'd be similar to my option #2 where a different downstream flow would get the original flow file while it was also being worked on by GetMongo. By keeping the child flow files connected to the original as a parent (via provenance and the session), we still have the lineage intact. Thoughts? > Make GetMongo support multiple commits and give some progress indication > > > Key: NIFI-4838 > URL: https://issues.apache.org/jira/browse/NIFI-4838 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > It shouldn't wait until the end to do a commit() call because the effect is > that GetMongo looks like it has hung to a user who is pulling a very large > data set. > It should also have an option for running a count query to get the current > approximate count of documents that would match the query and append an > attribute that indicates where a flowfile stands in the total result count. > Ex: > query.progress.point.start = 2500 > query.progress.point.end = 5000 > query.count.estimate = 17,568,231 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5320) Template instantiation failure when multiple component nar versions exist
[ https://issues.apache.org/jira/browse/NIFI-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518216#comment-16518216 ] Sivaprasanna Sethuraman commented on NIFI-5320: --- [~msclarke] Are you saying a template created in one NiFi version bundle, say 1.3.0 and then when imported to a different one, say NiFi version 1.7.0 fails with this error? I just created a template on NiFi 1.6.0 setup and successfully imported it to the latest master i.e. 1.7.0-SNAPSHOT. Am I missing something? > Template instantiation failure when multiple component nar versions exist > - > > Key: NIFI-5320 > URL: https://issues.apache.org/jira/browse/NIFI-5320 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0 >Reporter: Matthew Clarke >Priority: Minor > > NiFi allows users to load multiple versions of the same component nar. > For example: > nifi-scripting-nar:1.2.0 > nifi-scripting-nar:1.5.0 > - > Once I have done this, if i try to instantiate a template to the canvas that > has a processor from this nar bundle that is neither of these versions > (nifi-scripting-nar:1.3.0), it will fail to instantiate the entire template. > - > This error is displayed to user: > "Multiple versions of org.apache.nifi.processors.script.ExecuteScript exist. > No exact match for org.apache.nifi:nifi-scripting-nar:1.3.0" > - > When there are not multiple nar versions, template instantiation works fine > since NiFi will automatically select the current processor version. > - > NiFi should allow a user to instantiate a template in such a scenario by > either: > 1. Prompt user to select which of the available versions to use during > instantiation. > 2. Auto-select a new version (select most current version by default) > - -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5324) Implement syslog record readers
[ https://issues.apache.org/jira/browse/NIFI-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-5324: - Assignee: Otto Fowler > Implement syslog record readers > --- > > Key: NIFI-5324 > URL: https://issues.apache.org/jira/browse/NIFI-5324 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Otto Fowler >Priority: Major > > Creating this Jira based on discussion with [~ottobackwards] in the NiFi > HipChat room... > We currently have ListenSyslog with optional parsing when batch size is 1, > and ParseSyslog which also assumes 1 message per flow file. There is also > ListenTCPRecord and ListenUDPRecord which can be used with a GrokReader to > read log messages from the respective network connections. > The common scenario for wanting to parse the syslog messages is to extract a > field from the syslog message into an attribute and then use the attribute to > make decisions like routing/filtering. > Since the "1 message per flow file" pattern is generally something we try to > avoid, it would be nice if we could keep batches of syslog messages together > in a single flow file and then use record processors to process the batches. > For example, if we had a syslog record reader we could then use > PartitionRecord to divide a flow file of many syslog records into smaller > groups based on some field in the message, each group can then be routed > somewhere based on the group value. > Another example would be to use QueryRecord to run a SQL query that selects > specify syslog messages based on a field in the message. > It would also make it easy to convert syslog messages to a structured format > using ConvertRecord with a syslog reader and a writer like JSON or Avro. > We would likely want two syslog record readers, one for each of the RFC > formats. > One aspect to consider is related to the schema used/produced by the > reader... typically the readers/writers have a "Schema Access Strategy" where > they can obtain a schema from a schema registry, or from flow file > attributes, or something specific to the format like an embedded Avro schema. > In this case, the schema is somewhat pre-determined by the specific syslog > reader because the schema can only be at-most the fields produced by the > reader parsing the messages. So this may be a case where there is no schema > access strategy, and there are per-determined schemas. It is sort of like > the GrokReader where it creates a schema from the named fields in the > expression, except in this case there is no user defined expression, and the > named fields are dictated by the parser. > We may need to reuse syslog related code that is in nifi-standard-processors, > so it might require moving that code to nifi-processor-utils, or creating a > new nifi-syslog-utils module. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5324) Implement syslog record readers
Bryan Bende created NIFI-5324: - Summary: Implement syslog record readers Key: NIFI-5324 URL: https://issues.apache.org/jira/browse/NIFI-5324 Project: Apache NiFi Issue Type: Improvement Reporter: Bryan Bende Creating this Jira based on discussion with [~ottobackwards] in the NiFi HipChat room... We currently have ListenSyslog with optional parsing when batch size is 1, and ParseSyslog which also assumes 1 message per flow file. There is also ListenTCPRecord and ListenUDPRecord which can be used with a GrokReader to read log messages from the respective network connections. The common scenario for wanting to parse the syslog messages is to extract a field from the syslog message into an attribute and then use the attribute to make decisions like routing/filtering. Since the "1 message per flow file" pattern is generally something we try to avoid, it would be nice if we could keep batches of syslog messages together in a single flow file and then use record processors to process the batches. For example, if we had a syslog record reader we could then use PartitionRecord to divide a flow file of many syslog records into smaller groups based on some field in the message, each group can then be routed somewhere based on the group value. Another example would be to use QueryRecord to run a SQL query that selects specify syslog messages based on a field in the message. It would also make it easy to convert syslog messages to a structured format using ConvertRecord with a syslog reader and a writer like JSON or Avro. We would likely want two syslog record readers, one for each of the RFC formats. One aspect to consider is related to the schema used/produced by the reader... typically the readers/writers have a "Schema Access Strategy" where they can obtain a schema from a schema registry, or from flow file attributes, or something specific to the format like an embedded Avro schema. In this case, the schema is somewhat pre-determined by the specific syslog reader because the schema can only be at-most the fields produced by the reader parsing the messages. So this may be a case where there is no schema access strategy, and there are per-determined schemas. It is sort of like the GrokReader where it creates a schema from the named fields in the expression, except in this case there is no user defined expression, and the named fields are dictated by the parser. We may need to reuse syslog related code that is in nifi-standard-processors, so it might require moving that code to nifi-processor-utils, or creating a new nifi-syslog-utils module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-538) Warning on Fedora 28 when stopping minifi
Dustin Rodrigues created MINIFICPP-538: -- Summary: Warning on Fedora 28 when stopping minifi Key: MINIFICPP-538 URL: https://issues.apache.org/jira/browse/MINIFICPP-538 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Dustin Rodrigues "sem_wait: No such file or directory" is printed when stopping minifi on Fedora 28 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication
[ https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518065#comment-16518065 ] ASF GitHub Bot commented on NIFI-4838: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2448 @mattyb149 here's the use case that lead to this for the sake of discussion: > Client has a few huge collections. Client wants to be able to fetch very large chunks of them at a time. Client is unhappy that they have to wait for the full query execution in order to see anything happen in the UI. Client's non-production environments make it take a few hours of silent processing to finally get anything to commit to the session and show up in the UI. Client's technical folks probably would accept log statements at each iteration (where it makes sense) to show "yeah, I'm doing something" from GetMongo. So how about this third way that I could get done pretty quickly for 1.8... 1. Add RunMongoCollectionFetch as a no-input processor that works like the input sources referenced by you above. It includes full query control, progressive commits, progress attributes, etc. 2. Remove progressive commits from GetMongo, keep the option to calculate progress attributes and either way put info logger statements (that can be turned off) alerting that a new flowfile (or X num of them in the case of 1:1 result/flowfile config) was prepped. > Make GetMongo support multiple commits and give some progress indication > > > Key: NIFI-4838 > URL: https://issues.apache.org/jira/browse/NIFI-4838 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > It shouldn't wait until the end to do a commit() call because the effect is > that GetMongo looks like it has hung to a user who is pulling a very large > data set. > It should also have an option for running a count query to get the current > approximate count of documents that would match the query and append an > attribute that indicates where a flowfile stands in the total result count. > Ex: > query.progress.point.start = 2500 > query.progress.point.end = 5000 > query.count.estimate = 17,568,231 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2448 @mattyb149 here's the use case that lead to this for the sake of discussion: > Client has a few huge collections. Client wants to be able to fetch very large chunks of them at a time. Client is unhappy that they have to wait for the full query execution in order to see anything happen in the UI. Client's non-production environments make it take a few hours of silent processing to finally get anything to commit to the session and show up in the UI. Client's technical folks probably would accept log statements at each iteration (where it makes sense) to show "yeah, I'm doing something" from GetMongo. So how about this third way that I could get done pretty quickly for 1.8... 1. Add RunMongoCollectionFetch as a no-input processor that works like the input sources referenced by you above. It includes full query control, progressive commits, progress attributes, etc. 2. Remove progressive commits from GetMongo, keep the option to calculate progress attributes and either way put info logger statements (that can be turned off) alerting that a new flowfile (or X num of them in the case of 1:1 result/flowfile config) was prepped. ---
