[GitHub] metron pull request #1054: METRON-1606 Add capability to wrap json message a...
GitHub user ottobackwards reopened a pull request:
https://github.com/apache/metron/pull/1054
METRON-1606 Add capability to wrap json message as entity arrays
This PR adds the ability to configure the JSONMap parser to wrap messages
when using JSON Path queries in an entity with an array.
For example, if the json 'document' is
`{"foo": "one"}, {"bar","two"}`
you need to have it wrapped to work with it with jsonpath.
This PR allows this, resulting in
`{"configured name or default of message" : [ {{"foo": "one"},
{"bar","two"}]}`
Tests where added, along with sample data for integration, and a new sample
configuration.
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron json-wrap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1054.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1054
commit e991d8d43865149f957d530624a69539de5d7bb2
Author: Otto Fowler
Date: 2018-06-07T14:52:07Z
METRON-1606 Add capability to wrap json message as entity arrays
---
[GitHub] metron pull request #1054: METRON-1606 Add capability to wrap json message a...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/1054 ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 Great! I will give this a try asap ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 I have been on vacation, but will be reviewing Monday and Tuesday. Please do not commit ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 +1 from me. I was able to do the above, along with building metron from the instructions ansible-docker's readme.md. Thanks for sticking with it. ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 @merrimanr are you all set? ---
[GitHub] metron issue #1103: METRON-1554: Initial PCAP UI
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1103 I think we should rename from alert ui to investigate or something ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202758396
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -182,40 +185,61 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-if(getSensorParserConfig() != null) {
- cache =
CachingStellarProcessor.createCache(getSensorParserConfig().getCacheConfig());
-}
-initializeStellar();
-if(getSensorParserConfig() != null && filter == null) {
-
getSensorParserConfig().getParserConfig().putIfAbsent("stellarContext",
stellarContext);
- if
(!StringUtils.isEmpty(getSensorParserConfig().getFilterClassName())) {
-filter = Filters.get(getSensorParserConfig().getFilterClassName()
-, getSensorParserConfig().getParserConfig()
-);
+
+// Build the Stellar cache
+Map cacheConfig = new HashMap<>();
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ SensorParserConfig config = getSensorParserConfig(sensor);
+
+ if (config != null) {
+cacheConfig.putAll(config.getCacheConfig());
}
}
+cache = CachingStellarProcessor.createCache(cacheConfig);
-parser.init();
+// Need to prep all sensors
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ MessageParser parser =
entry.getValue().getMessageParser();
--- End diff --
I don't believe this is correct. We want to initialize stellar PER parser.
Each should have it's own stellar instance and cache.
---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202755740 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- So, this means that there is a kafka topic/spout per parser? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202797418 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- There is another, more likely use case where we have a transport wrapper on another message, and 1 topic split into many parsers as well. How can we handle that? Specifically -> Syslog (Many Msg types) -> kafka -> bolt -> Split per message I expect to add the ability for syslog parsing later, so set that aside. The issue is we *will* have more than one use case wrt topics. I am not going to say this PR needs to address it, but I would want us to understand our path forward and minimize the churn. It would be best if we did not have to redo this work when accounting for that. ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202798006
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -182,40 +185,61 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-if(getSensorParserConfig() != null) {
- cache =
CachingStellarProcessor.createCache(getSensorParserConfig().getCacheConfig());
-}
-initializeStellar();
-if(getSensorParserConfig() != null && filter == null) {
-
getSensorParserConfig().getParserConfig().putIfAbsent("stellarContext",
stellarContext);
- if
(!StringUtils.isEmpty(getSensorParserConfig().getFilterClassName())) {
-filter = Filters.get(getSensorParserConfig().getFilterClassName()
-, getSensorParserConfig().getParserConfig()
-);
+
+// Build the Stellar cache
+Map cacheConfig = new HashMap<>();
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ SensorParserConfig config = getSensorParserConfig(sensor);
+
+ if (config != null) {
+cacheConfig.putAll(config.getCacheConfig());
}
}
+cache = CachingStellarProcessor.createCache(cacheConfig);
-parser.init();
+// Need to prep all sensors
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ MessageParser parser =
entry.getValue().getMessageParser();
--- End diff --
We may need a test then.
---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 @justinleet the main things I saw that I would think of cutting down, or I though about looking into ( the idea may turn out to be bad ) are places where the bolt 'knows' a lot of weird or complicated initialization logic around the configurations or classes it uses, like what we do initializing Stellar, or in getComponentConfiguration. I'm ok with a follow on, I don't think this PR is creating that issue. ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202802349
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -182,40 +185,61 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-if(getSensorParserConfig() != null) {
- cache =
CachingStellarProcessor.createCache(getSensorParserConfig().getCacheConfig());
-}
-initializeStellar();
-if(getSensorParserConfig() != null && filter == null) {
-
getSensorParserConfig().getParserConfig().putIfAbsent("stellarContext",
stellarContext);
- if
(!StringUtils.isEmpty(getSensorParserConfig().getFilterClassName())) {
-filter = Filters.get(getSensorParserConfig().getFilterClassName()
-, getSensorParserConfig().getParserConfig()
-);
+
+// Build the Stellar cache
+Map cacheConfig = new HashMap<>();
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ SensorParserConfig config = getSensorParserConfig(sensor);
+
+ if (config != null) {
+cacheConfig.putAll(config.getCacheConfig());
}
}
+cache = CachingStellarProcessor.createCache(cacheConfig);
-parser.init();
+// Need to prep all sensors
+for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ MessageParser parser =
entry.getValue().getMessageParser();
--- End diff --
I'm still not sure sharing the stellar Context is a good thing. @cestella
---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202803106 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- I think I'm misunderstanding between your diagram and this implementation. There will be one kafka topic monitored by the bolt, then routed to the right parser, then output to a different spout per parser? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202808756 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- The bolt is actually executing the parser, not sending it to kafka though. Let's say that I route all my syslog stuff ( multiple message types ) through 1 kafka topic ( not tied to _ANY_ sensor. I would want to point this bolt at that one topic, and have it route to multiple parsers. I think then they all just go to enrichment? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r202812681 --- Diff: metron-platform/metron-parsers/README.md --- @@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. + +Multiple sensors can be aggregated into a single Storm topology. When this is done, there will be +multiple Kafka spouts, but only a single parser bolt which will handle delegating to the correct --- End diff -- what *is* the intermediate kafka step? That is what is confusing me. My understanding is that for each sensor you reference it will build a spout for that sensor parser topic, and then pass everything from those to this bolt, which will call the right parser and output to I'm not sure. Why have to have a sensor specific topic at all? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r203064655 --- Diff: use-cases/parser_chaining/README.md --- @@ -233,3 +233,10 @@ cat ~/data.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --b ``` You should see indices created for the `cisco-5-304` and `cisco-6-302` data with appropriate fields created for each type. + +# Aggregated Parsers with Parser Chaining +Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor. + --- End diff -- Should we explicitly say that when you use this type of topology so do NOT create a sensor topology for any sensor included in the aggregate? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r203083284 --- Diff: use-cases/parser_chaining/README.md --- @@ -233,3 +233,10 @@ cat ~/data.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --b ``` You should see indices created for the `cisco-5-304` and `cisco-6-302` data with appropriate fields created for each type. + +# Aggregated Parsers with Parser Chaining +Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor. + --- End diff -- yes. I'm not sure we need to document the UI or somewhere else? Do we have to document removing the default processors from ambari if you are going to aggregate them etc? Or how to do it in the management ui? We may need more practical steps. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 Ok @justinleet thanks for the diagram. That really helps. I did not see in the code how we were sending out to the sensor topic and then into the sensor, I though the bolt was just calling the parser ( which makes the sensor specific topic not make sense ). I think that this flow is a valid case, one where the data could be coming from either an aggregated flow into the topic *or* straight to the topic. My question looking at this, is why ( esp when people are asking about reduction of kafka overhead ) can't we have the *option* to eliminate the sensor topic completely? ---
[GitHub] metron pull request #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1099#discussion_r203095632 --- Diff: use-cases/parser_chaining/README.md --- @@ -233,3 +233,10 @@ cat ~/data.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --b ``` You should see indices created for the `cisco-5-304` and `cisco-6-302` data with appropriate fields created for each type. + +# Aggregated Parsers with Parser Chaining +Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor. + --- End diff -- I think adding the follow on is a great idea. I wonder if we shouldn't change the default install to aggregate the default sensors? I don't think that will work though, because the comma separation in ambari is the list, and you won't be able to do it. My main concern is that the reviewers and committers of this pr are going to be the only ones who can do it, and everyone in user land is going to be lost. If this is going to be expert only ( until the UI comes ) and not recommend, or a preview thing, maybe mark it as such. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 @justinleet I am fine with that as a follow on, I would like the task or issue created. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 A mechanism for the routing process to apply a transform or some such. @cestella may have a better design idea. What I would like us to do is remove the transport from the message where there are common wrappers. Example: All the message types delivered by syslog. The parsers should not have to all parser syslog AND the message. I imagine defining a transform/parser in the router that takes every message and transforms it ( parses syslog fields + structured data if 5424 into meta and MSG to the bytes ) and then passes it on to a parser that only needs to know the message. That way we can have simpler parsers, and even support the same message when transported in different wrappers. We can talk about if this is a function of parser chaining, such that each specialized parser will be second in a chain with the syslog unwrapped being first, or part of routing, but I think it is part of routing personally. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 All that being said I am a big +1 on this. Great work @justinleet, thanks for taking the time to work it through my thick skull. ---
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 Sure, actually I'll do a discuss thread when this all goes through. That way I can try again to get @cestella to comment ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 "The authentication will be handled by the hosts that allow loading of the UIs redirecting the browser to a KnoxSSO endpoint, handled in METRON-1665" How is this going to work in vagrant? What will be the new minimum required setup for security to use Metron's UI's now? ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 It seems strange to me to ONLY support SSO. Most things support a local configuration and SSO or 'other' on demand. ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 this might be worth a discuss thread @simonellistonball ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 I don't understand, how are you going to do the auth without the login screen? ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 or, maybe we are just missing each other here, and you can explain how the user will sign on. SSO doesn't mean no sign on. How will I now provide my user name and password in the app? ---
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1112 @simonellistonball, thank you. I didn't get that from the PR description. Sorry for the noise. ---
[GitHub] metron issue #865: METRON-1212 The bundle System and Maven Plugin (Feature B...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/865 ok, i give up ---
[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/865 ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 @merrimanr I'd like to get your sign off on this, now that @cestella and I have given a +1 ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 can one of you ( @cestella or @merrimanr ) merge? I can't right now ---
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 thanks again @jameslamb! ---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206625671
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/PcapServiceImpl.java
---
@@ -199,6 +208,37 @@ public InputStream getRawPcap(String username, String
jobId, Integer page) throw
return inputStream;
}
+ @Override
+ public Map getConfiguration(String username, String
jobId) throws RestException {
+Map configuration = new HashMap<>();
+try {
+ Map jobConfiguration = jobManager.getJob(username,
jobId).getConfiguration();
+ configuration.put(PcapOptions.BASE_PATH.getKey(),
PcapOptions.BASE_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.FINAL_OUTPUT_PATH.getKey(),
PcapOptions.FINAL_OUTPUT_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.START_TIME_MS.getKey(),
PcapOptions.START_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.END_TIME_MS.getKey(),
PcapOptions.END_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.NUM_REDUCERS.getKey(),
PcapOptions.NUM_REDUCERS.get(jobConfiguration, Integer.class));
+
+ boolean isFixedFilter =
PcapOptions.FILTER_IMPL.get(jobConfiguration, PcapFilterConfigurator.class)
instanceof FixedPcapFilter.Configurator;
+ if (isFixedFilter) {
+configuration.put(FixedPcapOptions.IP_SRC_ADDR.getKey(),
FixedPcapOptions.IP_SRC_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_ADDR.getKey(),
FixedPcapOptions.IP_DST_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_SRC_PORT.getKey(),
FixedPcapOptions.IP_SRC_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_PORT.getKey(),
FixedPcapOptions.IP_DST_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PROTOCOL.getKey(),
FixedPcapOptions.PROTOCOL.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PACKET_FILTER.getKey(),
FixedPcapOptions.PACKET_FILTER.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.INCLUDE_REVERSE.getKey(),
FixedPcapOptions.INCLUDE_REVERSE.get(jobConfiguration, String.class));
+ } else {
+configuration.put(QueryPcapOptions.QUERY.getKey(),
QueryPcapOptions.QUERY.get(jobConfiguration, String.class));
+ }
+} catch (JobNotFoundException e) {
+ // do nothing and return null pcapStatus
--- End diff --
We should log here.
---
[GitHub] metron pull request #1175: METRON-1453 Metron Parser for valid RFC 5424 Sysl...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/1175
METRON-1453 Metron Parser for valid RFC 5424 Syslog messages
This is a simple parser for *valid* [RFC
5424](http://www.rfc-base.org/txt/rfc-5424.txt) messages.
It produces JSON for the messages, including support for structured data.
All structured data will be flattened.
It's configuration allows for specifying how missing fields are handled (
"-" in the spec ).
They can be:
- DASH ( the default and same as spec)
- OMIT : the field will not be in the return set
- NULL : the field will be in the return set, but as null
The parser uses the
[simple-syslog-5424](https://github.com/palindromicity/simple-syslog-5424)
library.
Testing:
- Build should work ;)
- in full dev, parser configuration should be available
- if you have a syslog 5424 compliant data source, hook it up
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [-] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron syslog-5424
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1175.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1175
commit ea2ee761d45049395b585c72e44b406f32cbf881
Author: Otto Fowler
Date: 2018-08-24T20:50:30Z
METRON-1453 Metron Parser for valid RFC 5424 Syslog messages
---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213016887
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ /**
+ * The NilPolicy specifies how the parser handles missing fields in the
return
+ * It can:
+ * Omit the fields
+ * Have a value of '-' ( as spec )
+ * Have null values for the fields
+ * The default is to omit the fields from the return set.
+ */
+ private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+ @Override
+ public void configure(Map config) {
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+
+ SyslogParser parser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --
You ok with that as part of this pr?
---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213039514
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ /**
+ * The NilPolicy specifies how the parser handles missing fields in the
return
+ * It can:
+ * Omit the fields
+ * Have a value of '-' ( as spec )
+ * Have null values for the fields
+ * The default is to omit the fields from the return set.
+ */
+ private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+ @Override
+ public void configure(Map config) {
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+
+ SyslogParser parser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --
I don't think I want to touch all the parsers for this PR , there might be
more than one parser follow on depending on how many have configurations. Can
you create a jira or shall I?
---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213051917
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ /**
+ * The NilPolicy specifies how the parser handles missing fields in the
return
+ * It can:
+ * Omit the fields
+ * Have a value of '-' ( as spec )
+ * Have null values for the fields
+ * The default is to omit the fields from the return set.
+ */
+ private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+ @Override
+ public void configure(Map config) {
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+
+ SyslogParser parser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --
yeah
---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1175#discussion_r213706134
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+ public static final String NIL_POLICY_CONFIG = "nilPolicy";
+ private transient SyslogParser syslogParser;
+
+ @Override
+ public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String)
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+ }
+
+ @Override
+ public void init() {
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+try {
+ if (rawMessage == null || rawMessage.length == 0) {
+return null;
+ }
+
+ String originalString = new String(rawMessage);
+ JSONObject jsonObject = new
JSONObject(syslogParser.parseLine(originalString));
+
+ // be sure to put in the original string, and the timestamp.
+ // we wil just copy over the timestamp from the syslog
+ jsonObject.put("original_string", originalString);
+ jsonObject.put("timestamp",
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --
Great point, latest commit has the change and the test
---
[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1178 The work-around to this issue, and some documentation of it to the extent you feel necessary should go out to the users list. ---
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 Can you log an issue in upstream with your excellent description please? ---
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 @kylerichardson Let's talk over on the upstream issue ---
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 Fixed in upstream 0.0.8 I will update when it posts / tomorrow ---
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 New upstream integrated now. ---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1184 METRON-1761, allow application of grok statement multiple times This PR adds support for incoming messages to grok parsers that have multiple log lines. Instead of having to split the logs before sending to metron/kafka, you could just send the logs in batches. # todo testing ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron grok-split Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1184 commit 5cb7cb2b869694ceff37c7e07e16358e4b918b2c Author: Otto Fowler Date: 2018-09-04T10:50:38Z METRON-1761, allow application of grok statement multiple times ---
[GitHub] metron-bro-plugin-kafka issue #8: METRON-1768: Adjust versioning of metron-b...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 +1 ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1207 We have an overall deficiency here I believe. The Slot management is a oft forgotten and manual task, for adding new parsers from the ui etc, you ( I believe ) still have to remember to have already added a slot before starting your parser topology for example. We may want to consider an overall underlying approach to this that is usable where we fall short. ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1207 @mmiklavc I agree with that, but I also think that we should separate the default sensors from the regular metron install, and have them be their own optional thing. IE> you would choose to install the snort , yaf, bro things we install automatically now. ---
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1213#discussion_r221022015
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
+
+ protected transient Consumer onSuccess;
+ protected transient Consumer onError;
+
+ private HashSet sensorTypes;
+ private Map sensorToParserComponentMap;
+
+ // Stellar variables
+ private transient Context stellarContext;
+
+ public ParserRunner(HashSet sensorTypes) {
+this.sensorTypes = sensorTypes;
+ }
+
+ public Map getSensorToParserComponentMap() {
+return sensorToParserComponentMap;
+ }
+
+ public void setSensorToParserComponentMap(Map
sensorToParserComponentMap) {
+this.sensorToParserComponentMap = sensorToParserComponentMap;
+ }
+
+ public Context getStellarContext() {
+return stellarContext;
+ }
+
+ public void setOnSuccess(Consumer onSuccess) {
+this.onSuccess = onSuccess;
+ }
+
+ public void setOnError(Consumer onError) {
+this.onError = onError;
+ }
+
+ public void init(CuratorFramework client, Supplier
parserConfigSupplier) {
+if (parserConfigSupplier == null) {
+ throw new IllegalStateException("A parser config supplier must be
set before initializing the ParserRunner.");
+}
+initializeStellar(client, parserConfigSupplier);
+initializeParsers(parserConfigSupplier);
+ }
+
+ protected void initializeStellar(CuratorFramework client,
Supplier parserConfigSupplier) {
+Map cacheConfig = new HashMap<>();
+for (String sensorType: sensorTypes) {
+ SensorParserConfig config =
parserConfigSupplier.get().getSensorParserConfig(sensorType);
+
+ if (config != null) {
+cacheConfig.putAll(config.getCacheConfig());
+ }
+}
+Cache cache =
CachingStellarProcessor.createCache(cacheConfig);
+
+Context.Builder builder = new Context.Builder()
+.with(Context.Capabilities.ZOOKEEPER_CLIENT, () -> client)
+.with(Context.Capabilities.GLOBAL_CONFIG, () ->
parserConfigSupplier.get().getGlobalConfig())
+.with(Context.Capabilities.STELL
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 Hopefully it is all set now ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 I am sorry, I missed the comments on this. I will try to have something soon. ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc wrt failing the whole message or some sort of partial failure scheme. I don't like failing multiples if we _can_ parse some lines, but I don't see a good way to cleanly handle it. Or we need to talk it through. We could track exceptions and lines processed and if we got an exception for every line fail the file, else log the execeptions and a message about only parsing x of y lines and pass back a partial message list. Thoughts? ---
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 @JonZeolla let me know if you are all set ---
[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1175#discussion_r221426956 --- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec --- @@ -590,6 +591,8 @@ chkconfig --del metron-alerts-ui %changelog * Thu Aug 30 2018 Apache Metron - 0.6.1 - Update compiled css file name for Alerts UI +* Fri Aug 24 2018 Apache Metron - 0.5.1 --- End diff -- fixed ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 If you think the "every line fails" == fail, some fails = emit and log works, we can do that, but I don't know how or if we want to put things in the error stream. I need to look at it more closely. I think that solving this might be beyond this pr, with a documentation note as you state. ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc Can you take a look at the parser.validate() stuff in the bolt? Maybe the answer is put a dummy invalid record in there and fail validation for each parse failure? ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc I looked through the validation stuff more, I think that validation is the way to go here. The grok parser will add invalid message for each exception, parser failure, and then in the validation call fail those messages. It will have to be done so that the returned message makes sense when it is sent to the error topic. What do you think? @cestella ? ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc but we don't have messages to split, we have bytes. If we where going to leave the 'parser's as single object -> single result | single exceception', ie not change the interface and not subvert validate, then we would have to introduce 'splitStrategies' at the bolt. ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 So the idea would be that the JSONObject returned for the failed line ( that would be passed to handle error ) would be a new object that had the raw line, the exception type, the exception message in it. That would all get passed through to the Error stuff. We could put anything we wanted in there. ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 It think the API should be improved ideally, but that is in the future ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 Let me give it a shot, I'll document the semantics of the failure mode and we can look again ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 If it turns out that this is just such a duck tape job, we can always close the PR and open a jira for the new api ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 actually, I'm just going to close this. Once I step back from "how could I do this" to look at the big picture, it doesn't seem like a good idea. This is not a field issue, more of an improvement, so there is no need to hammer something in. ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 Closing this pr. I will create a jira for api improvement ---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/1184 ---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
GitHub user ottobackwards reopened a pull request: https://github.com/apache/metron/pull/1184 METRON-1761, allow application of grok statement multiple times This PR adds support for incoming messages to grok parsers that have multiple log lines. Instead of having to split the logs before sending to metron/kafka, you could just send the logs in batches. # todo testing ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron grok-split Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1184 commit 498313a8c12bcc15c8a179f5a97afff1d673d0b2 Author: Otto Fowler Date: 2018-09-04T10:50:38Z METRON-1761, allow application of grok statement multiple times commit 05b734cf18be120cbabea554a0d84bb0164193c9 Author: Otto Fowler Date: 2018-09-28T14:17:41Z Merge remote-tracking branch 'apache/master' into grok-split commit c2b3bb88d2a06e5cde39fd90a87f92207906eac4 Author: Otto Fowler Date: 2018-09-28T15:40:37Z per review, do not require derivation commit 91764c924a7a85e4f8146105857234ad7b046c72 Author: Otto Fowler Date: 2018-10-02T13:12:56Z Merge remote-tracking branch 'apache/master' into grok-split ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 re-opening for input ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @merrimanr @mmiklavc First pass of what we discussed ---
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1188 It may be time for a README for these scripts ---
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1188 +1 from me, great improvements. ---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r223833982
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
---
@@ -134,26 +144,102 @@ public void init() {
@SuppressWarnings("unchecked")
@Override
public List parse(byte[] rawMessage) {
+Optional> resultOptional =
parseOptionalResult(rawMessage);
+if (!resultOptional.isPresent()) {
+ return Collections.EMPTY_LIST;
+}
+Map errors =
resultOptional.get().getMessageThrowables();
+if (!errors.isEmpty()) {
+ throw new
RuntimeException(errors.entrySet().iterator().next().getValue());
+}
+
+return resultOptional.get().getMessages();
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public Optional>
parseOptionalResult(byte[] rawMessage) {
if (grok == null) {
init();
}
+if (multiLine) {
+ return parseMultiLine(rawMessage);
+}
+return parseSingleLine(rawMessage);
+ }
+
+ @SuppressWarnings("unchecked")
+ private Optional> parseMultiLine(byte[]
rawMessage) {
List messages = new ArrayList<>();
+Map errors = new HashMap<>();
String originalMessage = null;
// read the incoming raw data as if it may have multiple lines of logs
// if there is only only one line, it will just get processed.
try (BufferedReader reader = new BufferedReader(new StringReader(new
String(rawMessage, StandardCharsets.UTF_8 {
while ((originalMessage = reader.readLine()) != null) {
LOG.debug("Grok parser parsing message: {}", originalMessage);
-Match gm = grok.match(originalMessage);
-gm.captures();
-JSONObject message = new JSONObject();
-message.putAll(gm.toMap());
+try {
+ Match gm = grok.match(originalMessage);
+ gm.captures();
+ JSONObject message = new JSONObject();
+ message.putAll(gm.toMap());
-if (message.size() == 0)
- throw new RuntimeException("Grok statement produced a null
message. Original message was: "
- + originalMessage + " and the parsed message was: " +
message + " . Check the pattern at: "
- + grokPath);
+ if (message.size() == 0) {
+Throwable rte = new RuntimeException("Grok statement produced
a null message. Original message was: "
++ originalMessage + " and the parsed message was: " +
message + " . Check the pattern at: "
++ grokPath);
+errors.put(originalMessage, rte);
+continue;
+ }
+ message.put("original_string", originalMessage);
+ for (String timeField : timeFields) {
+String fieldValue = (String) message.get(timeField);
+if (fieldValue != null) {
+ message.put(timeField, toEpoch(fieldValue));
+}
+ }
+ if (timestampField != null) {
+message.put(Constants.Fields.TIMESTAMP.getName(),
formatTimestamp(message.get(timestampField)));
+ }
+ message.remove(patternLabel);
+ postParse(message);
+ messages.add(message);
+ LOG.debug("Grok parser parsed message: {}", message);
+} catch (Exception e) {
+ LOG.error(e.getMessage(), e);
+ errors.put(originalMessage, e);
+}
+ }
+} catch (IOException e) {
--- End diff --
That is right
---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r223836515
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -383,7 +408,7 @@ public void execute(Tuple tuple) {
}
}
- protected void handleError(byte[] originalMessage, Tuple tuple,
Throwable ex, OutputCollector collector) {
+ protected void handleError(Object originalMessage, Tuple tuple,
Throwable ex, OutputCollector collector) {
--- End diff --
Sorry, that was from an earlier rev. where I was using that.
---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r223836783
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineWithErrorsGrokParserTest.java
---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.metron.parsers.interfaces.MessageParserResult;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class MultiLineWithErrorsGrokParserTest {
+
+ /**
+ * Test that if a byte[] with multiple lines of log is passed in
+ * it will be parsed into the correct number of messages.
+ * @throws IOException if we can't read from disk
+ * @throws ParseException if we can't parse
+ */
+ @Test(expected = RuntimeException.class)
+ @SuppressWarnings("unchecked")
+ public void test() throws IOException, ParseException {
--- End diff --
done
---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r223837103
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
---
@@ -31,23 +35,41 @@
/**
* Take raw data and convert it to a list of messages.
*
- * @param rawMessage
+ * @param rawMessage the raw bytes of the message
* @return If null is returned, this is treated as an empty list.
*/
List parse(byte[] rawMessage);
/**
* Take raw data and convert it to an optional list of messages.
- * @param parseMessage
+ * @param parseMessage the raw bytes of the message
* @return If null is returned, this is treated as an empty list.
*/
default Optional> parseOptional(byte[] parseMessage) {
return Optional.ofNullable(parse(parseMessage));
}
+ /**
+ * Take raw data and convert it to messages. Each raw message may
produce multiple messages and therefore
+ * multiple errors. A {@link MessageParserResult} is returned, which
will have both the messages produced
+ * and the errors.
+ * @param parseMessage the raw bytes of the message
+ * @return Optional of {@link MessageParserResult}
+ */
+ default Optional> parseOptionalResult(byte[]
parseMessage) {
--- End diff --
Right, the Optional interface is the newer, so I 'built' on that approach.
---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r223836809
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.metron.parsers.interfaces.MessageParserResult;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class MultiLineGrokParserTest {
+
+ /**
+ * Test that if a byte[] with multiple lines of log is passed in
+ * it will be parsed into the correct number of messages.
+ * @throws IOException if we can't read from disk
+ * @throws ParseException if we can't parse
+ */
+ @Test
+ @SuppressWarnings("unchecked")
+ public void test() throws IOException, ParseException {
--- End diff --
done
---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc please see latest commit ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @merrimanr any comment? ---
[GitHub] metron pull request #1234: METRON-1820 Syslog support for new api -> multipl...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1234 METRON-1820 Syslog support for new api -> multiple messages and errors This PR adds support for the Syslog parser and having multiple messages and errors. Run a syslog 5424 source, with some invalid messages mixed in, the valid messages should still get indexed. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron syslog-error-handling Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1234.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1234 commit 373b74e092021e93b4b98c3e57f77a103f620ce6 Author: Otto Fowler Date: 2018-10-11T13:13:21Z support for new api -> multiple messages and errors ---
[GitHub] metron issue #1234: METRON-1820 Syslog support for new api -> multiple messa...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1234 I had missed support for the master throwable in the old List parse function. This function is not called by the Bolt anymore, but I corrected the issue in this and Grok. ---
[GitHub] metron pull request #1234: METRON-1820 Syslog support for new api -> multipl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1234#discussion_r224528191
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -61,16 +67,37 @@ public void configure(Map config) {
public void init() {
}
- @Override
@SuppressWarnings("unchecked")
+ @Override
public List parse(byte[] rawMessage) {
+Optional> resultOptional =
parseOptionalResult(rawMessage);
--- End diff --
It wouldn't, but the interface must be implemented. It is awkward, I'm not
sure how we want to proceed with that overall. When we added the first
parseOptional and moved the bolt to it we should have deprecated the old method
or something
---
[GitHub] metron pull request #1234: METRON-1820 Syslog support for new api -> multipl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1234#discussion_r224528495
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -61,16 +67,37 @@ public void configure(Map config) {
public void init() {
}
- @Override
@SuppressWarnings("unchecked")
+ @Override
public List parse(byte[] rawMessage) {
+Optional> resultOptional =
parseOptionalResult(rawMessage);
--- End diff --
I'm open to a different approach, I could throw a notimplementedexception
or something.
---
[GitHub] metron pull request #1234: METRON-1820 Syslog support for new api -> multipl...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1234#discussion_r224573925
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
---
@@ -61,16 +67,37 @@ public void configure(Map config) {
public void init() {
}
- @Override
@SuppressWarnings("unchecked")
+ @Override
public List parse(byte[] rawMessage) {
+Optional> resultOptional =
parseOptionalResult(rawMessage);
--- End diff --
I think that is a fine idea, I had thought of a new interface, but I
decided to follow the precedent of parseOptional.
---
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1213#discussion_r224746795
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java
---
@@ -137,11 +208,29 @@ private void
initializeParsers(Supplier parserConfigSuppli
}
}
+ /**
+ * Post-processes parsed messages by:
+ *
+ * Applying field transformations defined in the sensor parser
config
+ * Filtering messages using the configured MessageFilter class
+ * Validating messages using the MessageParser validate method
+ *
+ * If a message is successfully processed a message is returned in a
ProcessResult. If a message fails
+ * validation, a MetronError object is created and returned in a
ProcessResult. If a message is
+ * filtered out an empty Optional is returned.
+ *
+ * @param sensorType Sensor type of the message
+ * @param message Message parsed by the MessageParser
+ * @param rawMessage Raw message including metadata
+ * @param parser MessageParser for the sensor type
+ * @param parserConfigurations Parser configurations
+ */
@SuppressWarnings("unchecked")
- protected Optional processMessage(String sensorType,
JSONObject message, RawMessage rawMessage,
+ protected Optional processMessage(String sensorType,
JSONObject message, RawMessage rawMessage,
MessageParser parser,
--- End diff --
Is parser used?
---
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1213#discussion_r224755060
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java
---
@@ -137,11 +208,29 @@ private void
initializeParsers(Supplier parserConfigSuppli
}
}
+ /**
+ * Post-processes parsed messages by:
+ *
+ * Applying field transformations defined in the sensor parser
config
+ * Filtering messages using the configured MessageFilter class
+ * Validating messages using the MessageParser validate method
+ *
+ * If a message is successfully processed a message is returned in a
ProcessResult. If a message fails
+ * validation, a MetronError object is created and returned in a
ProcessResult. If a message is
+ * filtered out an empty Optional is returned.
+ *
+ * @param sensorType Sensor type of the message
+ * @param message Message parsed by the MessageParser
+ * @param rawMessage Raw message including metadata
+ * @param parser MessageParser for the sensor type
+ * @param parserConfigurations Parser configurations
+ */
@SuppressWarnings("unchecked")
- protected Optional processMessage(String sensorType,
JSONObject message, RawMessage rawMessage,
+ protected Optional processMessage(String sensorType,
JSONObject message, RawMessage rawMessage,
MessageParser parser,
--- End diff --
ignore the thing about Multiline... I'm mixing up PR's
---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r224996952 --- Diff: README.md --- @@ -11,6 +11,32 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + --- End diff -- What is the case for having more than one way to install? When would you not want to use bro-pkg? ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r224996990 --- Diff: README.md --- @@ -42,22 +68,47 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to demonstrate the example. -### Example 1 +### Example 1 - Send a list of logs to kafka The goal in this example is to send all HTTP and DNS records to a Kafka topic named `bro`. * Any configuration value accepted by librdkafka can be added to the `kafka_conf` configuration table. - * By defining `topic_name` all records will be sent to the same Kafka topic. - * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. + * The `topic_name` will default to send all records to a single Kafka topic called 'bro'. + * Defining `logs_to_send` will send the HTTP and DNS records to the brokers specified in your `Kafka::kafka_conf`. ``` @load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); --- End diff -- The example should be for multiple brokers, not just one since that will be the most common case won't it? ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r224997103 --- Diff: README.md --- @@ -144,23 +194,35 @@ event bro_init() &priority=-5 Notes * `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`. + * In Bro 2.5.x the bro project introduced a [logger function](https://www.bro.org/sphinx/cluster/index.html#logger) which removes the logging functions from the manager thread, and taking advantage of that is highly recommended. If you are running this plugin on Bro 2.4.x, you may encounter issues where the manager thread is taking on too much responsibility and pinning a single CPU core without the ability to spread the load across additional cores. In this case, it may be in your best interest to prefer using a bro logging predicate over filtering in your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common) in order to lesson the load of that thread. --- End diff -- s/lesson/lessen/ ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r224996917 --- Diff: README.md --- @@ -11,6 +11,32 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + +`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation. + --- End diff -- Do these libs have to be installed in any certain place? Relative to the Bro Install? ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r225028659 --- Diff: README.md --- @@ -11,6 +11,32 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + +`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation. + --- End diff -- I mean do we have to document that the libs have to be installed to where bro looks for libs? Like /usr/local vs /opt ---
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1213 @justinleet if you take master, I'll help resolve the conflicts from #1234 ---
[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/684 I don't think anything has been done with this. The feature branch wasn't made ---
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1213 oops, right ---
[GitHub] metron issue #870: METRON-1364: Add an implementation of Robust PCA outlier ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/870 @cestella ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r225631931 --- Diff: README.md --- @@ -11,6 +11,32 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + +`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation. + --- End diff -- ok ---
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1213 That is fine ---
[GitHub] metron issue #1233: METRON-1816: Date format Stellar function
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1233 we should support the ISO date format constants so that users don't have to copy those formats. ---
[GitHub] metron pull request #1233: METRON-1816: Date format Stellar function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1233#discussion_r226064136
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/DateFunctions.java
---
@@ -109,6 +110,13 @@ public static long getEpochTime(String date, String
format, Optional tim
return sdf.parse(date).getTime();
}
+ public static String getDateFormat(String format, Optional
epochTime, Optional timezone) {
+Long time = epochTime.orElseGet(System::currentTimeMillis);
+TimezonedFormat fmt = timezone.map(s -> new TimezonedFormat(format,
s)).orElseGet(() -> new TimezonedFormat(format));
+SimpleDateFormat sdf = formatCache.get(fmt).get();
--- End diff --
are there any errors we need to wrap here?
---
[GitHub] metron pull request #1233: METRON-1816: Date format Stellar function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1233#discussion_r226064248
--- Diff:
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/DateFunctionsTest.java
---
@@ -225,4 +226,36 @@ public void testDayOfYearNow() {
public void testDayOfYearNull() {
Object result = run("DAY_OF_YEAR(nada)");
}
+
+ @Test
+ public void testDateFormat() {
+Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss zzz',
epoch, 'EST')");
+assertEquals("Thu Aug 25 2016 08:27:10 EST", result);
+ }
+
+ @Test
+ public void testDateFormatDefault() {
+Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ')");
+
assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(true,
1)));
+ }
+
+ @Test
+ public void testDateFormatNow() {
+Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss zzz',
'GMT')");
+assertTrue(result.toString().endsWith("GMT"));
+ }
+
+ @Test
+ public void testDateFormatDefaultTimezone() {
+Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ',
epoch)");
+
assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(true,
1)));
+ }
+
+ /**
+ * If refer to variable that does not exist, expect ParseException.
+ */
+ @Test(expected = ParseException.class)
+ public void testDateFormatNull() {
+Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss zzz', nada,
'EST')");
+ }
--- End diff --
Can we have a test for an invalid format
---
