[GitHub] incubator-metron pull request: METRON-115 Fetch Threat Intel Data ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/85#issuecomment-215133778 +1 this looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-119 Move PCAP infrastructure...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/93 METRON-119 Move PCAP infrastructure from HBase As it stands, the existing approach to handling PCAP data has some issues handling high volume packet capture data. With the advent of a DPDK plugin for capturing packet data, we are going to hit some limitations on the throughput of consumption if we continue to try to push packet data into HBase at line-speed. Furthermore, storing PCAP data into HBase limits the range of filter queries that we can perform (i.e. only those expressible within the key). As of now, we require all fields to be present (source IP/port, destination IP/port and protocol), rather than allowing any wildcards. To address these issues, we should create a higher performance topology which attaches the appropriate header to the raw packet and timestamp read from Kafka (as placed onto kafka by the packet capture sensor) and appends this packet to a sequence file in HDFS. The sequence file will be rolled based on number of packets or time (e.g. 1 hrs worth of packets in a given sequence file). On the query side, we should adjust the middle tier service layer to start a MR job on the appropriate set of sequence files to filter out the appropriate packets. NOTE: the UI modifications to make this reasonable for the end-user will need to be done in a follow-on JIRA. In order to test this PR, I would suggest doing the following as the "happy path": 1. Install the pycapa library & utility via instructions [here](https://github.com/apache/incubator-metron/tree/master/metron-sensors/pycapa) 2. (if using singlenode vagrant) Kill the enrichment and sensor topologies via `for i in bro enrichment yaf snort;do storm kill $i;done` 3. Start the pcap topology via `/usr/metron/0.1BETA/bin/start_pcap_topology.sh` 4. Start the pycapa packet capture producer on eth1 via `/usr/bin/pycapa --producer --topic pcap -i eth1 -k node1:6667` 5. Watch the topology in the [Storm UI](http://node1:8744/index.html) and kill the packet capture utility from before when the number of packets ingested is over 1k. 6. Ensure that at at least 2 files exist on HDFS by running `hadoop fs -ls /apps/metron/pcap` 7. Choose a file (denoted by $FILE) and dump a few of the contents using the `pcap_inspector` utility via `/usr/metron/0.1BETA/bin/pcap_inspector.sh -i $FILE -n 5` 8. Choose one of the lines and note the source ip/port and dest ip/port 9. Go to the kibana app at [http://node1:5000](http://node1:5000) on the singlenode vagrant (ymmv on ec2) and input that query in the kibana PCAP panel. 10. Wait patiently while the MR job completes and the results are sent back in the form of a valid PCAP payload suitable for opening in wireshark 11. Open in wireshark to ensure the payload is valid. If the payload is not valid PCAP, then please look at the [job history](http://node1:19888/jobhistory) and note the reason for job failure if any. Also, please note changes and addition to the documentation for the [pcap service](https://github.com/cestella/incubator-metron/tree/METRON-119/metron-streaming/metron-api) and [pcap backend](https://github.com/cestella/incubator-metron/tree/METRON-119/metron-platform/metron-pcap-backend). You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-119 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/93.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #93 commit e5062606519bda57eb7c1a739317e4f2011cddd1 Author: cstella Date: 2016-04-28T17:51:57Z METRON-119 Move the PCAP topology from HBase commit 99bf1632a7e5ed3d36137ec326626c0b0f84d4bf Author: cstella Date: 2016-04-28T17:56:05Z Updating the documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-124: Mitigate issues found i...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/92#issuecomment-215559766 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-123 Pycapa requires 'kafka_b...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/94#issuecomment-215568818 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-119 Move PCAP infrastructure...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/93#issuecomment-215694256 @dlyle65535 Definitely agreed, I'll submit a change this morning to remove the warnings. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-119 Move PCAP infrastructure...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/93#discussion_r61566951
--- Diff:
metron-deployment/roles/metron_streaming/tasks/metron_topology.yml ---
@@ -16,18 +16,14 @@
#
---
- name: Submit Metron Parser topologies
- command: storm jar {{ metron_directory }}/lib/{{ metron_parsers_jar_name
}} org.apache.storm.flux.Flux --filter {{
metron_parsers_properties_config_path }} --remote {{ item }}
+ command: "{{ metron_directory }}/bin/start_parser_topology.sh {{ item }}"
with_items:
- "{{ storm_parser_topologies }}"
--- End diff --
So, adding pcap to the list of parser topologies won't do it because pcap
has a special script (start_pcap_topology.sh) due to it having a different
config file (all of the parser topologies share the same config). Also, it's
just a different sort of beast than a parser topology (i.e. we don't actually
parse anything, we just take the raw data, slap on a header and put it in HDFS).
That being said, what I think we need to do is start the pcap topology when
pycapa is installed. I'll have to look into where and how to do that in
ansible. If you have any thoughts or suggestions, I'd be all ears. ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-119 Move PCAP infrastructure...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/93#discussion_r61567528
--- Diff:
metron-deployment/roles/metron_streaming/tasks/metron_topology.yml ---
@@ -16,18 +16,14 @@
#
---
- name: Submit Metron Parser topologies
- command: storm jar {{ metron_directory }}/lib/{{ metron_parsers_jar_name
}} org.apache.storm.flux.Flux --filter {{
metron_parsers_properties_config_path }} --remote {{ item }}
+ command: "{{ metron_directory }}/bin/start_parser_topology.sh {{ item }}"
with_items:
- "{{ storm_parser_topologies }}"
--- End diff --
In retrospect, why don't we push this to a follow-on JIRA?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 122
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/96#issuecomment-215979113 @merrimanr Hmm, I liked that feature (being able to run individual roles against either a vagrant image or a prebuilt image). That will be useful, I think for letting people get metron and try stuff out on it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 122
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/96#issuecomment-215985677 I'm not sure that `ansible-playbook -i metron-deployment/inventory/singlenode-vagrant/hosts metron-deployment/playbooks/metron_install.yml --tags enrichment -u vagrant -k` is more intuitive than `run_enrichment.sh` for the relatively common task of redeploying. There is enough differing opinion here, though, that I think perhaps the prudent thing to do is punt any vagrant changes to another PR. Can we just keep the `run_enrichment.sh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-127: HDFSDataPrunerTest fail...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/97#issuecomment-216260955 +1 this looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-128 - Create Platform Launch...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/98#issuecomment-216294305 +1 this is great! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 122
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/96#issuecomment-216301128 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-135: Ambari Start needs to s...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/106#issuecomment-217462306 Big +1, applied this change last night and it worked like a charm on SNV --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Update README.md formatting
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/95#issuecomment-217462512 +1, this is great. Thanks for contributing! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/108 METRON-141: The ability to do threat triage We have the ability to mark messages as part of the enrichment topology as threat alerts, but we have no ability to prioritize those alerts. We should allow for the prioritization of messages that have some threat intelligence alert via a scoring mechanism. The general idea is for each message from each sensor: 1. Map threat conditions as defined by a lightweight query language to threat levels (a number). 2. More than one condition may match a given message, so allow for the choice of aggregation functions. 3. If the aggregated score is positive, then add a field (`threat.triage.level`) with the level to the JSON message. The above configuration is done at the per-sensor level. As part of this PR, the following were added out of necessity: * A query language DSL which is used in defining the conditions which map to threat levels. Docs at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#query-language) * The addition of a `triageConfig` section to the sensor enrichment config. Description at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#the-threatintel-configuration) * Actual documentation of our enrichment configuration. Documentation at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#enrichment-configuration) * A rewrite of the `zk_load_configs.sh` utility to support both pushing and pulling configs (the threat triage rules are part of the configs and until we have a UI, we need to add them). Docs at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#management-utility) You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-141 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #108 commit cfdffd48d6a5ea7c228c4b66f23c16cc55691f79 Author: cstella Date: 2016-05-06T22:46:02Z METRON-141: The ability to do threat triage --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-140 CSVExtractor gives index...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/111#issuecomment-217664922 Looks like we got 2 for the same JIRA, this PR and PR 109. Cool! It's great when people contribute. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-140 handle empty line in CSV...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/109#discussion_r62421908
--- Diff:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/extractor/csv/CSVExtractor.java
---
@@ -82,6 +83,14 @@ public LookupConverter getConverter() {
return Arrays.asList(new LookupKV(key, converter.toValue(values)));
}
+ private boolean ignore(String line) {
+if(null == line) {
+ return false;
--- End diff --
I think you mean to return "true" here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-140 handle empty line in CSV...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/109#discussion_r62421915
--- Diff:
metron-platform/metron-data-management/src/test/java/org/apache/metron/dataloads/extractor/csv/CSVExtractorTest.java
---
@@ -91,5 +85,9 @@ public void validate(ExtractorHandler handler) throws
IOException {
Iterable results =
handler.getExtractor().extract("#google.com,1.0,foo");
Assert.assertEquals(0, Iterables.size(results));
}
+{
+ Iterable results = handler.getExtractor().extract("");
+ Assert.assertEquals(0, Iterables.size(results));
+}
--- End diff --
Cool! Can you add a case for null and for a " "?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-140 handle empty line in CSV...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/109#issuecomment-217665302 This looks great outside of the couple of things I mentioned. Definitely like the null check. +1 imminent if you get those changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-137: Improve output on ignor...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/110#issuecomment-217665485 Cool, got my vote +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-138 Update EC2 README
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/107#issuecomment-217665540 "Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us." I LOL'd +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/108#discussion_r62583229
--- Diff:
metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java
---
@@ -74,7 +74,7 @@ public void test() throws Exception {
}
- @Test
+ /*@Test
--- End diff --
Whoops, absolutely right! Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/108#discussion_r62583706
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
---
@@ -84,7 +88,7 @@ public JSONObject joinMessages(Map
streamMessageMap) {
if(sourceType != null) {
SensorEnrichmentConfig config =
configurations.getSensorEnrichmentConfig(sourceType);
if (config != null) {
-return config.getEnrichmentFieldMap();
+return config.getEnrichment().getFieldMap();
--- End diff --
I think that the enrichment variable is initialized, but a null check is
never a bad idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-140 handle empty line in CSV...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/109#issuecomment-218142099 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 136 logrotate pcapservice
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/113#issuecomment-218212537 This looks good to me, but I'd very much like @dlyle65535 to chime in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/115#discussion_r62781383
--- Diff:
metron-platform/metron-integration-test/src/main/resources/sample/config/sensors/websphere.json
---
@@ -0,0 +1,22 @@
+{
+ "index": "websphere",
+ "batchSize": 1,
+ "enrichmentFieldMap":
--- End diff --
So, this structure got refactored a bit as of the last PR. Check out the
example
[here](https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#example)
The upshot is that instead of `enrichmentFieldMap` and `threatIntelFieldMap`
there are `enrichment` and `threatIntel` sections and `fieldMap` is in both.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/115#discussion_r62843136
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/websphere/GrokWebSphereParserTest.java
---
@@ -0,0 +1,115 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.websphere;
+
+import static org.junit.Assert.assertEquals;
+import java.util.List;
+import org.json.simple.JSONObject;
+import org.junit.Test;
+
+public class GrokWebSphereParserTest {
+
+ @Test
+ public void testParseLoginLine() throws Exception {
+
+ //Set up parser, parse message
+ GrokWebSphereParser parser = new GrokWebSphereParser();
--- End diff --
We probably want a negative case for each of these scenarios (login,
logout, rbm and other). What happens when you try to parse malformed lines or
an empty line..don't want it to crash the topology.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/115#issuecomment-218452417 Thanks so much for the contribution. This is going to be great when it gets in! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/115#issuecomment-218772767 Great Job! Have you tested this on the full-dev-vagrant at all? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 146 topology workers
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/116#issuecomment-218777995 Ah, ok, so this is failing the integration tests because you're not setting the topology.workers property. You can either set it * In [this](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/EnrichmentIntegrationTest.java#L151) block of code * In [this](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties) and [this](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-solr/src/main/config/solr.properties) file The first will just fix it for the integration test. The second will provide a default for the deployed properties. That being said, it should be being set via ansible, so providing a default isn't strictly required. I don't have a strong opinion either way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-151: Remove yum update from ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/117#issuecomment-218843462 Definitely +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63087348
--- Diff:
metron-platform/metron-api/src/main/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasy.java
---
@@ -97,6 +97,66 @@ private static boolean isValidPort(String port) {
}
return false;
}
+ /*
+ * (non-Javadoc)
+ *
+ * @see
+ *
com.cisco.opensoc.hbase.client.IPcapReceiver#getPcapsByIdentifiers(java.lang
--- End diff --
Could you change the javadocs here to reflect the method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63099841
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java
---
@@ -18,12 +18,20 @@
package org.apache.metron.common.query;
-import org.antlr.v4.runtime.*;
-import org.apache.metron.common.query.generated.*;
+import org.antlr.v4.runtime.ANTLRInputStream;
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.TokenStream;
+import org.apache.metron.common.query.generated.PredicateLexer;
+import org.apache.metron.common.query.generated.PredicateParser;
+
+import static org.apache.commons.lang3.StringUtils.isEmpty;
public class PredicateProcessor {
public boolean parse(String rule, VariableResolver resolver) {
+if (isEmpty(rule)) {
--- End diff --
what about rule != null && isEmpty(rule.trim())?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63099913
--- Diff:
metron-platform/metron-common/src/test/java/org/apache/metron/common/query/QueryParserTest.java
---
@@ -67,6 +67,7 @@ public void testSimpleOps() throws Exception {
Assert.assertTrue(run("foo== foo", v -> variableMap.get(v)));
Assert.assertTrue(run("empty== ''", v -> variableMap.get(v)));
Assert.assertTrue(run("spaced == 'metron is great'", v ->
variableMap.get(v)));
+Assert.assertTrue(run("", v -> variableMap.get(v)));
--- End diff --
How about a test case for null too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63100294
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilters.java
---
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.filter;
+
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+
+import java.util.function.Function;
+
+public enum PcapFilters {
+ FIXED(x -> new FixedPcapFilter()),
+ QUERY(x -> new QueryPcapFilter());
+
+ Function filter;
+
+ PcapFilters(Function filter) {
--- End diff --
It'd be cool if this was an actual interface like, Creator { T create();
} and it lived in metron-common, as opposed to Function. I bet we
have use for this pattern elsewhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63101688
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java
---
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.filter.query;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.query.PredicateProcessor;
+import org.apache.metron.common.query.VariableResolver;
+import org.apache.metron.pcap.PacketInfo;
+import org.apache.metron.pcap.PcapHelper;
+import org.apache.metron.pcap.filter.PcapFilter;
+import org.apache.metron.pcap.filter.PcapFilterConfigurator;
+import org.apache.metron.pcap.filter.PcapFilters;
+
+import java.util.EnumMap;
+import java.util.Map;
+
+public class QueryPcapFilter implements PcapFilter {
+ public static final String QUERY_STR_CONFIG = "mql";
+
+ public static class Configurator implements
PcapFilterConfigurator {
+@Override
+public void addToConfig(String query, Configuration conf) {
+ conf.set(QUERY_STR_CONFIG, query);
+ conf.set(PCAP_FILTER_NAME_CONF, PcapFilters.QUERY.name());
+}
+
+@Override
+public String queryToString(String fields) {
+ return fields.replaceAll("\\w", "_")
--- End diff --
I think you mean "\\s" there, not "\\w". \\w matches word, so the output
of this will be "___ ___" for input "foo bar". Might be worth while to add a
unit test around that too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63102232
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
---
@@ -33,36 +33,39 @@
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.log4j.Logger;
-import org.apache.metron.common.Constants;
import org.apache.metron.pcap.PcapHelper;
+import org.apache.metron.pcap.filter.PcapFilter;
+import org.apache.metron.pcap.filter.PcapFilterConfigurator;
+import org.apache.metron.pcap.filter.PcapFilters;
-import javax.annotation.Nullable;
import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
+import java.util.stream.Collectors;
public class PcapJob {
private static final Logger LOG = Logger.getLogger(PcapJob.class);
-
public static class PcapMapper extends Mapper {
public static final String START_TS_CONF = "start_ts";
public static final String END_TS_CONF = "end_ts";
PcapFilter filter;
long start;
long end;
+
@Override
protected void setup(Context context) throws IOException,
InterruptedException {
super.setup(context);
- filter = new PcapFilter(context.getConfiguration());
+ filter =
PcapFilters.valueOf(context.getConfiguration().get(PcapFilterConfigurator.PCAP_FILTER_NAME_CONF)).create();
+ filter.configure(context.getConfiguration());
start =
Long.parseUnsignedLong(context.getConfiguration().get(START_TS_CONF));
end =
Long.parseUnsignedLong(context.getConfiguration().get(END_TS_CONF));
}
@Override
protected void map(LongWritable key, BytesWritable value, Context
context) throws IOException, InterruptedException {
if(Long.compareUnsigned(key.get() ,start) >= 0 &&
Long.compareUnsigned(key.get(), end) <= 0) {
-boolean send =
Iterables.size(Iterables.filter(PcapHelper.toPacketInfo(value.copyBytes()),
filter)) > 0;
+boolean send =
PcapHelper.toPacketInfo(value.copyBytes()).stream().filter(filter).collect(Collectors.toList()).size()
> 0;
--- End diff --
That `collect(Collectors.toList())` is going to cause the whole stream to
be processed (gotta make that list). I think maybe what we want here is
`!Iterables.isEmpty(Iterables.filter(...))` as that will shortcut after the
first Iterator.hasNext() returns true.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63102275
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
---
@@ -33,36 +33,39 @@
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.log4j.Logger;
-import org.apache.metron.common.Constants;
import org.apache.metron.pcap.PcapHelper;
+import org.apache.metron.pcap.filter.PcapFilter;
+import org.apache.metron.pcap.filter.PcapFilterConfigurator;
+import org.apache.metron.pcap.filter.PcapFilters;
-import javax.annotation.Nullable;
import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
+import java.util.stream.Collectors;
public class PcapJob {
private static final Logger LOG = Logger.getLogger(PcapJob.class);
-
public static class PcapMapper extends Mapper {
public static final String START_TS_CONF = "start_ts";
public static final String END_TS_CONF = "end_ts";
PcapFilter filter;
long start;
long end;
+
@Override
protected void setup(Context context) throws IOException,
InterruptedException {
super.setup(context);
- filter = new PcapFilter(context.getConfiguration());
+ filter =
PcapFilters.valueOf(context.getConfiguration().get(PcapFilterConfigurator.PCAP_FILTER_NAME_CONF)).create();
+ filter.configure(context.getConfiguration());
start =
Long.parseUnsignedLong(context.getConfiguration().get(START_TS_CONF));
end =
Long.parseUnsignedLong(context.getConfiguration().get(END_TS_CONF));
}
@Override
protected void map(LongWritable key, BytesWritable value, Context
context) throws IOException, InterruptedException {
if(Long.compareUnsigned(key.get() ,start) >= 0 &&
Long.compareUnsigned(key.get(), end) <= 0) {
-boolean send =
Iterables.size(Iterables.filter(PcapHelper.toPacketInfo(value.copyBytes()),
filter)) > 0;
+boolean send =
PcapHelper.toPacketInfo(value.copyBytes()).stream().filter(filter).collect(Collectors.toList()).size()
> 0;
--- End diff --
It's also clearer of intent (not empty as opposed to size > 0)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/119#issuecomment-218894825 On the whole, this is great. Definitely a great feature and an impressive 2nd PR. Thanks for the contribution! Please make sure you didn't inadvertently regress the existing pcap functionality on the `full-dev-vagrant` image by following the testing plan in the original PR to move PCAP to hbase [here](https://github.com/apache/incubator-metron/pull/93) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63105345
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java
---
@@ -18,12 +18,20 @@
package org.apache.metron.common.query;
-import org.antlr.v4.runtime.*;
-import org.apache.metron.common.query.generated.*;
+import org.antlr.v4.runtime.ANTLRInputStream;
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.TokenStream;
+import org.apache.metron.common.query.generated.PredicateLexer;
+import org.apache.metron.common.query.generated.PredicateParser;
+
+import static org.apache.commons.lang3.StringUtils.isEmpty;
public class PredicateProcessor {
public boolean parse(String rule, VariableResolver resolver) {
+if (isEmpty(rule)) {
--- End diff --
Right, I wasn't worried about empty values (your case covered that); I was
worried about whitespace-only queries. In that case
`PredicateProcessor.parse(" ") != PredicateParser.parse("")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63111447
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java
---
@@ -18,12 +18,20 @@
package org.apache.metron.common.query;
-import org.antlr.v4.runtime.*;
-import org.apache.metron.common.query.generated.*;
+import org.antlr.v4.runtime.ANTLRInputStream;
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.TokenStream;
+import org.apache.metron.common.query.generated.PredicateLexer;
+import org.apache.metron.common.query.generated.PredicateParser;
+
+import static org.apache.commons.lang3.StringUtils.isEmpty;
public class PredicateProcessor {
public boolean parse(String rule, VariableResolver resolver) {
+if (isEmpty(rule)) {
--- End diff --
That works. || or && are equivalent in this situation
On Thu, May 12, 2016 at 18:56 Michael Miklavcic
wrote:
> In
>
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java
> <https://github.com/apache/incubator-metron/pull/119#discussion_r63111019>
> :
>
> >
> >
> > public class PredicateProcessor {
> >public boolean parse(String rule, VariableResolver resolver) {
> > +if (isEmpty(rule)) {
>
> On second thought, what about rule == null || isEmpty(rule.trim())? I
> can't think of a reason to handle null different from "" or " ". The trim
> would only be evaluated if rule != null, and now we're also covering the
> null condition. Thoughts?
>
> â
> You are receiving this because you commented.
> Reply to this email directly or view it on GitHub
>
<https://github.com/apache/incubator-metron/pull/119/files/b3af68484718a66393f4b0417bc525c462afed5d#r63111019>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63112571
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java
---
@@ -18,12 +18,20 @@
package org.apache.metron.common.query;
-import org.antlr.v4.runtime.*;
-import org.apache.metron.common.query.generated.*;
+import org.antlr.v4.runtime.ANTLRInputStream;
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.TokenStream;
+import org.apache.metron.common.query.generated.PredicateLexer;
+import org.apache.metron.common.query.generated.PredicateParser;
+
+import static org.apache.commons.lang3.StringUtils.isEmpty;
public class PredicateProcessor {
public boolean parse(String rule, VariableResolver resolver) {
+if (isEmpty(rule)) {
--- End diff --
wouldn't be an NPE on `rule != null && isEmpty(rule.trim())` Java would
shortcircuit before the `rule.trim()` call.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63114735
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java
---
@@ -18,12 +18,20 @@
package org.apache.metron.common.query;
-import org.antlr.v4.runtime.*;
-import org.apache.metron.common.query.generated.*;
+import org.antlr.v4.runtime.ANTLRInputStream;
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.TokenStream;
+import org.apache.metron.common.query.generated.PredicateLexer;
+import org.apache.metron.common.query.generated.PredicateParser;
+
+import static org.apache.commons.lang3.StringUtils.isEmpty;
public class PredicateProcessor {
public boolean parse(String rule, VariableResolver resolver) {
+if (isEmpty(rule)) {
--- End diff --
I vote for 2
On Thu, May 12, 2016 at 19:36 Michael Miklavcic
wrote:
> In
>
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java
> <https://github.com/apache/incubator-metron/pull/119#discussion_r63114632>
> :
>
> >
> >
> > public class PredicateProcessor {
> >public boolean parse(String rule, VariableResolver resolver) {
> > +if (isEmpty(rule)) {
>
> To clarify, I've added the following test cases:
>
> Assert.assertTrue(run(null, v -> variableMap.get(v)));
> Assert.assertTrue(run("", v -> variableMap.get(v)));
> Assert.assertTrue(run(" ", v -> variableMap.get(v)));
>
> The null results in an NPE from ANTLR.
> Either we:
> 1. Add an else to return true
> 2. Do this(my vote): if (rule == null || isEmpty(rule.trim())) { return
> true; }
> 3. Or throw a new exception for the parser
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
>
<https://github.com/apache/incubator-metron/pull/119/files/b3af68484718a66393f4b0417bc525c462afed5d#r63114632>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/119#discussion_r63115005
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java
---
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.filter.query;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.query.PredicateProcessor;
+import org.apache.metron.common.query.VariableResolver;
+import org.apache.metron.pcap.PacketInfo;
+import org.apache.metron.pcap.PcapHelper;
+import org.apache.metron.pcap.filter.PcapFilter;
+import org.apache.metron.pcap.filter.PcapFilterConfigurator;
+import org.apache.metron.pcap.filter.PcapFilters;
+
+import java.util.EnumMap;
+import java.util.Map;
+
+public class QueryPcapFilter implements PcapFilter {
+ public static final String QUERY_STR_CONFIG = "mql";
+
+ public static class Configurator implements
PcapFilterConfigurator {
+@Override
+public void addToConfig(String query, Configuration conf) {
+ conf.set(QUERY_STR_CONFIG, query);
+ conf.set(PCAP_FILTER_NAME_CONF, PcapFilters.QUERY.name());
+}
+
+@Override
+public String queryToString(String fields) {
+ return fields.replaceAll("\\w", "_")
--- End diff --
Actually the safest thing here is to return "". There is already a UUID in
the file name to ensure uniqueness.
On Thu, May 12, 2016 at 19:28 Michael Miklavcic
wrote:
> In
>
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java
> <https://github.com/apache/incubator-metron/pull/119#discussion_r63113925>
> :
>
> > +import java.util.EnumMap;
> > +import java.util.Map;
> > +
> > +public class QueryPcapFilter implements PcapFilter {
> > + public static final String QUERY_STR_CONFIG = "mql";
> > +
> > + public static class Configurator implements
PcapFilterConfigurator {
> > +@Override
> > +public void addToConfig(String query, Configuration conf) {
> > + conf.set(QUERY_STR_CONFIG, query);
> > + conf.set(PCAP_FILTER_NAME_CONF, PcapFilters.QUERY.name());
> > +}
> > +
> > +@Override
> > +public String queryToString(String fields) {
> > + return fields.replaceAll("\\w", "_")
>
> Will definitely want feedback on the formatting on this one since we're
> using the string rep for filenames.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
>
<https://github.com/apache/incubator-metron/pull/119/files/b3af68484718a66393f4b0417bc525c462afed5d#r63113925>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-219048736
Ok, tested this. Sorry it took so long (vagrant drama).
Ok, so this is what I did to test this.
Spin up the full-dev-vagrant:
1. Do a build by running `mvn clean integration-test` from the
`metron-platform` directory
2. From the `metron-deployment/vagrant/full-dev-platform` directory run
`vagrant up`
* Note: If you already have a vagrant machine running and you just want
to
redeploy code, you should be able to do `vagrant provision --tags
enrichment`
Log into the full-dev-vagrant and do the following:
* Optionally, stop some things to give myself enough memory headroom:
* Kill every running parser topology via the [storm
UI](http://node1:8744/index.html) to proceed (that VM is pretty packed atm).
* Stop pcap replay via `/etc/init.d/pcap-replay stop`
* Kill bro via `/usr/local/bin/broctl` and type `stop` and then `exit`
at the broctl prompt
* Marvel at the lack of fans running on your laptop
* Create the kafka topic for the sensor:
`/usr/hdp/current/kafka-broker/bin//kafka-topics.sh --zookeeper localhost:2181
--create --topic websphere --partitions 1 --replication-factor 1`
* Create text file with sample data in it called `WebsphereOutput.txt`.
This is essentially just
[WebsphereOutput.txt](https://raw.githubusercontent.com/DomenicPuzio/incubator-metron/af50623dcb764fda0281b3657c0f40c993c958f3/metron-platform/metron-integration-test/src/main/resources/sample/data/SampleInput/WebsphereOutput.txt)
* Start the parser topology using
`/usr/metron/0.1BETA/bin/start_parser_topology.sh`, in this case
`/usr/metron/0.1BETA/bin/start_parser_topology.sh websphere`
* Add your data to the kafka queue you created earlier via `cat
WebsphereOutput.txt |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic websphere`
* Wait a minute and then check the elasticsearch head plugin (if you
haven't installed it, you can via `/usr/share/elasticsearch/bin/plugin -install
mobz/elasticsearch-head`).
* Browse to the `Browser` tab and click on the index named
`${sensor_name}_index_${DATE}` (in my case: `websphere_index_2016.05.13.13`
* You should see some messages. Click on each one and inspect the JSON
to ensure it looks right.
For future parsers, this is a good template for acceptance testing. You
shouldn't have to go through a full maven provisioning each time if you
keep the VM around and just run `vagrant provision -tags enrichment`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/119#issuecomment-219049457 @nickwallen Agreed we should support BPF. This PR makes the filter pluggable and we already have the query language. We can have a follow-on PR for BPF support IMO. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/115#issuecomment-219048777 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Added vagrant install hostmanager s...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/121#issuecomment-219068764 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-132 Monit Integration
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/118#issuecomment-219158724 +1, looks great! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 142 Simplify Parser configur...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/120#issuecomment-219424532 This looks good to me. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/119#issuecomment-219425116 +1, got my vote. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Specify shorter control_path to avo...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/122#issuecomment-220619587 +1 this looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-156 Fixed Docs for Vagrant D...
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/124#discussion_r64049158 --- Diff: metron-deployment/vagrant/full-dev-platform/README.md --- @@ -62,41 +60,46 @@ Now that the hard part is done, start the Metron deployment process. vagrant provision ``` - In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. +### Explore Metron - ``` - ./run_ansible_role.sh web - ``` - or - ``` - vagrant --ansible-tags="web" provision - ``` - Will re-run the web role on the Vagrant image. This will re-install (if necessary) and start the UI. +Navigate to the following resources to explore your newly minted Apache Metron environment. - A collection of tags is specified as a comma separated list. + - [Metron](http://node1:8080) + - [Ambari](http://node1:5000) + - [Services](http://node1:2812) - ``` - ./run_ansbile_role.sh "sensors,enrichment" +Connecting to the host through SSH is as simple as running the following command. - ``` +``` +vagrant ssh +``` - Tags are listed in the playbooks, some frequently used tags: - + hdp-install - Install HDP - + hdp-deploy - Deploy and Start HDP Services (will start all Hadoop Services) - + sensors - Deploy and Start Sensors. - + enrichment - Deploy and Start Enrichment Topology. +### Working with Metron - Note: there is a convienence script, ```./run_enrichment_role.sh```, which runs the enrichment tag. +In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. The following commands will re-run the `web` role on the Vagrant image. This will install components (if necessary) and start the UI. -### Explore Metron +``` +./run_ansible_role.sh web +``` +or -Navigate to the following resources to explore your newly minted Apache Metron environment. +``` +vagrant --ansible-tags="web" provision +``` - - [Metron](http://node1:8080) - - [Ambari](http://node1:5000) + Using Tags -Connecting to the host through SSH is as simple as running the following command. +A collection of tags is specified as a comma separated list. + +``` +./run_ansbile_role.sh "sensors,enrichment" --- End diff -- This looks like a typo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron-126 added dfs.replication=1 ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/125#issuecomment-220620508 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-173 Updating the the Apache ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/126#issuecomment-220634820 +1, this looks great --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/127 METRON-174 Storm consumption of hbase enrichment reference data We should support streaming enrichment data into kafka and writing it out to HBase in a format suitable to be used in either the Simple HBase Enrichment Adapter or the Simple HBase Threat Intel Adapter. This should be fully backwards compatible with how we did parser topologies before, so you should be able to spin up a vagrant image and see data go through the indices. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-174 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/127.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #127 commit f9551fc597e29163245aea586b6ee2bb4c5b1150 Author: cstella Date: 2016-05-20T14:42:12Z METRON-174 Allow simple hbase enrichment reference data to be streamed into HBase via storm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-156 Fixed Docs for Vagrant D...
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/124#discussion_r64060254 --- Diff: metron-deployment/vagrant/full-dev-platform/README.md --- @@ -62,41 +60,46 @@ Now that the hard part is done, start the Metron deployment process. vagrant provision ``` - In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. +### Explore Metron - ``` - ./run_ansible_role.sh web - ``` - or - ``` - vagrant --ansible-tags="web" provision - ``` - Will re-run the web role on the Vagrant image. This will re-install (if necessary) and start the UI. +Navigate to the following resources to explore your newly minted Apache Metron environment. - A collection of tags is specified as a comma separated list. + - [Metron](http://node1:8080) + - [Ambari](http://node1:5000) + - [Services](http://node1:2812) - ``` - ./run_ansbile_role.sh "sensors,enrichment" +Connecting to the host through SSH is as simple as running the following command. - ``` +``` +vagrant ssh +``` - Tags are listed in the playbooks, some frequently used tags: - + hdp-install - Install HDP - + hdp-deploy - Deploy and Start HDP Services (will start all Hadoop Services) - + sensors - Deploy and Start Sensors. - + enrichment - Deploy and Start Enrichment Topology. +### Working with Metron - Note: there is a convienence script, ```./run_enrichment_role.sh```, which runs the enrichment tag. +In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. The following commands will re-run the `web` role on the Vagrant image. This will install components (if necessary) and start the UI. -### Explore Metron +``` +./run_ansible_role.sh web +``` +or -Navigate to the following resources to explore your newly minted Apache Metron environment. +``` +vagrant --ansible-tags="web" provision +``` - - [Metron](http://node1:8080) - - [Ambari](http://node1:5000) + Using Tags -Connecting to the host through SSH is as simple as running the following command. +A collection of tags is specified as a comma separated list. + +``` +./run_ansbile_role.sh "sensors,enrichment" --- End diff -- `ansbile_role.sh` should be `ansible_role.sh` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 145: Automate Creation of Va...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/123#issuecomment-220647462 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 173
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/128#issuecomment-220718342 Metron 173 is already merged. It looks like you have another JIRA (Metron 177) for the typos, please open up another PR with just the Metron-177 changes in them. In other words, create another branch from master called METRON-177 and make your typo changes there and then submit THAT branch as a PR against master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/129 METRON-178 Expose the filter capability in the Parser topologies. Allow users to specify the MessageFilter to use in the parser topologies. We already have this capability, we need to expose it via the zookeeper configs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron expose_parser_filter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/129.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #129 commit e3f86085a73656920c2f7f6be1e3e8273df8f210 Author: cstella Date: 2016-05-20T21:27:39Z METRON-178 Expose the filter capability in the Parser topologies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-177 Fix typos on the Metron ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/130#issuecomment-220976450 Ok, this looks good, +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 173
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/128#issuecomment-220976859 Since you made a METRON-177 PR, could you please close this one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-179 Fixed Docs for Vagrant D...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/124#issuecomment-220977119 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/131 METRON-183 Allow the simple hbase enrichment adapter and simple threat intel adapter to use multiple column families Allow the simple hbase enrichment adapter and simple threat intel adapter to use multiple column families. As it stands, the table and column family used are configured at topology submission time rather than via zookeeper. It makes sense to allow different enrichment types to be associated with different column families so that column family specific configuration, such as retention policies, can be used per enrichment type. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron cf_per_et Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/131.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #131 commit 00a0b13a5b307be39aad45031c1673de9a2f5ea4 Author: cstella Date: 2016-05-23T17:50:40Z Added the ability to support multiple column families in the simple hbase enrichment and threat intel. commit adc3765fcf84bf24d58ac8a4e5e20851ffae Author: cstella Date: 2016-05-23T20:46:35Z Updating documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/129#discussion_r64495854
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -59,10 +61,22 @@ public ParserBolt
withMessageFilter(MessageFilter filter) {
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
super.prepare(stormConf, context, collector);
this.collector = collector;
+if(getSensorParserConfig() == null) {
+ filter = new GenericMessageFilter(null);
+}
+else if(filter == null) {
+ filter = Filters.get(getSensorParserConfig().getFilterClassName()
+ , getSensorParserConfig().getParserConfig()
+ );
+}
parser.init();
writer.init();
}
+ protected SensorParserConfig getSensorParserConfig() {
--- End diff --
yeah, that's a good catch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/129#issuecomment-221437141 Yeah, upon further consideration, I think that probably an init method is probably the best thing for MessageFilter. I'll refactor. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/129#discussion_r64564714
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -59,10 +61,22 @@ public ParserBolt
withMessageFilter(MessageFilter filter) {
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
super.prepare(stormConf, context, collector);
this.collector = collector;
+if(getSensorParserConfig() == null) {
+ filter = new GenericMessageFilter(null);
+}
+else if(filter == null) {
+ filter = Filters.get(getSensorParserConfig().getFilterClassName()
+ , getSensorParserConfig().getParserConfig()
+ );
+}
parser.init();
writer.init();
}
+ protected SensorParserConfig getSensorParserConfig() {
--- End diff --
When I went to change it, I realized why I put it in the ParserBolt rather
than the ConfiguredParserBolt. The sensorType is part of the ParserBolt rather
than the ConfiguredParserBolt. I am going to make them both part of
ConfiguredParserBolt.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-100 GeoIP errors out silentl...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/134 METRON-100 GeoIP errors out silently in vagrant When we transitioned from passing the value to the adapters to passing a CacheKey object, we never adjusted the SQL statements. Also, this error may cause downstream issues whereby messages error out because enrichments are short circuited if one fails and the source.type never gets properly set. The JIRA is wrong, this bug is actually present in every environment and prevents GeoIP enrichment from working. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron enrichment_robustness Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/134.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #134 commit 6fafdcf66df4a1d1df21672f2644fd00953e75f9 Author: cstella Date: 2016-05-24T22:26:37Z Making the bolts and adapters slightly more robust so that failures for a single enrichment do not exception out and kill the topology. commit 865cbc942ceff92e47ca83e664ae6bc9c3f3cd5a Author: cstella Date: 2016-05-25T00:34:25Z Continued robustness improvements. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/133#issuecomment-221577078 Yeah, this makes sense. Do we need to adjust the `run_ansible_role.sh` to pass in the `--skip-tags` arg as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-184 Fixed 'creates' path to ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/132#issuecomment-221577928 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/133#issuecomment-221611276 Also, is there an ec2.py somewhere that didn't get checked in? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-186: Create a fieldMapping f...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/136 METRON-186: Create a fieldMapping functionality which allows for parsed fields to be transformed Currently the parsers take care of transforming raw data to the parsed JSON representation. Allow for a layer to be placed at a sensor level to transform input fields from the parsed messages to create new fields. For instance, mapping IANA protocol numbers to a standardized textual representation (i.e. 6 maps to TCP). This should be generalized, so that we can provide common functions and users can define the input fields and the output field to use. This configuration should live with the sensor configuration in zookeeper. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron transformers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/136.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #136 commit 0d3b31be29b4238e1923d4d812e40dca4afe64f2 Author: cstella Date: 2016-05-25T19:01:55Z Creating mapping infrastructure for parsing. commit 7e1366735a0ed8eb78ec7103759b6a2574beff2f Author: cstella Date: 2016-05-25T21:16:47Z Renamed and refactored a bit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/127#discussion_r64669063
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -60,7 +93,25 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
this.collector = collector;
parser.init();
-writer.init();
+
+if(isBulk) {
+ writerTransformer = config -> new ParserWriterConfiguration(config);
--- End diff --
The config object can change if zookeeper is updated, so we want an
indirection here.
On Wed, May 25, 2016 at 19:11 merrimanr wrote:
> In
>
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
> <https://github.com/apache/incubator-metron/pull/127#discussion_r64668872>
> :
>
> > @@ -60,7 +93,25 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
> > super.prepare(stormConf, context, collector);
> > this.collector = collector;
> > parser.init();
> > -writer.init();
> > +
> > +if(isBulk) {
> > + writerTransformer = config -> new
ParserWriterConfiguration(config);
>
> Curious why a Function is used here. Why not just instantiate a
> WriterConfiguration object for each case and pass that to
> messageWriter.init? Is there a benefit to doing it this way?
>
> â
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
>
<https://github.com/apache/incubator-metron/pull/127/files/e448abd8cccf6f912095474f034668198e58f442#r64668872>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/135#issuecomment-221739471 +1, spun up in single node vagrant and looks good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-186: Create a fieldMapping f...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/136#issuecomment-221862988 Yeah, I was thinking about that too. I think transformation is a more descriptive term. You're the 2nd person (in addition to myself) who has made that comment. I think that means I should call it transformation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-100 GeoIP errors out silentl...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/134#discussion_r64742580
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/jdbc/JdbcAdapter.java
---
@@ -38,6 +39,27 @@
private JdbcConfig config;
private String host;
+ protected boolean isConnectionClosed() {
+boolean isClosed = statement == null || connection == null;
+if(!isClosed) {
+ try {
+isClosed = statement.isClosed() || connection.isClosed();
+ } catch (SQLException e) {
+_LOG.error("Unable to maintain open JDBC connection: " +
e.getMessage(), e);
+isClosed = true;
+ }
+}
+return isClosed;
+ }
+
+ protected boolean resetConnectionIfNecessary() {
+if(isConnectionClosed())
+{
--- End diff --
So, I'll definitely make the class consistently K&R (I must've fat-fingered
the newline). We don't have a coherent style at the moment in this respect. I
prefer K&R bracing, so you'll see that in my code, but yeah, I agree, we need
to correct style as we see it.
Also, don't worry about nitpicking..we all have to live in this code. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-189: Add the ability to do g...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/138 METRON-189: Add the ability to do global validations on messages passing through the parser. Allow the user to specify field level or message level validations to ensure messages coming from the parser are valid. For instance, allow the ability ensure that a field is an IPv4 address. If a field is invalid, send to a separate stream from the parser bolt. Follow-on work should be done to send this stream to the index for after-the-fact inspection. I added the following validation functions: * `MQL` : Execute a Query Language statement. Expects the query string in the `condition` field of the config. * `IP` : Validates that the input fields are an IP addres. By default, if no configuration is set, it assumes `IPV4`, but you can specifythe type by passing in the config by passing in `type` with either `IPV6` or `IPV4`. * `DOMAIN` : Validates that the fields are all domains. * `EMAIL` : Validates that the fields are all email addresses * `URL` : Validates that the fields are all URLs * `DATE` : Validates that the fields are a date. Expects `format` in the config. * `INTEGER` : Validates that the fields are an integer. String representation of an integer is allowed. * `REGEX_MATCH` : Validates that the fields match a regex. Expects `pattern` in the config. * `NOT_EMPTY` : Validates that the fields exist and are not empty (after trimming.) Because of the nice overlap, I also added these functions to the query language, so query language rules can take advantage of `IS_IP(field1)` for instance. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron validation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/138.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #138 commit 258ae1cde4cdb5092edeb0dc6a50f0af2c1c Author: cstella Date: 2016-05-27T02:46:35Z Added validation framework. commit d2ae7b3a8a96787523755ce0107c64909e2729bd Author: cstella Date: 2016-05-27T03:39:52Z Updating validators to work with unit tests. commit ccc42b3d7390e98ade326c5651719d9b6d4533b8 Author: cstella Date: 2016-05-27T13:50:30Z Updating readme. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/133#issuecomment-222159692 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/139
METRON-190: Make start_parser_topology.sh more adaptable regarding storm
topology configuration
Add the ability to specify the message timeout, the max task parallelism,
the number of ackers, the number of workers. Also allow the user to pass in
JSON file containing a map of extra storm configs.
I added the following extra options to `start_parser_topology.sh`:
```
-e,--extra_options Extra options in the form of a
JSON file with a map for
content.
-mt,--message_timeoutMessage Timeout in Seconds
-mtp,--max_task_parallelism Max task parallelism
-na,--num_ackers Number of Ackers
-nw,--num_workersNumber of Workers
```
For instance, if you wanted to set some storm property on
the config called `topology.ticks.tuple.freq.secs` to 1000 and
`storm.local.dir` to `/opt/my/path`
you could create a file called `custom_config.json` containing
```
{
"topology.ticks.tuple.freq.secs" : 1000,
"storm.local.dir" : "/opt/my/path"
}
```
and pass `--extra_options custom_config.json` to `start_parser_topology.sh`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron
parser_parallelism_expose
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/139.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #139
commit 33815e0382b54fb6931bae4579223535e7abdc90
Author: cstella
Date: 2016-05-27T19:18:12Z
Exposed more storm topology config options.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-53339 Ok, I verified this on full dev vagrant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-222313782 @james-sirota Right you are! Just updated exposing the numTasks for the spout and the parser bolt. ``` -pnt,--parser_num_tasksParser Num Tasks -pp,--parser_p Parser Parallelism -snt,--spout_num_tasksSpout Num Tasks -sp,--spout_p Spout Parallelism ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222363251
The way to validate this is to
* Create a second column family on the enrichment HBase table, say `cf1`
* Push some enrichment data into the table in that column family.
Something similar to the malicious IP's from the blog post. I'd suggest
calling the enrichment type `malicious_ip`
* Modify the configs for one of the topologies you want to test in
`/usr/metron/0.1BETA/config/zookeeper/enrichment` to add a `config` section
under `threatIntel` marking that `malicious_ip` should come from column family
`cf1` like the following
```
{
"index": "bro",
"batchSize": 5,
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel" : [ "ip_dst_addr" ]
},
"fieldToTypeMap": {
"ip_dst_addr" : [ "malicious_ip" ]
},
"config" : {
"typeToColumnFamily" : {
"malicious_ip" : "cf1"
}
}
}
}
```
* Run some data through and ensure that enrichments still exist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/131#issuecomment-222363326 As it stands, we have the docs for the configs with where the configurations live. I think you are absolutely right that we need to move them closer to the place where they are used, rather than where the configuration objects are defined. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-222363512 good catch, I'll make those changes now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-189: Add the ability to do g...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/138#issuecomment-222363626 Yeah, the documentation situation is a bit incorrect. Documentation for configuration started to be placed where the configuration objects existed, rather than where they were used. Since we put the configuration in commons, then the documentation went in commons. I suggest strongly that we do a follow-on after we flush the PR queue to move docs around. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/127#issuecomment-222364875
In order to validate this, you can do the following:
* Configure a new parser, in this example I'll call it a `user` parser and
we'll parse some CSV data to map `username` to `ip` by creating a file
`/usr/metron/0.1BETA/config/zookeeper/enrichment/user.json` with
```
{
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
,"writerClassName" :
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
,"sensorTopic":"user"
,"parserConfig":
{
"shew.table" : "enrichment"
,"shew.cf" : "t"
,"shew.keyColumns" : "user"
,"shew.enrichmentType" : "user"
,"columns" : {
"user" : 0
,"ip" : 1
}
}
}
```
* Add a new `user` enrichment type to `bro` data by adding `ip_src_addr` to
`hbaseEnrichment` and associating `user` as a field type for `ip_src_addr` in
`/usr/metron/0.1BETA/config/zookeeper/enrichment/bro.json` like so
```
{
"index": "bro",
"batchSize": 5,
"enrichment": {
"fieldMap": {
"geo": [
"ip_dst_addr",
"ip_src_addr"
],
"host": [
"host"
],
"hbaseEnrichment" : [ "ip_src_addr" ]
},
"fieldToTypeMap":
{
"ip_src_addr" : [ "user"]
}
},
"threatIntel":{
"fieldMap":
{
"hbaseThreatIntel": ["ip_dst_addr", "ip_src_addr"]
},
"fieldToTypeMap":
{
"ip_dst_addr" : [ "malicious_ip" ]
,"ip_src_addr" : [ "malicious_ip" ]
}
}
}```
* Create the Kafka Queue as in the tutorials
* Using `/usr/metron/0.1BETA/bin/zk_load_configs.sh` push up the config you
just created. `/usr/metron/0.1BETA/bin/zk_load_configs.sh -m PUSH -z node1:2181
-i /usr/metron/0.1BETA/config/zookeeper`
* Create some reference CSV reference data with that looks like
`jsirota,192.168.168.1` into a csv file named `user.csv`
* Use the kafka console producer to push data into the `user` topic via
`cat user.csv | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
--broker-list node1:6667 --topic user`
* You should be able to check that the data gets into HBase by doing a
`scan 'enrichment'` from the `hbase shell`
* You should also be able to check, after new data has been run through,
that the data is enriched in elasticsearch. I would suggest bouncing the
enrichment topology to ensure that stale data in the caches get flushed, but
that is not strictly necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222452241 @james-sirota Did you push the new parser config to zookeeper via `/usr/metron/0.1BETA/bin/zk_load_configs.sh -m PUSH -z node1:2181 -i /usr/metron/0.1BETA/config/zookeeper` before you tried to start the topology? If not, then you must do that..it won't read off of disk, it reads from zookeeper. I'll make that not a NPE so it's more clear, though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222452433 If you did push the config before trying to start the parser, then please confirm that the `user` topology is in zookeeper via inspecting the output of `/usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z node1:2181` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request:
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/commit/ab8163bcc64d4b725ad61a5f6d8a74aad812a24a#commitcomment-17664548
You should not need the zookeeper information in those config objects since
it will get passed via the GenericEnrichmentBolt in the topology (the
quorum in the form host:port is passed as a constructor arg to those
bolts). I don't see the reason why you added this. Maybe you can explain
what you were doing to require this?
Casey
On Mon, May 30, 2016 at 2:09 AM, AromalAS wrote:
> While testing the ThreatIntelAdapter in my devlopment area,
> I have to add zookeeper configuration also in the configuration file to
> make it work.
>
> I have done below changes . Please confirm whether this code changes is
> absolutely necessary ?.
>
> Similar changes were done for the SimpleHBaseAdapter as well
> Remote.yaml
>
>-
>
>id: "simpleHBaseThreatIntelConfig"
>className:
>"org.apache.metron.enrichment.adapters.threatintel.ThreatIntelConfig"
>configMethods:
>
>- name: "withQuorum"
>args:
>- "${zk.ips}"
>- name: "withZkclientPort"
>args:
>- "${zk.port}"
>
>
> ThreatIntelConfig.java
>
> private String quorum;
>
> private String zkclientPort;
>
> public String getQuorum() {
> return quorum;
> }
> public void withQuorum(String quorum) {
> this.quorum = quorum;
> }
> public String getZkclientPort() {
> return zkclientPort;
> }
> public void withZkclientPort(String zkclientPort) {
> this.zkclientPort = zkclientPort;
> }
> ThreatIntelAdapter.java
>
> @Override <https://github.com/Override>
> public boolean initializeAdapter() {
> PersistentAccessTracker accessTracker;
> String hbaseTable = config.getHBaseTable();
> int expectedInsertions = config.getExpectedInsertions();
> double falsePositives = config.getFalsePositiveRate();
> String trackerHBaseTable = config.getTrackerHBaseTable();
> String trackerHBaseCF = config.getTrackerHBaseCF();
> long millisecondsBetweenPersist = config.getMillisecondsBetweenPersists();
> BloomAccessTracker bat = new BloomAccessTracker(hbaseTable,
> expectedInsertions, falsePositives);
> Configuration hbaseConfig = HBaseConfiguration.create();
> //added the below lines
> hbaseConfig.set("hbase.zookeeper.quorum", config.getQuorum());
> hbaseConfig.set("hbase.zookeeper.property.clientPort",
> config.getZkclientPort());
>
> }
>
> â
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/incubator-metron/commit/ab8163bcc64d4b725ad61a5f6d8a74aad812a24a#commitcomment-17662283>,
> or mute the thread
>
<https://github.com/notifications/unsubscribe/AAg-x66l3CdRyQGQ5PVE7Ld0WlBPHpmiks5qGn78gaJpZM4Ipg7M>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/127#issuecomment-222534318
Looks like it can't find the writerClassname field. Are you sure you ran a
build from this branch before the deploy?
On Mon, May 30, 2016 at 13:48 James Sirota wrote:
> /usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z 1xxx:2181
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> GLOBAL Config: global
> {
> "es.clustername": "metron",
> "es.ip": "xxx",
> "es.port": "9300",
> "es.date.format": ".MM.dd.HH"
> }
>
> PARSER Config: websphere
> {
>
>
"parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
> "sensorTopic":"websphere",
> "parserConfig":
> {
> "grokPath":"/patterns/websphere",
> "patternLabel":"WEBSPHERE",
> "timestampField":"timestamp_string",
> "dateFormat":" MMM dd HH:mm:ss"
> }
> }
>
> PARSER Config: bluecoat
> {
>
"parserClassName":"org.apache.metron.parsers.bluecoat.BasicBluecoatParser",
> "sensorTopic":"bluecoat",
> "parserConfig": {}
> }
>
> PARSER Config: squid
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "squid",
> "parserConfig": {
> "grokPath": "/patterns/squid",
> "patternLabel": "SQUID_DELIMITED",
> "timestampField": "timestamp"
> }
> }
>
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
>
>
> "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
> ,"writerClassName" :
> "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
> ,"sensorTopic":"user"
> ,"parserConfig":
> {
> "shew.table" : "enrichment"
> ,"shew.cf" : "t"
> ,"shew.keyColumns" : "user"
> ,"shew.enrichmentType" : "user"
> ,"columns" : {
> "user" : 0
> ,"ip" : 1
> }
> }
> }
>
> at
org.apache.metron.common.configuration.ConfigurationType.lambda$static$1(ConfigurationType.java:47)
> at
org.apache.metron.common.configuration.ConfigurationType$$Lambda$9/1684106402.apply(Unknown
Source)
> at
org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:78)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$0(ConfigurationsUtils.java:272)
> at
org.apache.metron.common.configuration.ConfigurationsUtils$$Lambda$7/785992331.visit(Unknown
Source)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:264)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:251)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:271)
> at
org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:115)
> at
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:177)
> at
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:161)
> at
org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:198)
>
> Caused by:
> com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
> Unrecognized field "writerClassName" (class
> org.apache.metron.common.configuration.SensorParserConfig), not marked as
> ignorable (3 known properties: , "parserConfig", "parserClassName",
> "sensorTopic"])
> at Source: java.io.StringReader@23bb844
>
<https://github.com/java.io.StringReader/incubator-metron/commit/23bb8443>;
> line: 3, column: 26
> at
>
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:79
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222638653 You sure Kafka is still up? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222640686 Try pulling data from that broker using the console consumer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #138: METRON-189: Add the ability to do global...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/138#discussion_r65462256
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -165,8 +174,17 @@ public void execute(Tuple tuple) {
}
}
+ private boolean isGloballyValid(JSONObject input, List
validators) {
+boolean ret = true;
+for(FieldValidator validator : validators) {
+ ret &= validator.isValid(input,
getConfigurations().getGlobalConfig());
--- End diff --
Yep, agreed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #138: METRON-189: Add the ability to do global...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/138#discussion_r65462284
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -140,18 +142,25 @@ public void execute(Tuple tuple) {
try {
boolean ackTuple = true;
if(sensorParserConfig != null) {
+List fieldValidations =
getConfigurations().getFieldValidations();
List messages = parser.parse(originalMessage);
for (JSONObject message : messages) {
if (parser.validate(message)) {
-if (filter != null && filter.emitTuple(message)) {
- ackTuple = !isBulk;
- message.put(Constants.SENSOR_TYPE, getSensorType());
- for (FieldTransformer handler :
sensorParserConfig.getFieldTransformations()) {
-if (handler != null) {
- handler.transformAndUpdate(message,
sensorParserConfig.getParserConfig());
+if(!isGloballyValid(message, fieldValidations)) {
+ message.put(Constants.SENSOR_TYPE, getSensorType()+
".invalid");
+ collector.emit(Constants.INVALID_STREAM, new
Values(message));
+}
+else if (filter != null && filter.emitTuple(message)) {
+ if (filter != null && filter.emitTuple(message)) {
--- End diff --
Yep, bad merge. Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #142: METRON-204: Field Transformation Domain ...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/142
METRON-204: Field Transformation Domain Specific Language
Similar to the domain specific query language, it would be nice to have a
domain specific language for transformations which is used as an optional
FieldTransformation implementation.
* A fixed set of transformation functions:
* `TO_LOWER(string)` : Transforms the first argument to a lowercase
string
* `TO_UPPER(string)` : Transforms the first argument to an uppercase
string
* `TO_STRING(string)` : Transforms the first argument to a string
* `TO_INTEGER(x)` : Transforms the first argument to an integer
* `TO_DOUBLE(x)` : Transforms the first argument to a double
* `TRIM(string)` : Trims whitespace from both sides of a string.
* `JOIN(list, delim)` : Joins the components of the list with the
specified delimiter
* `SPLIT(string, delim)` : Splits the string by the delimiter. Returns
a list.
* `GET_FIRST(list)` : Returns the first element of the list
* `GET_LAST(list)` : Returns the last element of the list
* `GET(list, i)` : Returns the i'th element of the list (i is 0-based).
* `MAP_GET(key, map, default)` : Returns the value associated with the
key in the map. If the key does not exist, the default will be returned. If
the default is unspecified, then null will be returned.
* `DOMAIN_TO_TLD(domain)` : Returns the TLD of the domain.
* `DOMAIN_REMOVE_TLD(domain)` : Remove the TLD of the domain.
* `REMOVE_TLD(domain)` : Removes the TLD from the domain.
* `URL_TO_HOST(url)` : Returns the host from a URL
* `URL_TO_PROTOCOL(url)` : Returns the protocol from a URL
* `URL_TO_PORT(url)` : Returns the port from a URL
* `URL_TO_PATH(url)` : Returns the path from a URL
* `TO_EPOCH_TIMESTAMP(dateTime, format, timezone)` : Returns the epoch
timestamp of the `dateTime` given the `format`. If the format does not have a
timestamp and you wish to assume a given timestamp, you may specify the
`timezone` optionally.
* A FieldTransformer implementation, `MTL` which exposes the transformation
language
Example MTL transformation:
Consider the following sensor parser config to add three new fields to a
message:
* `utc_timestamp` : The unix epoch timestamp based on the `timestamp`
field, a `dc` field which is the data center the message comes from and a
`dc2tz` map mapping data centers to timezones
* `url_host` : The host associated with the url in the `url` field
* `url_protocol` : The protocol associated with the url in the `url` field
```
{
...
"fieldTransformations" : [
{
"transformation" : "MTL"
,"output" : [ "utc_timestamp", "url_host", "url_protocol" ]
,"config" : {
"utc_timestamp" : "TO_EPOCH_TIMESTAMP(timestamp, '-MM-dd
HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
,"url_host" : "URL_TO_HOST(url)"
,"url_protocol" : "URL_TO_PROTOCOL(url)"
}
}
]
,"parserConfig" : {
"dc2tz" : {
"nyc" : "EST"
,"la" : "PST"
,"london" : "UTC"
}
}
}
```
Note that the `dc2tz` map is in the parser config, so it is accessible
in the functions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-204
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/142.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #142
commit cb9e925a199b3ee4b377f955066a959a91fc87c2
Author: cstella
Date: 2016-06-03T02:05:33Z
METRON-204: Field Transformation Domain Specific Language
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #143: METRON-197: Validation should be the las...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/143 METRON-197: Validation should be the last step in the ParserBolt Right now we are doing the validation prior to the messageFilter. We should only validate the parsed messages which passes through the filter. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-197 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/143.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #143 commit 53e3f61fe795b1620ee5847448ba1a2a13274120 Author: cstella Date: 2016-06-03T02:25:16Z Fixing global validation to run at the end of the validation pipeline. commit 134e4cd02a149452f9f774a6a85d6c7ccefabc17 Author: cstella Date: 2016-06-03T03:07:24Z Fixing ParserBolt to do validations in the proper place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 I will definitely create a doc on the JIRA about the DSL and would be happy to discuss. It's really just the extraction of the existing transformation functions from the query DSL into their own language. It seemed like a useful abstraction now that we have field transformations. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 Also, it seems like a big PR, but honestly it's deceptive. There was a lot of file moving and the generated Antlr code inflates the size. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 I uploaded a design doc to the JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #143: METRON-197: Validation should be the las...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/143#discussion_r66248395
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -143,14 +140,14 @@ public void execute(Tuple tuple) {
boolean ackTuple = true;
if(sensorParserConfig != null) {
List fieldValidations =
getConfigurations().getFieldValidations();
-List messages = parser.parse(originalMessage);
-for (JSONObject message : messages) {
- if (parser.validate(message)) {
+Optional> messages =
parser.parseOptional(originalMessage);
+for (JSONObject message :
messages.orElse(Collections.emptyList())) {
+ if (parser.validate(message) && filter != null &&
filter.emitTuple(message)) {
if(!isGloballyValid(message, fieldValidations)) {
message.put(Constants.SENSOR_TYPE, getSensorType()+
".invalid");
collector.emit(Constants.INVALID_STREAM, new
Values(message));
}
-else if (filter != null && filter.emitTuple(message)) {
+else {
ackTuple = !isBulk;
--- End diff --
The tuple will be acked. Note that ackTuple is initialized to true and is
only set to false if we are in a situation where we are doing a bulk write, in
which case the tuple will be acked in the bulk writer upon the writing of a
batch of entries. So, when this loop is ended, it will ack on line 164.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
