[GitHub] incubator-metron issue #519: METRON-832 Fixed CEF parser for Palo Alto FITW
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/519 +1 by inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/497 @mmiklavc Nice document! Planning to run through it myself later this week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #472: METRON-700: Add hadoop container to metron-dock...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/472 Thanks! Next time the storm Dockerfile needs a tweak, I'll rework it to use the packages approach too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #472: METRON-700: Add hadoop container to metron-dock...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/472 @merrimanr Just pushed the changes for adding the indexing configs. Should be ready to go now. Can you give it another glance? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #479: METRON-769 Added syslog prog to ASA patterns an...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/479 +1 by inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #472: METRON-700: Add hadoop container to metron-dock...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/472 @merrimanr That's interesting. I did get warnings in the logs about the indexing configs but it seemed like the defaults applied and I was still able to see data indexed into HDFS. Did you have to make any other modifications? Do you want me to add in the indexing configs to zookeeper as part of this PR? It wouldn't be difficult. I like the idea of creating base kafka/zookeeper and hbase container images that could be hosted on Docker Hub. This aligns with how we are handling the other containers and allows us to separate the Metron specific aspects into dedicated Dockerfiles/images. I would be in favor of going the Docker Hub route sooner than later to avoid the additional manual steps. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #471: METRON-755 Update GitHub PR Template
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/471 I'd prefer contributor comments at the top. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #472: METRON-700: Add hadoop container to metron-dock...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/472 ### Description Added a Hadoop/HDFS container to metron-docker. As part of the change, the geo enrichment database is now staged in HDFS to closer mimic a real cluster deployment. ### Testing Successfully ran through testing with bro sensor data based on the test plan previously documented in the [README](https://github.com/apache/incubator-metron/blob/master/metron-docker/README.md#run-sensor-data-end-to-end). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #472: METRON-700: Add hadoop container to metr...
GitHub user kylerichardson opened a pull request: https://github.com/apache/incubator-metron/pull/472 METRON-700: Add hadoop container to metron-docker Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification guildlines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html. ``` cd site-book bin/generate-md.sh mvn site:site ``` ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-700 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/472.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #472 commit a460c083534aae480fdadff148dea0cec0c56d12 Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-03-04T17:12:28Z Initial cut of hadoop image commit 1de2284bd2f4ca200467af15e07911c703cac4ad Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-03-04T20:59:42Z Setup storm access to hdfs container commit ad14085a96ebc7fa0a7d00ebbef006f9af595d4b Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-03-05T00:19:49Z Add license commit ccbba3f92264b7b89786177446ed3102548095ee Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-03-05T00:48:15Z Update README --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/467 @mmiklavc I ran into the same issue with installing pycapa on quick-dev. My solution was to tweak the playbook to rule the pycapa role as part of the sensor-stubs tag. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/467 +1 passes unit and integration tests, ran through @cestella's test script successfully Thanks for your patience and for fixing! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #462: METRON-734 Builds failing because of MaxMind DB...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/462 +1 to proceeded with commit based on @justinleet's Travis results On Thu, Feb 23, 2017 at 7:39 AM, Casey Stella <notificati...@github.com> wrote: > Given the fact that we are without a working build, the failure here is a > known sporadic failure and the Travis queue seems to be very backed up > lately, I move that we wait for 3 hours and then commit this PR on the > basis of Justin's Travis results linked above. > > Objections? > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/incubator-metron/pull/462#issuecomment-281981347>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AJ6R3Y0xIeA0OdvKmpmCvIeWmjYmdckXks5rfX35gaJpZM4MItYb> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #462: METRON-734 Builds failing because of MaxMind DB...
Github user kylerichardson commented on the issue:
https://github.com/apache/incubator-metron/pull/462
gotcha, must just be backed up today
On Wed, Feb 22, 2017 at 3:07 PM, Casey Stella <notificati...@github.com>
wrote:
> @kylerichardson <https://github.com/kylerichardson> nah, travis hasn't
> gotten to it yet. It's in the queue.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/incubator-metron/pull/462#issuecomment-281787260>,
> or mute the thread
>
<https://github.com/notifications/unsubscribe-auth/AJ6R3ZpPyaqnbvq473HkizS5jLsMlAq_ks5rfJV8gaJpZM4MItYb>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #462: METRON-734 Builds failing because of MaxMind DB...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/462 +1 builds and all tests passing. Maybe close and reopen to kick travis? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #441: METRON-646: Add index templates to metron-docke...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/441 Thanks, @merrimanr! I'll get that change pushed out later today. Waiting on #462 to be merged to fix the build errors we're seeing and then I'll rebase and push. On Wed, Feb 22, 2017 at 11:38 AM, merrimanr <notificati...@github.com> wrote: > Works great. I have just one very small request. Can you update the Usage > section of the README (where it lists the UI addresses) to point to the > Elasticsearch head plugin instead of just "http://192.168.99.100:9200/;? > That will keep someone from having to go research how to access that plugin. > > Other than that this gets my +1. Thanks for the contribution. > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/incubator-metron/pull/441#issuecomment-281724636>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AJ6R3fCV7799J8-ypEuDmvuX0LNY_U_kks5rfGRvgaJpZM4L3l4p> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #441: METRON-646: Add index templates to metron-docke...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/441 bump... I read back through the discuss thread from the dev list and we didn't land on any specifics. There seems to be general agreement that metron-docker: - fulfills a current need/desire for targeted development and testing (e.g. parsers) - could be used to make integration testing quicker and easier (requires exploration) How do we want to move this forward? I have this and a follow-on PR ready to go for adding HDFS into metron-docker. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #451: METRON-157: Added CEF Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/451 Not from me. +1, great contribution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #451: METRON-157: Added CEF Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/451#discussion_r100864330
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/DateUtils.java
---
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.utils;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * Various utilities for parsing and extracting dates
+ *
+ */
+public class DateUtils {
+
+ public static List DATE_FORMATS_CEF = new
ArrayList() {
+ {
+ // as per CEF Spec
+ add(new SimpleDateFormat("MMM dd HH:mm:ss.SSS zzz"));
+ add(new SimpleDateFormat("MMM dd HH:mm:ss.SSS"));
--- End diff --
What are the defaults in the case that no year and/or no timezone are
provided? For example, in SysLogUtils we assume any datetime more than 4 days
in the future is actually in the past. Would a parser config setting make sense
to hold these default assumptions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #451: METRON-157: Added CEF Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/451#discussion_r100862828
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/cef/CEFParser.java
---
@@ -0,0 +1,274 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.cef;
+
+import java.nio.charset.Charset;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CEFParser extends BasicParser {
+ private static final long serialVersionUID = 1L;
+
+ protected static final Logger LOG =
LoggerFactory.getLogger(CEFParser.class);
+ private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+ private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+ String syslogHost = "[a-z0-9\\.-_]+";
+
+ StringBuilder sb = new StringBuilder("(?");
+ sb.append(syslogTime);
+ sb.append("|");
+ sb.append(syslogTime5424);
+ sb.append(")?");
+
+ sb.append("(?");
+ sb.append(syslogHost);
+ sb.append(")?");
+
+ sb.append("(?");
+ sb.append(syslogPriority);
+ sb.append(")?");
+
+ sb.append(".*");
+
+ sb.append("CEF:0\\|");
+
+ headerBlock("DeviceVendor", sb);
+ sb.append("\\|");
+ headerBlock("DeviceProduct", sb);
+ sb.append("\\|");
+ headerBlock("DeviceVersion", sb);
+ sb.append("\\|");
+ headerBlock("DeviceEvent", sb);
+ sb.append("\\|");
+ headerBlock("Name", sb);
+ sb.append("\\|");
+ headerBlock("Severity", sb);
+ sb.append("\\|");
+
+ // extension capture:
+ sb.append("(?.*)");
+ String pattern = sb.toString();
+
+ p = Pattern.compile(pattern);
+
+ // key finder for extensions
+ pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+ }
+
+ @SuppressWarnings("unchecked")
+ public List parse(byte[] rawMessage) {
+ List messages = new ArrayList<>();
+
+ String cefString = new String(rawMessage, UTF_8);
+
+ Matcher matcher = p.matcher(cefString);
+
+ while (matcher.find()) {
+ JSONObject obj = new JSONObject();
+ if (matcher.matches()) {
+ LOG.info(String.format("Found %d groups",
matcher.groupCount()));
+ obj.put("DeviceVendor",
matcher.group("DeviceVendor"));
+ obj.put("DeviceProduct",
matcher.group("DeviceProduct"));
+ obj.put("DeviceVersion",
matcher.group("DeviceVersion"));
+ obj.put("DeviceEvent",
matcher.group("DeviceEvent"));
+ obj.put("Name", matcher.group("Name"));
+ obj.put("Severity&quo
[GitHub] incubator-metron issue #451: METRON-157: Added CEF Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/451 @simonellistonball, thanks for picking this one up! I have unassigned the JIRA from myself as I've clearly not had the time to work on it recently. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #441: METRON-646: Add index templates to metron-docke...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/441 Good point, @nickwallen. My hope for metron-docker is as a lightweight alternative to vagrant for development. That said, I would hate to see it become a large overhead for the community. Thanks for starting a dev list discussion. I'll post my thoughts there. -Kyle > On Feb 5, 2017, at 4:09 PM, Nick Allen <notificati...@github.com> wrote: > > Hi @kylerichardson - I don't want to throw cold water on your effort, but I am hesitant to create a third deployment code base for metron-docker (in addition to MPack and Ansible.) Do you think that is what this is or would become? > > Besides just the index templates, we'd have to add and support a lot of other functionality too. Seems like we should have a goal to move towards a single deployment mechanism that works across multiple platforms (Docker, Metal, etc). > > I don't even know if this is feasible, but it may be worth a community discussion. I'll kick something off. > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or mute the thread. > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #441: METRON-646: Add index templates to metro...
GitHub user kylerichardson opened a pull request: https://github.com/apache/incubator-metron/pull/441 METRON-646: Add index templates to metron-docker Enhance metron-docker environment by automatically deploying index templates to elasticsearch as part of the container startup. I've also included a maven clean goal for the metron-docker project for easier build/rebuild. Test plan and rebase with master to follow shortly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-646 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/441.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #441 commit daec3c61c0aade27a649bc6332add32866ccf332 Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-02-02T13:13:25Z Customize elasticsearch docker image Include head plugin and index templates commit 237d99ca9cb83c539f85d1082410b7729b1fc85b Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-02-02T14:37:49Z Add maven-clean-plugin to metron-docker pom commit 0e1f408a1b7b09fc670e58fbe5d276e00f2e75fc Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-02-04T20:32:19Z Correct clean plugin config commit 8b682bc18d6240fdb19c2925e417dad26c68361a Author: kylerichardson <kylerichards...@gmail.com> Date: 2017-02-05T19:06:13Z Correct permissions on wait-for-it.sh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #413: METRON-654 Create RPM Installer for Profiler
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/413 +1 Performed same sanity check testing as @nickwallen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #393: METRON-622: Create a Metron Docker Compose appl...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/393 +1 Awesome contribution. I'm already working on some follow on PRs for this :-). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #393: METRON-622: Create a Metron Docker Compose appl...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/393 @merrimanr Thanks for pointing out the location of the storm logs. I think it would be worth adding that to the README. I have a fix for the kafkazk image running on local docker-engine (using unix socket). It requires an additional ARG in the Dockerfile (and docker-compose.yml) and an extra sed statement when producing the advertised.listeners parameter. docker-compose.yml ``` ... kafkazk: build: context: ./kafkazk args: DOCKER_HOST: $DOCKER_HOST + BROKER_IP_ADDR: $BROKER_IP_ADDR METRON_VERSION: $METRON_VERSION ... ``` kafkazk/Dockerfile ``` ... +ARG DOCKER_HOST +ARG BROKER_IP_ADDR ARG METRON_VERSION ... RUN echo -n 'advertised.listeners=PLAINTEXT://' >> /opt/kafka_2.11-0.10.0.0/config/server.properties +RUN echo $DOCKER_HOST | sed "s/^$/"$BROKER_IP_ADDR":/g" | sed "s/tcp:\\/\\///g" | sed "s/:.*/:9092/g" >> /opt/kafka_2.11-0.10.0.0/config/server.properties ... ``` I would also suggest a note in the documentation. In the Setup section of the README, maybe something along the lines of... If you wish to use a local docker-engine install, please set an environment variable BROKER_IP_ADDR to the IP address of your host machine. This cannot be the loopback address. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #409: METRON-644 RPM builds only work with Docker for...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/409 @justinleet Good catch. Updated the README to remove the note about Docker for Mac. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #393: METRON-622: Create a Metron Docker Compose appl...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/393 I'll created METRON-646 for the elasticsearch image customizations. I already have a start on those changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #393: METRON-622: Create a Metron Docker Compose appl...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/393 I've run this up and successfully tested it using the examples provided in the README. It works as documented on docker-machine/boot2docker. Nice job. One showstopper for me. I can't seem to find the topology logs in the storm container. I checked /var/log/storm and no topology specific logs were ever written. For debugging new parsers, etc. this will be important to have available. I also want to highlight a few nice-to-haves that I would be perfectly happy submitting as separate, follow-on PRs. - Custom the elasticsearch image (1) to have the elasticsearch-head plugin installed as part of the image build, (2) copy the es_templates into the image so they are available, (3) for bonus points, deploy the templates on container start - Load zookeeper config on container start - Improve kafkazk Dockerfile for local docker-engine (Linux); current problem is that DOCKER_HOST in this case defaults to empty string and the default argument in the Dockerfile is never hit - Add an HDFS container to allow for complete testing of the indexing topology --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #393: METRON-622: Create a Metron Docker Compose appl...
Github user kylerichardson commented on the issue:
https://github.com/apache/incubator-metron/pull/393
Thanks for the explanation @merrimanr. I totally agree on the need for the
local IDE to have access to the containerized services. It's an easy enough
fix for me to manipulate the DOCKER_HOST env variable on Fedora and avoid
using docker-machine (although, I do see that as a big win for ease of use
on Mac and Windows).
I haven't had much luck with vagrant myself so am super excited for a
docker alternative to quick dev. As an aside, there are some Ansible
modules for docker (
https://docs.ansible.com/ansible/list_of_cloud_modules.html#docker) that we
could look into for incorporating some of the setup scripts and image
building to make it even more user friendly. Might just be a low priority
nice to have but thought I'd throw it out there.
On Mon, Dec 19, 2016 at 4:32 PM, merrimanr <notificati...@github.com> wrote:
> @kylerichardson <https://github.com/kylerichardson> thank you for
> reviewing it! I updated the documentation to include Docker for Mac or
> Docker for Windows. One of the primary requirements for the way I use this
> is that containers must be accessible from my local environment where my
> IDE is running so services that broadcast their host address are tricky.
> The Kafka advertised listener thing was the single most challenging issue
I
> faced so not surprised you hit it too. The Kafka Dockerfile is wired to
> pull the DOCKER_HOST from an input argument and the compose file is wired
> to pass the local DOCKER_HOST environment variable as the DOCKER_HOST
input
> argument to the Kafka Dockerfile. So you should be able to set your local
> DOCKER_HOST environment variable as such:
>
> $ export DOCKER_HOST="tcp://:2376"
>
> When you run "eval $(docker-machine env metron-machine)" that's pretty
> much what Docker Machine is doing, setting local environment variables to
> match the desired host. Then after you build the environment, the
> advertised.listener property should be set to without you having to
> manually change it.
>
> I only used the virtualbox drive because that's what came out of the box.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/incubator-metron/pull/393#issuecomment-268084862>,
> or mute the thread
>
<https://github.com/notifications/unsubscribe-auth/AJ6R3a4JoffRcq9etOoUAJExjJAeue-aks5rJvf_gaJpZM4LJZHT>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #396: METRON-625: Parser Filters cannot be specified ...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/396 +1 by inspection. Thanks for fixing! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #393: METRON-622: Create a Metron Docker Compose appl...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/393 @merrimanr I'm super excited about this one! Thanks for your work! While kicking the tires on it, I did notice something that isn't really a bug but got me thinking... The README assumes the developer is using a Mac. I personally use Fedora for most of my development and testing of Metron. In my case, I found that without using docker-machine, the kafkazk service did not start up properly due to the use of sed on the DOCKER_HOST variable in the Dockerfile to set the advertised.listeners parameter in kafka server.properties. I was using the docker-engine on the localhost and thought I would avoid the hassle of docker-machine. It was easy enough to solve by manually manipulating the server.properties file in the container, but I was wondering if this could be avoided somehow? I assume that had I installed docker-machine and used the provided script it would have worked as expected even on Fedora (I'm planning to test that case as well), but was curious why the use of the virtualbox driver? Is this a Mac specific requirement? Thanks again for the contribution! I know this will make coding/testing new parsers a lot faster. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #384: METRON-603 Update website to use Apache ...
Github user kylerichardson commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/384#discussion_r90926228 --- Diff: site/_includes/primary-nav-items.html --- @@ -10,6 +10,6 @@ News - https://github.com/apache/incubator-metron/releases; target="new">Download + https://dist.apache.org/repos/dist/release/incubator/metron/; target="new">Download --- End diff -- @cestella Wow, looks like I need to do my homework a little better :). Thanks! I'll close this out as well as the associated JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #384: METRON-603 Update website to use Apache ...
GitHub user kylerichardson opened a pull request: https://github.com/apache/incubator-metron/pull/384 METRON-603 Update website to use Apache release repo Update the DOWNLOAD link on the navigation menu of the website to point to the Apache release repository. Verified site by following instructions on wiki [1]. NOTE: I'm holding off committing my recent PR (#382) to asf-site until this is fully merged to ensure consistency on the website. [1] https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-603 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/384.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #384 commit 63da11df5ba29390aa3352f4dbf50985dc3d3a49 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-12-02T02:50:36Z METRON-603 Update website download links to use apache release repo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #382: METRON-598 Add Kyle Richardson to commit...
GitHub user kylerichardson opened a pull request: https://github.com/apache/incubator-metron/pull/382 METRON-598 Add Kyle Richardson to committers Add Kyle Richardson to list of committers on website Verified site by following instructions on wiki [1]. [1] https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-598 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/382.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #382 commit ffc9c28155226efb699b0d3694fcde9c25a4168a Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-12-01T17:43:46Z Added Kyle Richardson to list of committers on website --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87226233
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java
---
@@ -0,0 +1,175 @@
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements
MessageParser,Serializable{
+
+ protected static final Logger LOG =
LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new
SimpleDateFormat("-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws
IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
--- End diff --
I agree with the desire for consistency; however, I would potentially be
concerned about the user experience of putting scripts (especially ones with
external library dependencies) into zookeeper. We would need to make this as
clean as possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87228757
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java
---
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements
MessageParser,Serializable{
+
+ protected static final Logger LOG =
LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new
SimpleDateFormat("-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws
IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
+ Path path = new Path(streamName);
+ if(fs.exists(path)) {
+ return fs.open(path);
+ } else {
+ return getClass().getResourceAsStream(streamName);
+ }
+ }
+
+ @Override
+ public void init() {
+ engine = new
ScriptEngineManager().getEngineByName(this.language);
+ try{
+ InputStream commonStream =
openInputStream(this.commonScript);
+ if (commonStream == null) {
+ throw new RuntimeException(
+ "Unable to initialize "+this.language+" Parser:
Unable to load " + this.commonScript + " from either classpath or HDFS");
+ }
+
+ engine.eval(new InputStreamReader(commonStream));
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Loading parser-specific functions from: " +
this.scriptPath);
+ }
+
+ InputStream patterInputStream =
openInputStream(this.scriptPath);
+ if (patterInputStream == null) {
+ throw new RuntimeException("Script parser unable to
initialize "+this.language+" parser: Unable to load " + this.scriptPath
+ + " from either classpath or HDFS");
+
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87221966
--- Diff:
metron-platform/metron-parsers/src/main/resources/scripts/python/common ---
@@ -0,0 +1,20 @@
+ # Licensed to the Apache Software Foundation (ASF) under one
+ # or more contributor license agreements. See the NOTICE file
+ # distributed with this work for additional information
+ # regarding copyright ownership. The ASF licenses this file
+ # to you under the Apache License, Version 2.0 (the
+ # "License"); you may not use this file except in compliance
+ # with the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+def MetronMessage(name):
+ message={}
+ message["source"]="userlog"
--- End diff --
Can we update this to set the source to the passed in name variable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Github user kylerichardson commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/338#discussion_r87221525 --- Diff: metron-platform/metron-parsers/pom.xml.orig --- @@ -0,0 +1,370 @@ + + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; --- End diff -- This appears to be a local backup of the POM file. Can it be removed in favor of the git history? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87221814
--- Diff:
metron-platform/metron-parsers/src/main/resources/scripts/groovy/common ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def MetronMessage(){
+ def message=[:];
+ message["source"]="userlog";
--- End diff --
For the other languages you have passed in a name variable and used it for
the source, can we do that here to be consistent?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 A big thank you to @ottobackwards for helping to troubleshoot the CI build fails. This should be good to go now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Ok, need some helping figuring out why the CI build keeps failing... I get several of these at the end of the log: ``` Running org.apache.metron.parsers.integration.JSONMapIntegrationTest 2016-11-01 15:54:52 FATAL KafkaServer:116 - [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown kafka.common.KafkaException: Socket server failed to bind to localhost:6667: Address already in use. ``` and prior to that I see: ``` Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.64 sec <<< FAILURE! - in org.apache.metron.parsers.integration.YafIntegrationTest test(org.apache.metron.parsers.integration.YafIntegrationTest) Time elapsed: 8.637 sec <<< ERROR! java.lang.NoClassDefFoundError: org/slf4j/event/LoggingEvent ``` This occurred for both of the CI builds since I rebased to the latest master. Any ideas? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
GitHub user kylerichardson reopened a pull request: https://github.com/apache/incubator-metron/pull/276 METRON-363 Fix Cisco ASA Parser I've rewritten the ASA parser which can be extended, as needed, to new ASA message types by editing the bundled asa patterns file and the static map used for grok patterns in the code. I've also tried to make it easier to deploy the asa topology by including zookeeper config files and creating the kafka topic during metron install. Sample data is also included for integration testing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-363 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/276.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #276 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson closed the pull request at: https://github.com/apache/incubator-metron/pull/276 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Rebased against master to incorporate the global junit version change. Should be good to go now pending Travis. Thanks again to everyone for all of the suggestions, feedback, and testing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r85651135
--- Diff: metron-platform/metron-parsers/src/main/resources/patterns/asa ---
@@ -107,7 +108,7 @@ COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer}
%{QS:agent}
LOGLEVEL
([A|a]lert|ALERT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
#== Cisco ASA ==
-CISCO_TAGGED_SYSLOG ^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}(
%{SYSLOGHOST:sysloghost})? ?:? %%{CISCOTAG:ciscotag}:
--- End diff --
The ASA patterns build off of several of the more generic patterns
referenced earlier in the file; however, I should be able to reduce it down to
just the ones being used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #325: METRON-512 up default junit to 4.12
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/325 Great idea! +1 (non-binding) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Any other feedback or suggestions for me? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Whew, got the CI build to finally pass. All integration and unit tests are passing. I've also re-testing in the single node vm environment I described above. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
GitHub user kylerichardson reopened a pull request: https://github.com/apache/incubator-metron/pull/276 METRON-363 Fix Cisco ASA Parser I've rewritten the ASA parser which can be extended, as needed, to new ASA message types by editing the bundled asa patterns file and the static map used for grok patterns in the code. I've also tried to make it easier to deploy the asa topology by including zookeeper config files and creating the kafka topic during metron install. Sample data is also included for integration testing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-363 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/276.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #276 commit 5be7c60448f73fcc72c81451a67ef1e40fd29793 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-08-16T01:12:42Z Initial rewrite of Cisco ASA parser Summary of changes: - Complete rewrite of ASA parser including new test suite - ZK configurations for ease of topology deployment (parser and enrichment) - Add field constant for original_string in metron-common - Minor changes to ASA patterns file for (1) Syslog severity/facility capture (2) Interface capture on CISCOFW106006_106007_106010 - Updates to various POMs to allow easier validation of logging during unit testing (1) Exclusions for slf4j-log4j12 on various dependencies for metron-parsers and metron-integration-test (2) Explicit dependency on slf4j-api for metron-parsers (3) Test dependency on slf4j-simple for metron-parsers commit c87e6edaf0e308be9f417e07016508f87067ae0c Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-20T02:33:09Z METRON-363 Reworked parser to handle nulls and field validation Includes the following: - Static map for ASA message patterns (vs pattern discovery) - Minor changes to ASA patterns file - Broke out common syslog parsing elements - Broke out reusable field validations commit a8c4903dd0bcac18e15c98aca7264dce1c455bee Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-27T00:30:16Z METRON-363 Add integration test and sample data Includes the following: - Extend BasicParser - Handle both types of syslog timestamps (with and without year) - Include integration test and supporting sample data commit 011d389bdf43f1790384dbcd13ec7da148c53ef2 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-27T00:40:51Z METRON-363 Add license and kafka topic commit 04a936d75cf782254105993b2804912b4659257a Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-28T00:29:21Z METRON-363 Adjust log level commit abd7fb92fe4c38530e10141d0aba6bd07a335ae8 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-08T01:11:22Z METRON-363 Enhance logging, remove unused code commit a885ecc762a8d5296d7c7ebfe7600c910ce3478b Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-11T17:40:25Z METRON-363 Refactored and enhanced based on feedback Changes include: (1) New/additional unit tests (2) Reworked Syslog Timestamp (no year) logic (3) Enhanced error checking and logging (introduced new ParseException) commit fb6ed83eab8704607dc75c37982b0f98b819047d Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-12T13:54:54Z METRON-363 Default to UTC in zookeeper config commit d7d327a3b03584fd3d03d4f6468d54c15786bda7 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-13T02:10:14Z METRON-363 Update tests commit 4e3cba6682eaf3130325d4c27bf32240ad7a0a92 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-18T00:33:34Z METRON-363 Refactor to add Clock dependency for testing commit db8686615533470e8a3273ee268f2eb0efb4999c Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-18T01:15:29Z METRON-363 Add tests for back dating RFC3164 timestamps --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson closed the pull request at: https://github.com/apache/incubator-metron/pull/276 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r83768223
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/SyslogUtils.java
---
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.utils;
+
+import org.apache.metron.parsers.ParseException;
+
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.TemporalAccessor;
+import java.util.regex.Pattern;
+
+import static java.time.temporal.ChronoField.*;
+
+public class SyslogUtils {
+
+public static long parseTimestampToEpochMillis(String logTimestamp,
ZoneId timeZone) throws ParseException {
+// RFC3164 (standard syslog timestamp; no year)
+// MMM ppd HH:mm:ss
+// Oct 9 2015 13:42:11
+if
(Pattern.matches("[A-Z][a-z]{2}(?:(?:\\s{2}\\d)|(?:\\s\\d{2}))\\s\\d{2}:\\d{2}:\\d{2}",
logTimestamp)) {
+DateTimeFormatter inputFormat =
DateTimeFormatter.ofPattern("MMM ppd HH:mm:ss").withZone(timeZone);
+
+TemporalAccessor inputDate = inputFormat.parse(logTimestamp);
+int inputMonth = inputDate.get(MONTH_OF_YEAR);
+int inputDay = inputDate.get(DAY_OF_MONTH);
+int inputHour = inputDate.get(HOUR_OF_DAY);
+int inputMinute = inputDate.get(MINUTE_OF_HOUR);
+int inputSecond = inputDate.get(SECOND_OF_MINUTE);
+
+ZonedDateTime currentDate = ZonedDateTime.now(timeZone);
+int normalizedYear = currentDate.getYear();
+
+/**
+ * Since no year is provided, one must be derived.
+ * During the month of January (first 31 days of the year),
assume logs coming in from
+ * November (11) and December (12) are from the previous
year.
+ */
+if (currentDate.getDayOfYear() <= 31 && inputMonth >= 11)
+normalizedYear--;
+ZonedDateTime normalizedTimestamp =
ZonedDateTime.of(normalizedYear, inputMonth, inputDay, inputHour, inputMinute,
inputSecond, 0, timeZone);
+return normalizedTimestamp.toInstant().toEpochMilli();
+}
+
+// CISCO timestamp (standard syslog + year)
+// MMM dd HH:mm:ss
+// Oct 09 2015 13:42:11
+else if
(Pattern.matches("[A-Z][a-z]{2}\\s\\d{2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2}",
logTimestamp))
+return convertToEpochMillis(logTimestamp,
DateTimeFormatter.ofPattern("MMM dd HH:mm:ss").withZone(timeZone));
+
+// RFC5424 (ISO timestamp)
+// 2015-10-09T13:42:11.52Z or 2015-10-09T13:42:11.52-04:00
+else if
(Pattern.matches("\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(?:\\.\\d+)?(?:Z|[+-]\\d{2}:\\d{2})",
logTimestamp))
+return convertToEpochMillis(logTimestamp,
DateTimeFormatter.ISO_OFFSET_DATE_TIME);
+
+else
+throw new ParseException(String.format("Unsupported date
format: '%s'", logTimestamp));
--- End diff --
My thought here was that there may be some situations where we want to
handle a parsing error without blowing up and sending the message to the error
queue. It was a bit of "future proofing" on my part I suppose.
For consistency, would it be better to revert to using a RuntimeException?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r83520638
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/utils/SyslogUtilsTest.java
---
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.utils;
+
+import org.apache.metron.parsers.ParseException;
+import org.junit.Test;
+
+import java.time.ZoneOffset;
+
+import static org.junit.Assert.*;
+
+public class SyslogUtilsTest {
--- End diff --
Agreed. There currently isn't test coverage for that logic.
I was trying to avoid having to add a dependency on a Clock object but it
may be the only way to throughly test this code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r83520042
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/SyslogUtils.java
---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.utils;
+
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+
+public class SyslogUtils {
+
+public static long convertToEpochMillis(String logTimestamp, String
logTimeFormat) {
+ZonedDateTime timestamp = ZonedDateTime.parse(logTimestamp,
DateTimeFormatter.ofPattern(logTimeFormat).withZone(ZoneOffset.UTC));
+return timestamp.toEpochSecond() * 1000;
+}
+
+public static long parseTimestampToEpochMillis(String logTimestamp) {
+if (logTimestamp.length() < 20) {
+ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
+int year = now.getYear();
+if (now.getDayOfYear() == 1 &&
!now.getMonth().toString().substring(0,3).equals(logTimestamp.substring(0,3).toUpperCase()))
+year--;
--- End diff --
I like the idea of checking how far the date in the current year would be
in the future and basing the back date decision on that. Let me work on that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #308: Metron-498 Grok patterns are now read from zook...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/308 The ASA patterns file `/incubator-metron/metron-platform/metron-parsers/src/main/resources/patterns/asa` is being used (or soon will be) by PR #276. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #307: METRON-499 Check for Metron Jar Fails During Qu...
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/307 +1 (non-binding) Thanks for fixing. Tested in quick dev. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 @nickwallen Apologies, I should have been more specific. I tested using the same steps provided earlier in the PR. That said, my "single node vm" testing is not done with vagrant. Currently I'm not able to successfully use the quick dev environment based on my setup (e.g. Windows). I'm working to remedy that. For "single node vm" testing, I actually run two vms, one Fedora host which I do development on and use to run the ansible deployment and a second Centos 6 (base install from snapshot) host which I deploy Metron onto. For testing this PR, I deployed Metron without the sensors to by Centos 6 vm for testing and ran through the steps provided above. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Thanks. Looks like re-opening did the trick. I've done my best to incorporate everyone's feedback into this version. Re-tested in single node vm successfully. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
GitHub user kylerichardson reopened a pull request: https://github.com/apache/incubator-metron/pull/276 METRON-363 Fix Cisco ASA Parser I've rewritten the ASA parser which can be extended, as needed, to new ASA message types by editing the bundled asa patterns file and the static map used for grok patterns in the code. I've also tried to make it easier to deploy the asa topology by including zookeeper config files and creating the kafka topic during metron install. Sample data is also included for integration testing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-363 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/276.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #276 commit 5be7c60448f73fcc72c81451a67ef1e40fd29793 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-08-16T01:12:42Z Initial rewrite of Cisco ASA parser Summary of changes: - Complete rewrite of ASA parser including new test suite - ZK configurations for ease of topology deployment (parser and enrichment) - Add field constant for original_string in metron-common - Minor changes to ASA patterns file for (1) Syslog severity/facility capture (2) Interface capture on CISCOFW106006_106007_106010 - Updates to various POMs to allow easier validation of logging during unit testing (1) Exclusions for slf4j-log4j12 on various dependencies for metron-parsers and metron-integration-test (2) Explicit dependency on slf4j-api for metron-parsers (3) Test dependency on slf4j-simple for metron-parsers commit c87e6edaf0e308be9f417e07016508f87067ae0c Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-20T02:33:09Z METRON-363 Reworked parser to handle nulls and field validation Includes the following: - Static map for ASA message patterns (vs pattern discovery) - Minor changes to ASA patterns file - Broke out common syslog parsing elements - Broke out reusable field validations commit a8c4903dd0bcac18e15c98aca7264dce1c455bee Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-27T00:30:16Z METRON-363 Add integration test and sample data Includes the following: - Extend BasicParser - Handle both types of syslog timestamps (with and without year) - Include integration test and supporting sample data commit 011d389bdf43f1790384dbcd13ec7da148c53ef2 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-27T00:40:51Z METRON-363 Add license and kafka topic commit 04a936d75cf782254105993b2804912b4659257a Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-28T00:29:21Z METRON-363 Adjust log level commit abd7fb92fe4c38530e10141d0aba6bd07a335ae8 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-08T01:11:22Z METRON-363 Enhance logging, remove unused code commit a885ecc762a8d5296d7c7ebfe7600c910ce3478b Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-11T17:40:25Z METRON-363 Refactored and enhanced based on feedback Changes include: (1) New/additional unit tests (2) Reworked Syslog Timestamp (no year) logic (3) Enhanced error checking and logging (introduced new ParseException) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson closed the pull request at: https://github.com/apache/incubator-metron/pull/276 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Not entirely sure why the CI build failed. The error was: ``` testExample1(org.apache.metron.profiler.integration.ProfilerIntegrationTest) Time elapsed: 35.546 sec <<< FAILURE! java.lang.AssertionError: expected:<1950.0> but was:<390.0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:553) at org.junit.Assert.assertEquals(Assert.java:683) at org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample1(ProfilerIntegrationTest.java:140) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) ``` Slightly earlier in the log: ``` 106738 [Curator-Framework-0] ERROR o.a.c.ConnectionState - Connection timed out for connection string (127.0.0.1:51857) and timeout (15000) / elapsed (18872) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [metron-common-0.2.1BETA.jar:?] at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [metron-common-0.2.1BETA.jar:?] at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) [metron-common-0.2.1BETA.jar:?] at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:806) [metron-common-0.2.1BETA.jar:?] at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) [metron-common-0.2.1BETA.jar:?] at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) [metron-common-0.2.1BETA.jar:?] at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) [metron-common-0.2.1BETA.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_31] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31] ``` I'm thi
[GitHub] incubator-metron pull request #302: METRON-492 Run metron_common build check...
GitHub user kylerichardson opened a pull request: https://github.com/apache/incubator-metron/pull/302 METRON-492 Run metron_common build check as local_action When testing various deployment options, noticed the logic for the metron_common role didn't work as expected. Updated to check for metron jar locally (where the playbook was initiated) and fail when it doesn't exist. Tested successfully in single node vm. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-492 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/302.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #302 commit bb7da9f667d9e87615c1623a543a74d70cdac53a Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-10-08T01:21:51Z METRON-492 Run metron_common build check as local_action --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r82490277
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/SyslogUtils.java
---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.utils;
+
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+
+public class SyslogUtils {
+
+public static long convertToEpochMillis(String logTimestamp, String
logTimeFormat) {
+ZonedDateTime timestamp = ZonedDateTime.parse(logTimestamp,
DateTimeFormatter.ofPattern(logTimeFormat).withZone(ZoneOffset.UTC));
+return timestamp.toEpochSecond() * 1000;
+}
+
+public static long parseTimestampToEpochMillis(String logTimestamp) {
+if (logTimestamp.length() < 20) {
+ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
+int year = now.getYear();
+if (now.getDayOfYear() == 1 &&
!now.getMonth().toString().substring(0,3).equals(logTimestamp.substring(0,3).toUpperCase()))
+year--;
--- End diff --
Sure. I see how this is not entirely obvious. I'm trying to solve an edge
case here where a message comes in for parsing without a year in the timestamp
on January 1st but the message was actually generated on the device on December
31st. I'll add in some comments for clarity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r82489921
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/SyslogUtils.java
---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.utils;
+
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+
+public class SyslogUtils {
+
+public static long convertToEpochMillis(String logTimestamp, String
logTimeFormat) {
+ZonedDateTime timestamp = ZonedDateTime.parse(logTimestamp,
DateTimeFormatter.ofPattern(logTimeFormat).withZone(ZoneOffset.UTC));
+return timestamp.toEpochSecond() * 1000;
+}
+
+public static long parseTimestampToEpochMillis(String logTimestamp) {
+if (logTimestamp.length() < 20) {
+ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
--- End diff --
Of course you're right, the timestamp will not always be in UTC. ASA logs
consumed via syslog (either raw off the wire or through another syslog server)
will generally follow the syslog standard.
There are a number of possibilities to explore here. If we assume that we
will be collecting the raw syslog from the ASAs off the wire, the timestamp
will not include the timezone/offset. This code assumes the device is logging
in UTC, which, to your point, is probably a bad assumption. I made this
assumption because it seems to me we would want all of the timestamps indexed
to be in the same timezone and the easiest way to accomplish that would be to
normalize all of the telemetry data to UTC.
Question for the team. How are other parsers handling timezone? Are they
passing through the device timezone?
The way I'm thinking of solving this is by adding a configuration option to
the parser to specify the device timezone. (This would require that all ASAs
put through the parser we configured to the same timezone though.) I would then
convert the timestamp to UTC prior to writing it into the metron normalized
JSON message.
Any feedback or other ideas on solving this one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r81454727
--- Diff: metron-deployment/roles/metron_kafka_topics/defaults/main.yml ---
@@ -21,6 +21,7 @@ topics_to_create:
- { topic: "bro", num_partitions: 1, replication_factor: 1,
retention_gb: 10 }
- { topic: "yaf", num_partitions: 1, replication_factor: 1,
retention_gb: 10 }
- { topic: "snort", num_partitions: 1, replication_factor: 1,
retention_gb: 10 }
+ - { topic: "asa", num_partitions: 1, replication_factor: 1,
retention_gb: 10 }
--- End diff --
That makes sense. I was thinking about building out the monit scripts, etc
to make this as easy as possible for the user to deploy out-of-the-box, but
that's a future PR. Is that something that would be valuable to folks? Either
way, I can remove this from the current PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r81454659
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG =
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator =
InetAddressValidator.getInstance();
+
+private static final Map<String, String> patternMap =
ImmutableMap.<String, String>builder()
+.put("ASA-2-106001", "CISCOFW106001")
+ .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+ .put("ASA-3-106014", "CISCOFW106014")
+ .put("ASA-6-106015", "CISCOFW106015")
+ .put("ASA-1-106021", "CISCOFW106021")
+ .put("ASA-4-106023", "CISCOFW106023")
+ .put("ASA-5-106100", "CISCOFW106100")
+ .put("ASA-6-110002", "CISCOFW110002")
+ .put("ASA-6-302010", "CISCOFW302010")
+ .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302020", "CISCOFW302020_302021")
+ .put("ASA-6-302021", "CISCOFW302020_302021")
+ .put("ASA-6-305011", "CISCOFW305011")
+ .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+ .put("ASA-4-313005", "CISCOFW313005")
+ .put("ASA-4-402117", "CISCOFW402117")
+ .put("ASA-4-402119", "CISCOFW402119")
+ .put("ASA-4-419001", "CISCOFW419001")
+ .put("ASA-4-419002", "CISCOFW419002")
+ .put("ASA-4-54", "CISCOFW54")
+ .put("ASA-6-602303", "CISCOFW602303_602304")
+ .put("ASA-6-602304", "CISCOFW602303_602304")
+ .put("ASA-7-710001",
"CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710002",
"CISCOFW710001_710002_710003_710005_710006")
+
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r81454500
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG =
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator =
InetAddressValidator.getInstance();
+
+private static final Map<String, String> patternMap =
ImmutableMap.<String, String>builder()
+.put("ASA-2-106001", "CISCOFW106001")
+ .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+ .put("ASA-3-106014", "CISCOFW106014")
+ .put("ASA-6-106015", "CISCOFW106015")
+ .put("ASA-1-106021", "CISCOFW106021")
+ .put("ASA-4-106023", "CISCOFW106023")
+ .put("ASA-5-106100", "CISCOFW106100")
+ .put("ASA-6-110002", "CISCOFW110002")
+ .put("ASA-6-302010", "CISCOFW302010")
+ .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302020", "CISCOFW302020_302021")
+ .put("ASA-6-302021", "CISCOFW302020_302021")
+ .put("ASA-6-305011", "CISCOFW305011")
+ .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+ .put("ASA-4-313005", "CISCOFW313005")
+ .put("ASA-4-402117", "CISCOFW402117")
+ .put("ASA-4-402119", "CISCOFW402119")
+ .put("ASA-4-419001", "CISCOFW419001")
+ .put("ASA-4-419002", "CISCOFW419002")
+ .put("ASA-4-54", "CISCOFW54")
+ .put("ASA-6-602303", "CISCOFW602303_602304")
+ .put("ASA-6-602304", "CISCOFW602303_602304")
+ .put("ASA-7-710001",
"CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710002",
"CISCOFW710001_710002_710003_710005_710006")
+
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r81454486
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG =
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator =
InetAddressValidator.getInstance();
+
+private static final Map<String, String> patternMap =
ImmutableMap.<String, String>builder()
+.put("ASA-2-106001", "CISCOFW106001")
+ .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+ .put("ASA-3-106014", "CISCOFW106014")
+ .put("ASA-6-106015", "CISCOFW106015")
+ .put("ASA-1-106021", "CISCOFW106021")
+ .put("ASA-4-106023", "CISCOFW106023")
+ .put("ASA-5-106100", "CISCOFW106100")
+ .put("ASA-6-110002", "CISCOFW110002")
+ .put("ASA-6-302010", "CISCOFW302010")
+ .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302020", "CISCOFW302020_302021")
+ .put("ASA-6-302021", "CISCOFW302020_302021")
+ .put("ASA-6-305011", "CISCOFW305011")
+ .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+ .put("ASA-4-313005", "CISCOFW313005")
+ .put("ASA-4-402117", "CISCOFW402117")
+ .put("ASA-4-402119", "CISCOFW402119")
+ .put("ASA-4-419001", "CISCOFW419001")
+ .put("ASA-4-419002", "CISCOFW419002")
+ .put("ASA-4-54", "CISCOFW54")
+ .put("ASA-6-602303", "CISCOFW602303_602304")
+ .put("ASA-6-602304", "CISCOFW602303_602304")
+ .put("ASA-7-710001",
"CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710002",
"CISCOFW710001_710002_710003_710005_710006")
+
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/276#discussion_r81453779
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG =
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator =
InetAddressValidator.getInstance();
+
+private static final Map<String, String> patternMap =
ImmutableMap.<String, String>builder()
+.put("ASA-2-106001", "CISCOFW106001")
+ .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+ .put("ASA-3-106014", "CISCOFW106014")
+ .put("ASA-6-106015", "CISCOFW106015")
+ .put("ASA-1-106021", "CISCOFW106021")
+ .put("ASA-4-106023", "CISCOFW106023")
+ .put("ASA-5-106100", "CISCOFW106100")
+ .put("ASA-6-110002", "CISCOFW110002")
+ .put("ASA-6-302010", "CISCOFW302010")
+ .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302020", "CISCOFW302020_302021")
+ .put("ASA-6-302021", "CISCOFW302020_302021")
+ .put("ASA-6-305011", "CISCOFW305011")
+ .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+ .put("ASA-4-313005", "CISCOFW313005")
+ .put("ASA-4-402117", "CISCOFW402117")
+ .put("ASA-4-402119", "CISCOFW402119")
+ .put("ASA-4-419001", "CISCOFW419001")
+ .put("ASA-4-419002", "CISCOFW419002")
+ .put("ASA-4-54", "CISCOFW54")
+ .put("ASA-6-602303", "CISCOFW602303_602304")
+ .put("ASA-6-602304", "CISCOFW602303_602304")
+ .put("ASA-7-710001",
"CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710002",
"CISCOFW710001_710002_710003_710005_710006")
+
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/276#discussion_r81453224 --- Diff: metron-platform/metron-integration-test/src/main/sample/data/asa/raw/asa_raw --- @@ -0,0 +1,128 @@ +<167>Jan 5 08:52:35 10.22.8.216 %ASA-7-609001: Built local-host inside:10.22.8.205 +<166>Jan 5 08:52:35 10.22.8.216 %ASA-6-302021: Teardown ICMP connection for faddr 10.22.8.74/0(LOCAL\user.name) gaddr 10.22.8.205/0 laddr 10.22.8.205/0 +<167>Jan 5 08:52:35 10.22.8.216 %ASA-7-609002: Teardown local-host inside:10.22.8.205 duration 0:00:00 +<142>Jan 5 08:52:35 10.22.8.201 %ASA-6-302014: Teardown TCP connection 488167725 for Outside_VPN:147.111.72.16/26436 to DMZ-Inside:10.22.8.53/443 duration 0:00:00 bytes 9687 TCP FINs +<166>Jan 5 08:52:35 10.22.8.216 %ASA-6-302014: Teardown TCP connection 212805593 for outside:10.22.8.223/59614(LOCAL\user.name) to inside:10.22.8.78/8102 duration 0:00:07 bytes 3433 TCP FINs (user.name) +<174>Jan 5 14:52:35 10.22.8.212 %ASA-6-302013: Built inbound TCP connection 76245503 for outside:10.22.8.233/54209 (10.22.8.233/54209) to inside:198.111.72.238/443 (198.111.72.238/443) (user.name) +<166>Jan 5 08:52:35 10.22.8.216 %ASA-6-302013: Built inbound TCP connection 212806031 for outside:10.22.8.17/58633 (10.22.8.17/58633)(LOCAL\user.name) to inside:10.22.8.12/389 (10.22.8.12/389) (user.name) +<142>Jan 5 08:52:35 10.22.8.201 %ASA-6-302014: Teardown TCP connection 488168292 for DMZ-Inside:10.22.8.51/51231 to Inside-Trunk:10.22.8.174/40004 duration 0:00:00 bytes 2103 TCP FINs --- End diff -- I took the existing test data found in .../sample/data/SampleInput/AsaOutput and added to it data from some of my test devices. The data I added has been scrubbed/anonymized. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 @nickwallen @cestella Thanks very much for the feedback! Much appreciated. I'll get started on these changes and respond to your questions as soon as I can. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 **Testing** It occurs to me I haven't outlined how to test or how I tested this code (apologies, this is my first PR). All my testing was performed on a single node vm (no sensors). This should mimic the quick-dev environment (unfortunately, I haven't had much luck with vagrant due to my primary OS being Windows). Test Steps 1) Deploy single node vm using metron_full_install ansible playbook (I can provide my host and group_vars if anyone is interested) 2) Stop unused parsers `monit stop pcap-parser` `monit stop yaf-parser` `monit stop bro-parser` `monit stop snort-parser` 3) Install elasticsearch head `/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head` 4) Start the asa parser topology `start_parser_topology.sh -k node1:6667 -z node1:2181 -s asa` 5) Use the console producer to load raw asa events into kafka `/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic asa < asa_raw.txt` For test data I used the sample data provided for integration testing and raw data collected from one of my devices. 6) Verify events in elasticsearch Using the head plugin, I could browse the asa_index_* index and see the enriched events Future enhancements 1) I could not add the asa* indexes to kibana. I believe an elasticsearch template is required. I'll be working on that as a future PR. 2) Minor bug in one of the ansible roles (metron_common). The logic to verify the jars exist is done remotely and should be done locally. I'll submit a separate JIRA and PR for this fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Currently my branch doesn't have build_utils. Going to rebase and see if that fixes the CI build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 I've tested in a slimmed down single node vm (no sensors) but not in vagrant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser
GitHub user kylerichardson opened a pull request: https://github.com/apache/incubator-metron/pull/276 METRON-363 Fix Cisco ASA Parser I've rewritten the ASA parser which can be extended, as needed, to new ASA message types by editing the bundled asa patterns file and the static map used for grok patterns in the code. I've also tried to make it easier to deploy the asa topology by including zookeeper config files and creating the kafka topic during metron install. Sample data is also included for integration testing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kylerichardson/incubator-metron METRON-363 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/276.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #276 commit 1519be62a361d29f6eaa15fb9f641873d87675e0 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-08-16T01:12:42Z Initial rewrite of Cisco ASA parser Summary of changes: - Complete rewrite of ASA parser including new test suite - ZK configurations for ease of topology deployment (parser and enrichment) - Add field constant for original_string in metron-common - Minor changes to ASA patterns file for (1) Syslog severity/facility capture (2) Interface capture on CISCOFW106006_106007_106010 - Updates to various POMs to allow easier validation of logging during unit testing (1) Exclusions for slf4j-log4j12 on various dependencies for metron-parsers and metron-integration-test (2) Explicit dependency on slf4j-api for metron-parsers (3) Test dependency on slf4j-simple for metron-parsers commit a1284084ecfde20c16f338972e9b1f0dc7d7ae78 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-20T02:33:09Z METRON-363 Reworked parser to handle nulls and field validation Includes the following: - Static map for ASA message patterns (vs pattern discovery) - Minor changes to ASA patterns file - Broke out common syslog parsing elements - Broke out reusable field validations commit 5e6468120534e04cacbe4d21910eb797971dd816 Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-27T00:30:16Z METRON-363 Add integration test and sample data Includes the following: - Extend BasicParser - Handle both types of syslog timestamps (with and without year) - Include integration test and supporting sample data commit aeca74aa35c0c45ec74a96a7a976bf8557b246cd Author: kylerichardson <kylerichards...@gmail.com> Date: 2016-09-27T00:40:51Z METRON-363 Add license and kafka topic --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
