[GitHub] incubator-metron pull request #302: METRON-492 Run metron_common build check...

[kylerichardson]

GitHub user kylerichardson opened a pull request:

https://github.com/apache/incubator-metron/pull/302

METRON-492 Run metron_common build check as local_action

When testing various deployment options, noticed the logic for the 
metron_common role didn't work as expected. Updated to check for metron jar 
locally (where the playbook was initiated) and fail when it doesn't exist.

Tested successfully in single node vm.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kylerichardson/incubator-metron METRON-492

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/302.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #302


commit bb7da9f667d9e87615c1623a543a74d70cdac53a
Author: kylerichardson 
Date:   2016-10-08T01:21:51Z

METRON-492 Run metron_common build check as local_action




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[kylerichardson]

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r82490277
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/SyslogUtils.java
 ---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.utils;
+
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+
+public class SyslogUtils {
+
+public static long convertToEpochMillis(String logTimestamp, String 
logTimeFormat) {
+ZonedDateTime timestamp = ZonedDateTime.parse(logTimestamp, 
DateTimeFormatter.ofPattern(logTimeFormat).withZone(ZoneOffset.UTC));
+return timestamp.toEpochSecond() * 1000;
+}
+
+public static long parseTimestampToEpochMillis(String logTimestamp) {
+if (logTimestamp.length() < 20) {
+ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
+int year = now.getYear();
+if (now.getDayOfYear() == 1 && 
!now.getMonth().toString().substring(0,3).equals(logTimestamp.substring(0,3).toUpperCase()))
+year--;
--- End diff --

Sure. I see how this is not entirely obvious. I'm trying to solve an edge 
case here where a message comes in for parsing without a year in the timestamp 
on January 1st but the message was actually generated on the device on December 
31st. I'll add in some comments for clarity.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[kylerichardson]

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r82489921
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/SyslogUtils.java
 ---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.utils;
+
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+
+public class SyslogUtils {
+
+public static long convertToEpochMillis(String logTimestamp, String 
logTimeFormat) {
+ZonedDateTime timestamp = ZonedDateTime.parse(logTimestamp, 
DateTimeFormatter.ofPattern(logTimeFormat).withZone(ZoneOffset.UTC));
+return timestamp.toEpochSecond() * 1000;
+}
+
+public static long parseTimestampToEpochMillis(String logTimestamp) {
+if (logTimestamp.length() < 20) {
+ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
--- End diff --

Of course you're right, the timestamp will not always be in UTC. ASA logs 
consumed via syslog (either raw off the wire or through another syslog server) 
will generally follow the syslog standard.

There are a number of possibilities to explore here. If we assume that we 
will be collecting the raw syslog from the ASAs off the wire, the timestamp 
will not include the timezone/offset. This code assumes the device is logging 
in UTC, which, to your point, is probably a bad assumption. I made this 
assumption because it seems to me we would want all of the timestamps indexed 
to be in the same timezone and the easiest way to accomplish that would be to 
normalize all of the telemetry data to UTC.

Question for the team. How are other parsers handling timezone? Are they 
passing through the device timezone?

The way I'm thinking of solving this is by adding a configuration option to 
the parser to specify the device timezone. (This would require that all ASAs 
put through the parser we configured to the same timezone though.) I would then 
convert the timestamp to UTC prior to writing it into the metron normalized 
JSON message.

Any feedback or other ideas on solving this one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #289: METRON-461: Install Metron Data Manageme...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/289


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #294: METRON-487: Correct the license in the S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/294


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #292: METRON-171 add .class to .gitignore

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/292


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: community demo tomorrow

[P. Taylor Goetz]

Hi James,

My apologies for missing/forgetting about those threads. It looks like you guys 
are doing the right thing.

-Taylor

> On Oct 6, 2016, at 10:11 PM, James Sirota  wrote:
> 
> Hi Taylor,
> 
> We setup a recurring demo meeting to run twice a month last month per this 
> thread:
> http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201609.mbox/raw/%3C1139681474481923%40web32j.yandex.ru%3E/
> 
> and then announced the first one in the series 2 weeks ago here:
> http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201609.mbox/raw/%3C508251474574231%40web9m.yandex.ru%3E/
> 
> We then had the meeting and posted a summary and a video here:
> http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201609.mbox/raw/%3C182111474664208%40web22m.yandex.ru%3E
> 
> That meeting was held over zoom at 11am PST on Sept.23.  Per that thread the 
> next demo meeting should be held tomorrow at 11AM PST.  However, we still 
> have a lot of pull requests outstanding so I was asking if it makes sense to 
> hold the meeting next week instead so we can get all the pull requests in.
> 
> Per your point I will send reminders about the recurring meeting 72 hours 
> prior to the meeting with the agenda and will solicit feedback if anyone 
> would like to change the time or the tool for the virtual meeting.  If there 
> are no objections we will skip the meeting on Friday and will do it mid week 
> next week once we get most of the pull requests merged.
> 
> Thanks,
> James
> 
> 06.10.2016, 17:09, "P. Taylor Goetz" :
>> I'd love to see a demo, but with my mentor hat on I would say wait. Try not 
>> to schedule things too soon so people in other time zones have a chance to 
>> participate. This is what's behind the 72 hr. wait period for votes at the 
>> ASF.
>> 
>> I haven't checked, but I think the Metron community is largely based in 
>> North America. As the community grows, this will undoubtedly change to 
>> include all parts (and time zones) of the world.
>> 
>> Solicit times and dates from the community, then collectively choose one. 
>> Also decide on a medium (ghangouts, zoom, webex, etc.). Afterwords, 
>> summarize what was discussed and send it to the mailing list so those who 
>> couldn't attend know what happened.
>> 
>> -Taylor
>> 
>>>  On Oct 6, 2016, at 7:41 PM, James Sirota  wrote:
>>> 
>>>  Does anyone want to do a demo tomorrow? Or should we push it off till next 
>>> week so that we have a chance to review and commit a few more pull 
>>> requests? Looks like we have 19 open right now. I think that's the most 
>>> we've ever gotten in a 2-week time frame. Great job, community!
>>> 
>>>  If we do want to demo tomorrow, can you respond to this thread with what 
>>> you want to demo?
>>> 
>>>  ---
>>>  Thank you,
>>> 
>>>  James Sirota
>>>  PPMC- Apache Metron (Incubating)
>>>  jsirota AT apache DOT org
> 
> ---
> Thank you,
> 
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org



signature.asc
Description: Message signed with OpenPGP using GPGMail

[GitHub] incubator-metron pull request #301: METRON-490 Stellar Validation of Require...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/301

METRON-490 Stellar Validation of Required Parameters

### [METRON-490](https://issues.apache.org/jira/browse/METRON-490)

Currently, each Stellar function handles validation of required function 
parameters in its own way.  In some cases, we have functions that throw an 
IndexOutOfBoundsException, in other cases the function will produce its own 
error message and exception.

There needs to be a standard mechanism to validate required function 
parameters.  This mechanism should handle missing parameters gracefully and 
provide an error message that makes sense to a user.  The handling should be 
accessible across all functions.

### Changes

* Added an additional Stellar annotation called `requiredParams`.  
* Added a method to `BaseStellarFunction.validate` to validate the 
`requiredParams`.
* Use of the annotation field and validation mechanism is optional.
* Modified the Stellar STATS_* functions to use this functionality.
* If an incorrect type is specified, an IllegalArgumentException is thrown 
with a message like 
   ```
   SOME_FUNCTION: unexpected parameter [2]: expected class 
java.lang.Integer, actual class java.lang.String`
   ```
* If a required parameter is missing, an IllegalArgumentException is thrown 
with a message like:
```
SOME_FUNCTION: parameter missing [2]: expected class java.lang.Integer
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-490

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/301.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #301






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #298: METRON-432: Fix pcap field resolver to return o...

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/incubator-metron/pull/298
  
this is a fairly minor change, but makes sense.  +1 builds, passes tests 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[james-sirota]

Github user james-sirota commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r82446542
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
 ---
@@ -43,6 +43,7 @@
 ,DST_PORT("ip_dst_port")
 ,PROTOCOL("protocol")
 ,TIMESTAMP("timestamp")
+,ORIGINAL("original_string")
--- End diff --

I agree. I think we've needed something like this for a while. it will help 
standardize our parsing. great job noticing this.  I am definitely +1 on 
refactoring grok to look like this 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #292: METRON-171 add .class to .gitignore

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/incubator-metron/pull/292
  
+1 by inspection. thanks for catching this 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #295: METRON-371: Changing logging level to INFO when...

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/incubator-metron/pull/295
  
+1 by inspection 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #297: METRON-488: Snort should use a proper CSV imple...

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/incubator-metron/pull/297
  
Does the same apply to other parsers as well? or just snort?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #300: METRON-489: RemoveSubdomains Stellar Fun...

[james-sirota]

Github user james-sirota commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/300#discussion_r82441035
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/NetworkFunctions.java
 ---
@@ -78,13 +78,16 @@ public Object apply(List list) {
 
 @Override
 public Object apply(List objects) {
+  if(objects.isEmpty()) {
+return null;
+  }
   Object dnObj = objects.get(0);
   InternetDomainName idn = toDomainName(dnObj);
   if(idn != null) {
 String dn = dnObj.toString();
 String tld = idn.publicSuffix().toString();
-String suffix = Iterables.getFirst(Splitter.on(tld).split(dn), 
null);
-if(suffix != null)
+String suffix = dn.substring(0, dn.length() - tld.length());
+if(suffix != null )
--- End diff --

www.subdomain.com.com is not a valid TLD.  for a list of valid domains see 
here: https://publicsuffix.org/list/effective_tld_names.dat

If the system sees this kind of domain this should immediately be flagged 
as alert and triaged with a very high score


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #299: METRON-425 Stellar transformation fails ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/299


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #299: METRON-425 Stellar transformation fails to hand...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/299
  
+1, I'm satisfied updating the docs and the unit test addresses the 
original concern seen.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #278: Metron 451 SerDeUtils - java.lang.ClassNotFound...

[danieljue]

Github user danieljue commented on the issue:

https://github.com/apache/incubator-metron/pull/278
  
Roger that, will devise a test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #275: METRON-459 Bad file location for org.apa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/275


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #275: METRON-459 Bad file location for org.apache.met...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/275
  
This one is good to go.  Thank you for the contribution! 👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Ambari Mpack

[Nick Allen]

I've been working with the Ambari Mpack on a live cluster over the past
couple days.  Realizing that we're in the very early iterations and need
some time to polish up the rough edges, I am very impressed with the work
that has been done so far.  The user experience is light years beyond our
Ansible deployment mechanism.  This is going to be really solid.

Many thanks to everyone who has been involved in that work; Justin, Mike,
David, others?



-- 
Nick Allen 

[GitHub] incubator-metron issue #299: METRON-425 Stellar transformation fails to hand...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/299
  
To validate Casey's point, I just ran a quickly modified version of your 
unit test with things single quotes, and things seem to line up quite nicely at 
that point on special chars and not.  I think we just update the README with 
the list of keywords and set a unit test up with the single quotes.  At that 
point, I'm good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #300: METRON-489: RemoveSubdomains Stellar Fun...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/300#discussion_r82378741
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/NetworkFunctions.java
 ---
@@ -78,13 +78,16 @@ public Object apply(List list) {
 
 @Override
 public Object apply(List objects) {
+  if(objects.isEmpty()) {
+return null;
+  }
   Object dnObj = objects.get(0);
   InternetDomainName idn = toDomainName(dnObj);
   if(idn != null) {
 String dn = dnObj.toString();
 String tld = idn.publicSuffix().toString();
-String suffix = Iterables.getFirst(Splitter.on(tld).split(dn), 
null);
-if(suffix != null)
+String suffix = dn.substring(0, dn.length() - tld.length());
+if(suffix != null )
--- End diff --

Could you just drop this if-else?  Substring never returns null, so it can 
be dropped entirely.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Opinionated Data Flows

[Nick Allen]

Whether it is explicit or implicit, I think that would be one of the major
benefits of having the expressiveness of a DSL.  I can choose to have some
enrichments run in parallel (the split/join that you are referring to) or
have some enrichment runs serially.

Having enrichments run serially is not something you can easily do with
Metron today.  You cannot use the output of one enrichment as the input to
another.

As a simple example, I have a blacklist of countries for which my
organization should not be doing business.  I need to use the IP to find
the location and then use the location to match against a blacklist.  I
need these enrichments to run serially.

source("netflow")
  -> parser("Netflow")
  -> exists("ip_src_addr")
  -> src_country = geo["ip_src_addr"].country
  -> is_alert = blacklist["src_country"]
  ...




On Thu, Oct 6, 2016 at 6:25 PM, Matt Foley  wrote:

> Would splitting and joining be implicit or explicit, for multi-path
> topologies?
> 
> From: zeo...@gmail.com 
> Sent: Thursday, October 06, 2016 11:03 AM
> To: dev@metron.incubator.apache.org
> Subject: Re: [DISCUSS] Opinionated Data Flows
>
> It should also be smart enough to handle an order like:
>
> source("bro")
>   -> parser("BasicBroParser")
>   -> exists("ip_src_addr")
>   -> geo_ip_src = geo["ip_src_addr"]
>   -> application = assets["ip_src_addr"].application
>   -> owner = assets["ip_src_addr"].owner
>   -> exists("ip_dst_addr")
>   -> geo_ip_dst = geo["ip_dst_addr"]
>   -> elasticsearch("bro-index")
>
> Without duplicate hits of the topologies.
>
> Jon
>
> On Thu, Oct 6, 2016 at 1:55 PM Nick Allen  wrote:
>
> > Here is quick example with some hypothetical syntax.  Whatever that
> syntax
> > might be, it would be very simple, easy to understand, and leverage
> > high-level concepts specific to Metron.
> >
> > This flow consumes Bro data, ensures there are valid source/destination
> > IPs, performs geo-enrichment, asset enrichment and finally persists the
> > data in Elasticsearch.
> >
> >
> > source("bro")
> >   -> parser("BasicBroParser")
> >   -> exists("ip_src_addr")
> >   -> exists("ip_dst_addr")
> >   -> geo_ip_src = geo["ip_src_addr"]
> >   -> geo_ip_dst = geo["ip_dst_addr"]
> >   -> application = assets["ip_src_addr"].application
> >   -> owner = assets["ip_src_addr"].owner
> >   -> elasticsearch("bro-index")
> >
> >
> >
> >
> > On Thu, Oct 6, 2016 at 12:58 PM, Nick Allen  wrote:
> >
> > > Chasing this bad idea down even further leads me to something even
> > > crazier.
> > >
> > > Stellar 1.0 can only operate within a single topology and in most cases
> > > only on a single message.  Stellar 2.0 could be the mechanism that
> allows
> > > users to define their own data flows and what "useful bits of Metron
> > > functionality" get plugged-in.
> > >
> > > Once, you have a DSL that allows users to define what they want Metron
> to
> > > do, then the underlying implementation mechanism (which is currently
> > Storm)
> > > can also be swapped-out.  If we have an even faster Storm
> implementation,
> > > then we swap in the Storm NG engine.  Maybe we want Metron to also run
> in
> > > Flink, then we just swap-in a Flink engine.
> > >
> > >
> > >
> > >
> > > On Thu, Oct 6, 2016 at 12:52 PM, Nick Allen 
> wrote:
> > >
> > >> I totally "bird dogged the previous thread" as Casey likes to call it.
> > :)
> > >>  I am extracting this thought into a separate thread before I start
> > >> throwing out even more, crazier ideas.
> > >>
> > >> In general, Metron is very opinionated about data flows right now.  We
> > >>> have Parser topologies that feed an Enrichment topology, which then
> > feeds
> > >>> an Indexing topology.  We have useful bits of functionality (think
> > Stellar
> > >>> transforms, Geo enrichment, etc) that are closely coupled with these
> > >>> topologies (aka data flows).
> > >>>
> > >>
> > >>
> > >>> When a user wants to parse heterogenous data from a single topic,
> > that's
> > >>> not easy.  When a user wants enriched output to land in unique topics
> > by
> > >>> sensor type, well, that's also not easy.When a user wanted to
> skip
> > >>> enrichment of data sources, we actually re-architected the data flow
> > to add
> > >>> the Indexing topology.
> > >>>
> > >>
> > >>
> > >>> In an ideal world, a user should be responsible for defining the data
> > >>> flow, not Metron.  Metron should provide the "useful bits of
> > functionality"
> > >>> that a user can "plugin" wherever they like.  Metron itself should
> not
> > care
> > >>> how the data is moving or what step in the process it is at.
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Nick Allen 
> > >>
> > >
> > >
> > >
> > > --
> > > Nick Allen 
> > >
> >
> >
> >
> > --
> > Nick Allen 
> >
> --
>
> Jon
>



-- 
Nick Allen 

[GitHub] incubator-metron pull request #297: METRON-488: Snort should use a proper CS...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/297

METRON-488: Snort should use a proper CSV implementation

Right now if you have a custom snort rule (e.g. alert tcp any any -> any 
any (msg:'snort alert message having a ,(comma) to check csv parsing'; 
sid:999158; ) ) the snort parser will fail to parse because it's splitting on 
the comma naively.
It should use the existing CSV parsing infrastructure that we have and that 
is used in the CSVParser.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron snort_delim_bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/297.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #297


commit f0a57334d0d80e298e5ea25f1b114ae0d6db4b11
Author: cstella 
Date:   2016-10-06T18:14:46Z

Updating the snort parser to use the CSVExtractor infrastructure, which is 
a thin layer on top of OpenCSV

commit 90d863034c52fe1d2860ec9f8ff63a6fd3267887
Author: cstella 
Date:   2016-10-07T06:55:19Z

Whoops, forgot to call init.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #300: METRON-489: RemoveSubdomains Stellar Fun...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/300

METRON-489: RemoveSubdomains Stellar Function behaves incorrectly for some 
domains

com.com throws an exception
www.subdomain.com.com returns subdomain.com
Unsure if other standard weirdness with TLDs get handled like this (e.g. 
net.net, co.uk.co.uk)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
remove_subdomains_edge_cases

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/300.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #300


commit 17c386a56aa9963e5c3947e53f20292518747891
Author: cstella 
Date:   2016-10-07T08:02:48Z

METRON-489: RemoveSubdomains Stellar Function behaves incorrectly for some 
domains




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---