Re: Cloudtrail use case

2017-10-05 Thread Justin Leet
I know we've had discussions about migrating stuff into docs before. It might be worth resurrecting a more use case focused version of that, instead of starting on the wiki. I assume the end goal is availability in the site-book, so even if it's not in a perfect place, I'd rather the effort be sp

Re: Cloudtrail use case

2017-10-05 Thread Nick Allen
We don't really have a location in the source code for use cases like this right now. But I think it is so important that we get use cases like this published somewhere. For now, you could add this to the Wiki. Then later on we can figure out how to handle that. On Thu, Oct 5, 2017 at 6:49 PM,

Re: Cloudtrail use case

2017-10-05 Thread Laurens Vets
Yes, that's what I meant :) I sent my mail too soon. On 2017-10-05 15:48, Nick Allen wrote: If you mean that you would be willing to do the work, then yes absolutely! I think that would be great. :) On Thu, Oct 5, 2017 at 6:45 PM, Laurens Vets wrote: Hi, Would anyone be interested in addin

Re: Cloudtrail use case

2017-10-05 Thread Laurens Vets
On 2017-10-05 15:45, Laurens Vets wrote: Hi, Would anyone be interested in adding a full AWS Cloudtrail use case to the Metron documentation? I would roughly consist of: - Apache NiFi configuration to retrieve Cloudtrail logs from S3 and send it to Metron via Kafka. - Complete Metron sensor conf

Re: Cloudtrail use case

2017-10-05 Thread Nick Allen
If you mean that you would be willing to do the work, then yes absolutely! I think that would be great. :) On Thu, Oct 5, 2017 at 6:45 PM, Laurens Vets wrote: > Hi, > > Would anyone be interested in adding a full AWS Cloudtrail use case to the > Metron documentation? I would roughly consist of:

Cloudtrail use case

2017-10-05 Thread Laurens Vets
Hi, Would anyone be interested in adding a full AWS Cloudtrail use case to the Metron documentation? I would roughly consist of: - Apache NiFi configuration to retrieve Cloudtrail logs from S3 and send it to Metron via Kafka. - Complete Metron sensor configuration (enrichment, alerting, etc...)

[GitHub] metron pull request #780: METRON-1220: Create documentation around alert nes...

2017-10-05 Thread justinleet
Github user justinleet closed the pull request at: https://github.com/apache/metron/pull/780 ---

[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...

2017-10-05 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/780 Kicking Travis. ---

[GitHub] metron pull request #780: METRON-1220: Create documentation around alert nes...

2017-10-05 Thread justinleet
GitHub user justinleet reopened a pull request: https://github.com/apache/metron/pull/780 METRON-1220: Create documentation around alert nested field ## Contributor Comments Adding some docs around the requirement of an "alert" field on new sensors. There's an open discu

[GitHub] metron pull request #780: METRON-1220: Create documentation around alert nes...

2017-10-05 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/780#discussion_r143064505 --- Diff: Upgrading.md --- @@ -2,6 +2,48 @@ This document constitutes a per-version listing of changes of configuration which are non-backwards c

[GitHub] metron pull request #780: METRON-1220: Create documentation around alert nes...

2017-10-05 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/780#discussion_r143063049 --- Diff: Upgrading.md --- @@ -2,6 +2,48 @@ This document constitutes a per-version listing of changes of configuration which are non-backwards c

[GitHub] metron pull request #780: METRON-1220: Create documentation around alert nes...

2017-10-05 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/780#discussion_r143064409 --- Diff: README.md --- @@ -118,3 +118,9 @@ Some useful utilities that cross all of these parts of the architecture: * [Model as a Service](metron-an

[GitHub] metron pull request #780: METRON-1220: Create documentation around alert nes...

2017-10-05 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/780#discussion_r143062420 --- Diff: README.md --- @@ -118,3 +118,9 @@ Some useful utilities that cross all of these parts of the architecture: * [Model as a Service](metron-an

[GitHub] metron issue #791: METRON-1235: Document the properties pulled from the glob...

2017-10-05 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/791 I'm good with where it's at now. The color is good to have. It's really nice to have this well documented. I'm +1, pending Travis. Although I'll be mildly impressed if you break Travis w

[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...

2017-10-05 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/791#discussion_r143059336 --- Diff: metron-platform/metron-enrichment/README.md --- @@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper. There are two types of confi

[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...

2017-10-05 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/791#discussion_r143058876 --- Diff: metron-platform/metron-enrichment/README.md --- @@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper. There are two types of confi

[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...

2017-10-05 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/791#discussion_r143058984 --- Diff: metron-platform/metron-enrichment/README.md --- @@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper. There are two types of confi

[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...

2017-10-05 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/791#discussion_r143058611 --- Diff: metron-platform/metron-enrichment/README.md --- @@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper. There are two types of con

[GitHub] metron issue #791: METRON-1235: Document the properties pulled from the glob...

2017-10-05 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/metron/pull/791 Thanks @justinleet I added and provided some color; let me know what you think! ---

[GitHub] metron issue #791: METRON-1235: Document the properties pulled from the glob...

2017-10-05 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/791 One more config I can think of that should be added. `geo.hdfs.file` is also in the global config. It's in Ambari as `geoip_url`. It looks like a small blurb should probably be added

[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...

2017-10-05 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/791 METRON-1235: Document the properties pulled from the global configuration ## Contributor Comments Our global config is very flexible and multiple parts of the system pull configuration from it.

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-05 Thread Michael Miklavcic
Justin, thanks for the feedback! I'm inclined to agree with you about using the high level client. It's a bummer that we still need to do jar shading, but I think that's a reasonable short term sacrifice considering the other benefits. And they're angling towards slowly removing the ES core dep ove

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-05 Thread Justin Leet
Do we intend on (or have interest in) supporting ES across major version for a given version of Metron? I'm not convinced it's worth the work of using the low level client. This really only seems useful for ES clusters that are being used outside Metron and need to be on a different ES major vers

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-05 Thread Michael Miklavcic
I think it might help the discussion to share my impressions of looking over the new API recommendations from ES. I've summarized some info provided by ES back in December 2016 regarding the reasons for switching to a new client model. [1] *Summary points:* Pre-5.x had Java API - binary exchange

[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...

2017-10-05 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/780 @nickwallen I figured out what the problem was. I'd replaced dashes with spaces and lowercased it, but left the period in `2.x`. After removing the period, links worked. @cestella Added

[GitHub] metron issue #767: METRON-1196 Increment master version number to 0.4.2 for ...

2017-10-05 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/767 I don't see any commits since my comments, so that still needs to be addressed I think ---

[GitHub] metron issue #767: METRON-1196 Increment master version number to 0.4.2 for ...

2017-10-05 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/767 @mattf-horton @ottobackwards Is there any more work for this? ---

[GitHub] metron pull request #790: METRON-1234 fix for WARNING 'dependencies.dependen...

2017-10-05 Thread dbist
GitHub user dbist opened a pull request: https://github.com/apache/metron/pull/790 METRON-1234 fix for WARNING 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-yarn-api:jar ## Contributor Comments getting WARNING 'dependen

[GitHub] metron pull request #789: METRON-1233: Remove description of Global configur...

2017-10-05 Thread DimDroll
GitHub user DimDroll opened a pull request: https://github.com/apache/metron/pull/789 METRON-1233: Remove description of Global configuration from Enrichment doc It confuses reader. Description of Global configuration is available ahead of the enrichment and listed here: https:

Need suggestion on how to configure HCP Big Data for Development and Testing

2017-10-05 Thread Ashikin Abdullah
Hi, can anyone help me to suggest appropriate deployment for Hortonworks Cybersecurity Package within this environment. We have Dell PowerEdge VRTX with 4 nodes, M630 x 4 and HDD 6006 x 25 (shared storage). Therefore, how to manage all this resources to properly configured HCP? Thanks in advance.

[GitHub] metron pull request #784: METRON-1229: Management UI type is part of the dec...

2017-10-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/784 ---

[GitHub] metron pull request #783: METRON-1228: Configuration Management PUSH immedia...

2017-10-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/783 ---

[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...

2017-10-05 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/metron/pull/780 Three things IMO: 1. Could we add a link to the documentation you have from the front page README.md under the heading "Notes on Adding a New Sensor" 2. Could we also add a link to the docume

[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...

2017-10-05 Thread merrimanr
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/768 Thanks @iraghumitra the bulk actions are working now. Just need to address the e2e test suggestions and I think this will be good to go. ---

[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...

2017-10-05 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/780 > @justinleet: Sidenote, does anybody know how to actually link to a head in a different doc? Anything inside the same doc works fine, but across docs would be better. The following exam

[GitHub] metron issue #784: METRON-1229: Management UI type is part of the declaratio...

2017-10-05 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/784 +1 by inspection. ---

[GitHub] metron pull request #787: METRON-1232: Alert status changes are not reflecte...

2017-10-05 Thread iraghumitra
Github user iraghumitra closed the pull request at: https://github.com/apache/metron/pull/787 ---

[GitHub] metron pull request #787: METRON-1232: Alert status changes are not reflecte...

2017-10-05 Thread iraghumitra
GitHub user iraghumitra reopened a pull request: https://github.com/apache/metron/pull/787 METRON-1232: Alert status changes are not reflected in list view ## Contributor Comments The Jira has all the description of the issue. In addition, the e2e test for removing search ch

[GitHub] metron pull request #788: METRON-1223: Support for adding comments to alerts

2017-10-05 Thread iraghumitra
GitHub user iraghumitra opened a pull request: https://github.com/apache/metron/pull/788 METRON-1223: Support for adding comments to alerts ## Contributor Comments This PR adds support for adding arbitrary text notes to an alert. This can be achieved by using METRON-1051.The com

[GitHub] metron pull request #787: METRON-1232: Alert status changes are not reflecte...

2017-10-05 Thread iraghumitra
GitHub user iraghumitra opened a pull request: https://github.com/apache/metron/pull/787 METRON-1232: Alert status changes are not reflected in list view ## Contributor Comments The Jira has all the description of the issue. In addition, the e2e test for removing search chip

Configuring HCP Big Data for Development

2017-10-05 Thread Ashikin Abdullah
Hi, can anyone suggest appropriate deployment for Hortonworks Cybersecurity Package within this environment. We have Dell PowerEdge VRTX with 4 nodes and 4 HDD M630 (shared storage) x 25. Therefore, how to manage all this resources to properly configured HCP? Hope you guys can help me. Thanks in