[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/22 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/22#issuecomment-184695462 +1 from me as well. Great job --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user merrimanr commented on the pull request: https://github.com/apache/incubator-metron/pull/22#issuecomment-184694405 Looks good to me. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/22#issuecomment-183747542 Of course, I have attached a design doc to the [JIRA](https://issues.apache.org/jira/browse/METRON-35) . This is really a single feature as leaving out any part will leave the whole feature nonfunctional. It can seem a bit complex, but it fits within the overall architecture built for the enrichments. I detailed this and how it fits within the overall architecture in the design doc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/22#issuecomment-183613907 I want to point out a couple of other things this PR provides that aren't strictly associated with the feature above, but are general cleanup tasks: * Removed lingering hbase-site.xml which have a bad habit of finding their way onto the classpath and confusing HBase in integration tests * The split of integration tests (defined as a test that ends with "IntegrationTest") into the integration-test maven lifecycle phase * Using the shade maven plugin to relocate our guava dependency so that we can use a more recent version of Guava than 12 (which is the most recent that HBase will allow due to google's habit of aggressive removal of deprecated code). This comes up when running HBase in minicluster mode as well as in situations when running bolts which have to package the hbase-client. * General cleanup of the build to use the version properties instead of hard coding different versions of common components (e.g. hbase-client, storm-core, etc.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user ddutta commented on the pull request: https://github.com/apache/incubator-metron/pull/22#issuecomment-183620620 Is there a design doc for this large checkin? Then it will be easier to review. Or maybe if we split this into more manageable chunks since this is a new feature beyond infra/devops. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/22 METRON-35 Implement threat intelligence message enrichment Create the infrastructure to * Bulk ingest threat intelligence feeds from CSV and Stix data sources into HBase * Enrich messages who have fields which match the threat intelligence data in HBase * Create the infrastructure to remove unused threat intelligence data * Augment the Packet capture topology to incorporate a malicious IP threat intel tagger The tagging infrastructure much meet the following criteria: * They are downstream of the enrichments * The threat intelligence bolts execute in parallel with a similar architecture as the enrichments (i.e. split and join). You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron Threat_Intel_Feeds Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/22.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22 commit 5cf5409472d9557f7725ad14a8bcca3663c364aa Author: cstellaDate: 2016-02-03T21:30:13Z Added ThreatIntelBulkLoader commit 77105eb645dd357d512aa1d52e9d28e3641003f3 Author: cstella Date: 2016-02-04T16:00:16Z updating threat intel loader. commit 4fcaebcdc38cbf56df89137883c92725e80a88e6 Author: cstella Date: 2016-02-04T16:40:44Z Adding shell script to execute the threat intel feeds. commit 0d390fc0d86af24976649828a8853aec10ab9b0c Author: cstella Date: 2016-02-03T21:30:13Z Added ThreatIntelBulkLoader commit 8256e22f679896c18df8cbfc2dd0bc67a7718b32 Author: cstella Date: 2016-02-04T16:00:16Z updating threat intel loader. commit e5aeb99fb29da3d00eabe53252d88a3345d5e34a Author: cstella Date: 2016-02-04T16:40:44Z Adding shell script to execute the threat intel feeds. commit cfcd709bbbef3e24a5c75b41d07beae9934fe843 Author: cstella Date: 2016-02-04T16:52:37Z Merge branch 'Threat_Intel_Feeds' of github.com:cestella/incubator-metron into Threat_Intel_Feeds commit 5ca646a94f91ec6745abda8fe27a585f1a15904e Author: cstella Date: 2016-02-05T22:31:11Z Moving around some components to common, refactoring some dependencies to allow hbase integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used keys. commit b7721d375c79e0380d0799ad895faa8b44546e76 Author: cstella Date: 2016-02-05T22:31:22Z Moving around some components to common, refactoring some dependencies to allow hbase integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used keys. commit 6e026600e41e766a4af0e8c0caa0dc2c882d0bd9 Author: cstella Date: 2016-02-08T18:37:15Z Adding uni ttests for the bulk load/delete jobs. commit 32b198cd241a296f0f1c90cbcdbdb2bcaa3e9dd6 Author: cstella Date: 2016-02-08T19:17:40Z Merge branch 'master' into Threat_Intel_Feeds commit 5c0283c09217f29863ec75c49fd32b420d4e970c Author: cstella Date: 2016-02-09T17:52:02Z Updating to add new extractor, Stix extractor commit 110ed867a0ba7ed638fab7eeb99ffe5e03dcb17e Author: cstella Date: 2016-02-09T18:05:51Z Added test for stix extractor. commit 3cc67d58c08ef8b7cbe2d360512bdfa968e2888e Author: cstella Date: 2016-02-09T20:01:49Z Changed the bloom filter persistent access tracker to use HBase instead of HDFS commit d49496dcb34208fdf997c01a50379ef297a9f3e4 Author: cstella Date: 2016-02-09T20:21:58Z Updating poms to allow more memory. commit c46b4c5b2cd816e50bda050fa51c0e6b28fcf3c2 Author: cstella Date: 2016-02-09T23:15:51Z we really need to stop shipping hbase-site.xmls around. commit 920223ab2c39e834fddea18353997111d8693488 Author: cstella Date: 2016-02-10T20:18:49Z Made HBase Bolt more adaptable. commit 580257e27b917bd029eecab49a3b6b8aac375fde Author: cstella Date: 2016-02-10T20:27:00Z Merge branch 'master' into Threat_Intel_Feeds commit 560877b6c29903fd80b23cb846176dca801336dc Author: cstella Date: 2016-02-10T20:50:51Z HBaseBolt was so wrong. commit 5221eb9d9f4bef6cf580efbb6a3a6848cbeda45c Author: cstella Date: 2016-02-11T14:46:13Z Adding a ThreatIntelAdapter to the