[GitHub] [metron] anandsubbu commented on issue #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings
anandsubbu commented on issue #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings URL: https://github.com/apache/metron/pull/1500#issuecomment-527413556 +1 * Spun up full dev and did some sanity testing on the UIs * Ran `npm audit`on both metron-alerts and metron-config directories. Both reported 1 low severity vulnerability (as already mentioned in the PR description) Thanks @sardell for the fix! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (METRON-2243) [UI] Update npm dependencies to resolve audit warnings
[ https://issues.apache.org/jira/browse/METRON-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-2243: Assignee: Shane Ardell > [UI] Update npm dependencies to resolve audit warnings > -- > > Key: METRON-2243 > URL: https://issues.apache.org/jira/browse/METRON-2243 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > After running `npm audit` in both UIs, we have a few dependencies and quite a > few nested dependencies needing version updates. This should be as simple as > running `npm audit fix`, although there may be a few dependencies needing > major version updates that require manual updates. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [metron] luozhenwei commented on issue #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings
luozhenwei commented on issue #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings URL: https://github.com/apache/metron/pull/1500#issuecomment-527438789 Hello, may I ask you a question? For testing purposes, I compiled metron according to the source code on github. Reference links are as follows: https://github.com/apache/metron/tree/master/metron-deployment/packaging/docker/ansible-docker. The last command to execute is MVN clean package-DskipTests. The result is successful, showing build success. Excuse me, how many jar packages are generated by this operation? How to start metron in docker? Or can we say something about the next operation? I can't find duying in the community anymore. Thank you very much! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] MohanDV commented on issue #1483: METRON-2217 Migrate current HBase client from HTableInterface to Table
MohanDV commented on issue #1483: METRON-2217 Migrate current HBase client from HTableInterface to Table URL: https://github.com/apache/metron/pull/1483#issuecomment-527448402 Thanks @mmiklavc ! it's +1 with verification from my side . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] sardell commented on issue #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings
sardell commented on issue #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings URL: https://github.com/apache/metron/pull/1500#issuecomment-527448548 @luozhenwei It sounds like your questions are unrelated to this PR, so they have a low likelihood of being answered here. Did you search the [Metron user mailing list](https://lists.apache.org/list.html?u...@metron.apache.org) to see if someone has asked a similar question? You should start by searching for answers there and asking if you cannot find the answers you're looking for. Though we aren't perfect, we try to actively answer as much as we can there. We also have an active Slack channel. You can join by emailing u...@metron.apache.org and asking for an invite to the channel. Hope this helps! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (METRON-2245) Allow full dev to pull all downloads via a web proxy
Dale Richardson created METRON-2245: --- Summary: Allow full dev to pull all downloads via a web proxy Key: METRON-2245 URL: https://issues.apache.org/jira/browse/METRON-2245 Project: Metron Issue Type: Improvement Reporter: Dale Richardson To speed deployment up (and catch those downloads that vagrant cachier seem to miss), allow web proxy configurations to be passed through to the full dev build processes, so all downloads can be proxied. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [metron] asfgit closed pull request #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings
asfgit closed pull request #1500: METRON-2243: [UI] Update npm dependencies to resolve audit warnings URL: https://github.com/apache/metron/pull/1500 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1489: METRON-2221: Notebook import fails through Zeppelin REST API
mmiklavc commented on issue #1489: METRON-2221: Notebook import fails through Zeppelin REST API URL: https://github.com/apache/metron/pull/1489#issuecomment-527497172 This looks good @anandsubbu - can you check off "Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?" along with any other tasks you performed? Anything that's n/a please just replace the checkbox [] with "n/a". +1 by inspection pending the checklist updates. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1261: METRON-1860 new developer option for ansible in docker to deploy to vagrant
mmiklavc commented on issue #1261: METRON-1860 new developer option for ansible in docker to deploy to vagrant URL: https://github.com/apache/metron/pull/1261#issuecomment-527511923 @luozhenwei - it's generally best to ask these questions on the user or dev list, but here's the list of jars we depend on in our full dev (e.g. https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) env: ``` [root@node1 ~]# ls -1 /usr/metron/0.7.2/lib metron-common-0.7.2.jar metron-data-management-0.7.2.jar metron-elasticsearch-storm-0.7.2-uber.jar metron-enrichment-common-0.7.2-uber.jar metron-enrichment-storm-0.7.2-uber.jar metron-maas-service-0.7.2-uber.jar metron-management-0.7.2.jar metron-parsers-0.7.2-uber.jar metron-parsers-common-0.7.2-uber.jar metron-parsing-storm-0.7.2-uber.jar metron-pcap-backend-0.7.2.jar metron-performance-0.7.2.jar metron-profiler-repl-0.7.2.jar metron-profiler-spark-0.7.2.jar metron-profiler-storm-0.7.2-uber.jar metron-rest-0.7.2.jar metron-solr-storm-0.7.2-uber.jar ``` 17 jars. The Docker deploy you linked to hasn't been maintained/updated in quite some time, so I'm not entirely sure what its current state is. @merrimanr may have more detail on this. If you're just trying to explore Metron, I would run up full dev on centos6 via the instructions in the link I provided above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] anandsubbu commented on issue #1489: METRON-2221: Notebook import fails through Zeppelin REST API
anandsubbu commented on issue #1489: METRON-2221: Notebook import fails through Zeppelin REST API URL: https://github.com/apache/metron/pull/1489#issuecomment-527538202 Thanks @mmiklavc , done. I will wait for any comments from @sardell on the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc edited a comment on issue #1483: METRON-2217 Migrate current HBase client from HTableInterface to Table
mmiklavc edited a comment on issue #1483: METRON-2217 Migrate current HBase
client from HTableInterface to Table
URL: https://github.com/apache/metron/pull/1483#issuecomment-525102320
## Test Plan
### Enrichments
This will cover enrichments, threat intel, and the bulk loading utilities
that write data to HBase
Test basic enrichment
Spin up full dev
Optional - free up resources. We're going to be spinning up some additional
topologies. The resources in full dev are limited, so you'll probably want to
stop non-critical topologies in order to have enough Storm slots.
```
for parser in bro__snort__yaf profiler pcap batch_indexing; do storm kill
$parser; don
```
Follow the following [updated] blog series steps here to get some data into
Metron using Squid along with an enrichment
1.
https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
2.
https://cwiki.apache.org/confluence/display/METRON/2016/04/28/Metron+Tutorial+-+Fundamentals+Part+2%3A+Creating+a+New+Enrichment
Test threat intel
1.
https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence
Test multi-threading
For the final step, we'll deviate from the blog a bit so we can test that
the thread pool doesn't cause any deadlocking/threading issues on the new HBase
connection approach. Taken from
https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment.
Follow the steps in the blog tutorial for setting up the user streaming
enrichment, but instead of modifying/using bro as suggested at the end, follow
the below instructions.
Let's load the original whois list from step 1 as a threatintel for added
fun. This way we can run multiple enrichments and also have it trigger threat
intel from the same messages. Create a file `blocklist2.csv` with the following
contents:
```
[root@node1: ~]
# cat blocklist2.csv
aliexpress.com,squidblacklist.org
pravda.ru,squidblacklist.org
google.com,squidblacklist.org
brightsideofthesun.com,squidblacklist.org
microsoftstore.com,squidblacklist.org
autonews.com,squidblacklist.org
facebook.com,squidblacklist.org
ebay.com,squidblacklist.org
recruit.jp,squidblacklist.org
lada.ru,squidblacklist.org
aliexpress.com,squidblacklist.org
```
Load the threat intel into HBase
`${METRON_HOME}/bin/flatfile_loader.sh -i blocklist2.csv -t threatintel -c t
-e threatintel_extractor_config.json`
Clear the squid logs
```
rm /var/log/squid/access.log
touch /var/log/squid/access.log
chown squid:squid /var/log/squid/access.log
service squid restart
```
Re-run new squid client commands similar to step 1. Rather than a fraction
of the records matching on domain for the whois enrichment, we'll have them all
match for this test.
```
squidclient
"https://www.google.com/maps/place/Waterford,+WI/@42.7639877,-88.2867248,12z/data=!4m5!3m4!1s0x88059e67de9a3861:0x2d24f51aad34c80b!8m2!3d42.7630722!4d-88.2142563";
squidclient
"http://www.help.1and1.co.uk/domains-c40986/transfer-domains-c79878";
squidclient
"https://community.cisco.com/t5/technology-and-support/ct-p/technology-support";
squidclient "https://www.capitalone.com/support-center";
squidclient "https://www.cnn.com/about";
squidclient "https://contact.nba.com/";
squidclient "https://www.espn.com/nfl/team/_/name/cle/cleveland-browns";
```
Update your squid.json enrichment to include Stellar enrichments. We're
going to duplicate the `whois` enrichment multiple times for the sake of
simplicity.
```
# cat $METRON_HOME/config/zookeeper/enrichments/squid.json
{
"enrichment" : {
"fieldMap" : {
"hbaseEnrichment" : [ "domain_without_subdomains" ],
"stellar" : {
"config" : {
"e1" : {
"user" : "ENRICHMENT_GET('user', ip_src_addr, 'enrichment', 't')"
},
"e2" : {
"dws1" : "ENRICHMENT_GET('whois', domain_without_subdomains,
'enrichment', 't')"
},
"e3" : {
"dws2" : "ENRICHMENT_GET('whois', domain_without_subdomains,
'enrichment', 't')"
},
"e4" : {
"dws3" : "ENRICHMENT_GET('whois', domain_without_subdomains,
'enrichment', 't')"
},
"e5" : {
"dws4" : "ENRICHMENT_GET('whois', domain_without_subdomains,
'enrichment', 't')"
},
"e6" : {
"dws5" : "ENRICHMENT_GET('whois', domain_without_subdomains,
'enrichment', 't')"
}
}
}
},
"fieldToTypeMap" : {
"domain_without_subdomains" : [ "whois" ]
},
"config" : { }
},
"threatIn
[jira] [Created] (METRON-2246) rpm-docker: minimise use of bind mounts due to performance
Dale Richardson created METRON-2246: --- Summary: rpm-docker: minimise use of bind mounts due to performance Key: METRON-2246 URL: https://issues.apache.org/jira/browse/METRON-2246 Project: Metron Issue Type: Improvement Reporter: Dale Richardson Docker bind mounts on MacOS have a reputation for being extremely slow. The rpm-docker module uses bind mounts to share out the rpm-docker project directory to the RPM creation docker image, which then uses it as a workspace when creating the METRON RPMS, and means of pushing the resulting RPM's back to the hosting build box. Bind mounts remain the most convenient way of bulk file export from a docker image, but the processes can be sped up by building the rpm-docker image with the input file already included, so that any file I/O by the rpm building process in its workspace does not suffer a performance penalty. -- This message was sent by Atlassian Jira (v8.3.2#803003)
