Re: Removing historical data

2016-06-14 Thread Nick Allen
The standard deployment does setup some cleanup tasks that purges 'old' data. This seems to be different from what you're asking though. What do you imagine using as your "backup space" if not /dev/null? On Tue, Jun 14, 2016 at 7:45 AM, Kuba Sienkiewicz wrote: > PS. I'm saying about automat

[GitHub] incubator-metron pull request #145: METRON-212: Allow additional Elasticsear...

2016-06-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/145 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...

2016-06-14 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/145 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] incubator-metron pull request #150: METRON-222: Address ConcurrentModificati...

2016-06-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/150 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron pull request #148: METRON-215: Fixing an NPE in the Message...

2016-06-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/148 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron issue #148: METRON-215: Fixing an NPE in the MessageParser

2016-06-14 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/148 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] incubator-metron issue #148: METRON-215: Fixing an NPE in the MessageParser

2016-06-14 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/148 +1. Good fix! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishe

[GitHub] incubator-metron pull request #144: METRON-211 Enable creation of new Kafka ...

2016-06-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/144 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...

2016-06-14 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/145 👍 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] incubator-metron issue #150: METRON-222: Address ConcurrentModificationExcep...

2016-06-14 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/150 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...

2016-06-14 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/145 Bump - anyone? Bueller? @nickwallen - those changes you requested have been for a bit. --- If your project is set up for it, you can reply to this email and have your reply ap

[GitHub] incubator-metron issue #150: METRON-222: Address ConcurrentModificationExcep...

2016-06-14 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/150 💯 - really solid work, Casey. Looks like it was a bear to track down. I ran this in EC2 with 2 workers and 15 executors for 16 minutes - no error. 1 worker and 15

[GitHub] incubator-metron issue #144: METRON-211 Enable creation of new Kafka topics ...

2016-06-14 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/144 +1 this looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wi

Re: [GitHub] incubator-metron issue #135: METRON-152: Add support for ES 2.3.x and Kibana...

2016-06-14 Thread Nick Allen
You are correct that the Kibana 4 interface does not currently have a dashboard that gets created automatically like the older Kibana 3 dashboard. I am actually working on that now per METRON-219; https://issues.apache.org/jira/browse/METRON-219. The link below is a fairly good introduction to us

[GitHub] incubator-metron pull request #151: METRON-223: Invalid and Erroneous messag...

2016-06-14 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/151 METRON-223: Invalid and Erroneous messages should go to kafka for further analysis in the Parser Topology Right now we send error messages and invalid messages to separate streams in the

[GitHub] incubator-metron issue #150: METRON-222: Address ConcurrentModificationExcep...

2016-06-14 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/150 I don't see a difference in the clone method vs the method currently implemented. I think we should evaluate in a follow-on JIRA whether any of the clones are needed. It seems that none

[GitHub] incubator-metron issue #150: METRON-222: Address ConcurrentModificationExcep...

2016-06-14 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/150 Good deal on tracking this one down. Think it's worth switching back to the .clone method in the BulkMessageWriterBolt? --- If your project is set up for it, you can reply to this email

Re: Removing historical data

2016-06-14 Thread Kuba Sienkiewicz
PS. I'm saying about automatically moving data away from hdfs to some sort of backup space. Sorry for inaccuracy in previous email (removing). wt., 14.06.2016 o 13:32 użytkownik Kuba Sienkiewicz napisał: > Hi all, > I've tested metron for 2 days and I already have over 20GB of data (w/o > any sp

Removing historical data

2016-06-14 Thread Kuba Sienkiewicz
Hi all, I've tested metron for 2 days and I already have over 20GB of data (w/o any special network traffic, server just stood untouched). What are good practices with such big amounts of data that metron stores? Also do metron support removing historical data automatically? Best Regards, Jakub Si