Re: Storm indexing topology possible memory issues

I feel like that's less than the sensor stubs send in full dev, is that truly the number? I think Ryan is on the right track with his suggestions. Happy to chat more on irc, ping me if you're around when I'm on. Jon On Mon, Aug 14, 2017, 19:17 Laurens Vets wrote: > Hey

Re: Storm indexing topology possible memory issues

Hey Ryan, Thank you for that information. The problem is that we don't know how many logs we'll be ingesting and we wanted to scale out when we got into problems... What would the minimum requirements be for a simple cluster ingesting maybe 1-25000 messages a day or so... On 2017-08-14

Re: Storm indexing topology possible memory issues

Laurens, 2 nodes with 32G of RAM is really small considering all the different components included with Metron. Assuming this is a demo or POC cluster, otherwise it's WAY under-sized. This isn't specific to Metron by the way, it applies to any HDP (or similar distribution) cluster. You will

Re: Storm indexing topology possible memory issues

Try increasing nofile and nproc for your storm service account. Jon On Mon, Aug 14, 2017, 12:46 Laurens Vets wrote: > Hi List, > > I'm seeing the following errors in our indexing topology: > > kafkaSpout: > java.lang.OutOfMemoryError: GC overhead limit exceeded at >

Storm indexing topology possible memory issues

Hi List, I'm seeing the following errors in our indexing topology: kafkaSpout: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272) at org.apache.kafka.common.utils.Utils.toArray(Utils.java:265) at

Re: Offset lag tool?

It's part of kafka, actually. You can find it documented at https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker On Mon, Aug 14, 2017 at 11:32 AM, Laurens Vets wrote: > From the Performance-tuning-guide.md: "You will find the

Offset lag tool?

From the Performance-tuning-guide.md: "You will find the offset lag tool indispensable while verifying your settings." Probably because it's Monday, but I can't seem to find this offset lag tool anywhere...

Re: Issues with indexing topology

Hi Laurens, We're doing additional testing, but it seems to be fixed, yes. Are you experiencing the same problem? 2017-08-14 6:12 GMT+02:00 Laurens Vets : > Hi Guillem, > > Did you eventually fix the problem? > > On 2017-08-01 11:00, Guillem Mateos wrote: > > On the