I'll see what I can do re batch sizes. I can pretty much gather any metric you want. Which one do you need?
On 2017-08-18 03:22, [email protected] wrote: > I'm grasping at straws but maybe play with your batch sizes? I hear now > there's a time based option too that I haven't played with. > > It would be helpful to have more metrics about resource utilization over time > in your cluster, are you able to gather anything and maybe put it into a > visualization tool? > > Jon > > On Thu, Aug 17, 2017, 16:35 Laurens Vets <[email protected]> wrote: > >> That seems close to the issue that I'm having... But there's a part I >> don't quite understand in my case. >> >> Everything's working fine for days and then suddenly, Java throws an >> error (see other mail to the mailing list) and I can't get Metron stable >> again. It's always either the indexingBolt or hdfsIndexingBolt. The >> error I see the most after this is "java.lang.OutOfMemoryError: GC >> overhead limit exceeded"... >> >> It seems that somewhere there's a big pileup of messages which Metron >> suddenly can't process. >> >> Any ideas on how to further troubleshoot this? >> >> On 2017-08-17 11:10, [email protected] wrote: >>> I used to run into similar issues when my environment was resource >>> constrained but never ran it to root cause. It has been a long time >>> since >>> I was in this scenario to re-test. >>> >>> https://issues.apache.org/jira/projects/METRON/issues/METRON-485 >>> >>> Jon >>> >>> On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> Simple question, when I stop Metron, Kafka & Storm via Ambari, I still >>>> see the storm worker processes running, is this expected? >>>> > > -- > > Jon
