[ https://issues.apache.org/jira/browse/SOLR-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185978#comment-16185978 ]
Erik Persson edited comment on SOLR-11413 at 9/29/17 3:26 PM: -------------------------------------------------------------- Example error messages using Solr 6.6.0. {quote}2017-09-28 19:09:16.226 ERROR (metrics-graphite-reporter-2-thread-1) [ ] c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report. Exception was suppressed. java.lang.IllegalStateException: Already connected at com.codahale.metrics.graphite.Graphite.connect(Graphite.java:111) at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:240) at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235) at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-09-28 19:09:16.226 WARN (metrics-graphite-reporter-1-thread-1) [ ] c.c.m.g.GraphiteReporter Unable to report to Graphite java.io.IOException: Stream closed at java.io.BufferedWriter.ensureOpen(BufferedWriter.java:116) at java.io.BufferedWriter.write(BufferedWriter.java:221) at java.io.Writer.write(Writer.java:157) at com.codahale.metrics.graphite.Graphite.send(Graphite.java:138) at com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled(GraphiteReporter.java:328) at com.codahale.metrics.graphite.GraphiteReporter.reportMetered(GraphiteReporter.java:305) at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:255) at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235) at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-09-28 19:09:16.234 ERROR (metrics-graphite-reporter-3-thread-1) [ ] c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report. Exception was suppressed. java.lang.NullPointerException{quote} was (Author: erikpersson): Example error messages using Solr 6.6.0. {{2017-09-28 19:09:16.226 ERROR (metrics-graphite-reporter-2-thread-1) [ ] c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report. Exception was suppressed. java.lang.IllegalStateException: Already connected at com.codahale.metrics.graphite.Graphite.connect(Graphite.java:111) at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:240) at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235) at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-09-28 19:09:16.226 WARN (metrics-graphite-reporter-1-thread-1) [ ] c.c.m.g.GraphiteReporter Unable to report to Graphite java.io.IOException: Stream closed at java.io.BufferedWriter.ensureOpen(BufferedWriter.java:116) at java.io.BufferedWriter.write(BufferedWriter.java:221) at java.io.Writer.write(Writer.java:157) at com.codahale.metrics.graphite.Graphite.send(Graphite.java:138) at com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled(GraphiteReporter.java:328) at com.codahale.metrics.graphite.GraphiteReporter.reportMetered(GraphiteReporter.java:305) at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:255) at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235) at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-09-28 19:09:16.234 ERROR (metrics-graphite-reporter-3-thread-1) [ ] c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report. Exception was suppressed. java.lang.NullPointerException}} > SolrGraphiteReporter fails to report metrics due to non-thread safe code > ------------------------------------------------------------------------ > > Key: SOLR-11413 > URL: https://issues.apache.org/jira/browse/SOLR-11413 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics > Affects Versions: 6.6, 7.0 > Reporter: Erik Persson > Attachments: SOLR-11413.patch > > > Symptom: > Intermittent errors writing graphite metrics. Errors indicate use of sockets > which have already been closed. > Cause: > SolrGraphiteReporter caches and shares dropwizard Graphite instances. These > reporters are not thread safe as they open and close an instance variable of > type GraphiteSender. On modern bare metal hardware this problem was observed > consistently, and resulted in the majority of metrics failing to be delivered > to graphite. > Proposed Fix: > Graphite (and PickledGraphite) are not designed to be cached, and should not > be. > Test: > Patch file includes test which forces error. > Alternative Fixes Considered: > * Totally change solr metrics architecture to use a single metrics registry - > seems undesirable and impractical > * Create a synchronized or otherwise thread-safe implementation of dropwizard > graphite reporter - should be fixed upstream in dropwizard and not obviously > preferred to current model -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org