https://bugzilla.wikimedia.org/show_bug.cgi?id=63490
Bug ID: 63490 Summary: Logstash occasionally hangs as a result of a memory leak caused by non-UTF-8 input Product: Wikimedia Version: wmf-deployment Hardware: All OS: All Status: NEW Severity: normal Priority: Unprioritized Component: Logstash Assignee: wikibugs-l@lists.wikimedia.org Reporter: bda...@wikimedia.org CC: aschulz4...@gmail.com, bda...@wikimedia.org, g...@wikimedia.org, o...@wikimedia.org Web browser: --- Mobile Platform: --- Logstash stops processing log events occasionally in both beta and production. I happened to catch the cause in the logs today: {:timestamp=>"2014-04-03T12:03:43.449000+0000", :message=>"Failed to flush outgo ing items", :outgoing_count=>1, :exception=>java.lang.OutOfMemoryError: GC overh ead limit exceeded, :backtrace=>["java.nio.HeapByteBuffer.<init>(HeapByteBuffer. java:57)", "java.nio.ByteBuffer.allocate(ByteBuffer.java:329)", "sun.nio.cs.Stre amDecoder.<init>(StreamDecoder.java:249)", "sun.nio.cs.StreamDecoder.<init>(StreamDecoder.java:229)", "sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:68)", "java.io.InputStreamReader.<init>(InputStreamReader.java:74)", "org.jruby.util.RubyDateFormatter.compilePattern(RubyDateFormatter.java:261)", "org.jruby.util.RubyDateFormatter.compileAndFormat(RubyDateFormatter.java:360)", "org.jruby.RubyTime.strftime(RubyTime.java:425)", "org.jruby.RubyTime$INVOKER$i$1$0$strftime.call(RubyTime$INVOKER$i$1$0$strftime.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)", "org.jruby.ast.AttrAssignTwoArgNode.interpret(AttrAssignTwoArgNode.java:36)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)", "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)", "org.jruby.runtime.Block.call(Block.java:101)", "org.jruby.RubyProc.call(RubyProc.java:290)", "org.jruby.RubyProc.call19(RubyProc.java:271)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "rubyjit.Cabin::Channel$$publish_0CC14B6056DE0968D0FA81C715E46013862F8AD3.block_0$RUBY$__file__(file:/opt/logstash/logstash.jar!/cabin/channel.rb:161)", "rubyjit$Cabin::Channel$$publish_0CC14B6056DE0968D0FA81C715E46013862F8AD3$block_0$RUBY$__file__.call(rubyjit$Cabin::Channel$$publish_0CC14B6056DE0968D0FA81C715E46013862F8AD3$block_0$RUBY$__file__)", "org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)", "org.jruby.runtime.Block.yield(Block.java:142)", "org.jruby.RubyArray.eachCommon(RubyArray.java:1606)", "org.jruby.RubyArray.each(RubyArray.java:1613)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)"], :level=>:warn} This looks to be a known bug upstream related to "malformed utf-8" errors: https://logstash.jira.com/browse/LOGSTASH-1389 We get malformed utf-8 errors occasionally mostly from database inserts: {:timestamp=>"2014-04-03T10:44:52.106000+0000", :message=>"Received an event tha t has a different character encoding than you configured.", :text=>"1235761 web wikidatawiki-bcb77336: 0.6721 20.2M DatabaseBase::query: Writes done: INSERT INTO `text` (old_id,old_text,old_flags) VALUES (NULL,'=\\x8EA\\x8B\\xC20 \\x85\\ xFF\\x8A \\x82\\x97 L[5\\xF6\\xBC\\xB7]\\x84BAX\\xF10\\xAD\\x93:4\\x8D\\xBBMV]K\ \xFF\\xBB\\xB1\\x8A\\x97\\xE1\\\\r3ッ \\x83% \\xC8z ܾ\\x8Emq\\xDB\\xE40 8\\x90\\ xAB:\\xFE\\xF1|\\xB2\\xEF{\\x85\\xFE\\xD2l>\\xB6E\\xFE\\xD9\\\\\\\\\\xBB\\xF2\\x 9B\\xEAb\\xFCE\\xC3\\xE8\\xC8A\\xB6\\xDB \\xA8 r\\xFB\\xD0=\\xB4a\\xC2 \\xCD \\x 81\\x90\\x89\\x9A p\\xBEc[\\x83\\\\0\\xF637\\xC1\\xC9k \\xB6ߧ\\xBB 5y,\\xD5*\\x9 9\\xA6\\xA9\\x8E\\xD7z\\xB1\\x8AҹT\\x91>$2\\\\\\\"J\\xAA\\xA8T\\xCBx\\x89\\xA4\\ tS \\x92:\\xB4\\\\rd2 \\xD2#\\xC2 b \\xDB\\xE6\\xC5Cֳ\\xFF\\u007F\\x90\\xB0\\xA7 \\xC4 \\xBE \\xEE','utf-8,gzip')\\n", :expected_charset=>"UTF-8", :level=>:warn} -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. _______________________________________________ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l