Does 'netstat -an' show incoming sockets for these threads? What Solr release is this?
Is this one long upload of 20m documents without committing? Are you doing periodic commits, or automatic commits (in solrconfig.xml)? How large are the documents? Could this be a jetty bug? Have you tried this on tomcat? Lance On Sun, Jun 20, 2010 at 4:50 PM, <karl.wri...@nokia.com> wrote: > So far they've stayed around for 72 hours and counting. > Also, I don't care what the stack trace says - CPU is listing as 500%. So it > may be momentarily "blocked" but then it must loop. > > Karl > ________________________________________ > From: ext Lance Norskog [goks...@gmail.com] > Sent: Saturday, June 19, 2010 8:51 PM > To: dev@lucene.apache.org > Subject: Re: Doppleganger threads after ingestion completed > > "Chewing up cpu" or "blocked". The stack trace says it's blocked. > > The sockets are abandoned by the program, yes, but TCP/IP itself has a > complex sequence for shutting down sockets that takes a few minutes. > If these sockets stay around for hours, then there's a real problem. > (In fact, there is a bug in the TCP/IP specification, 40 years old, > that causes zombie sockets that never shut down.) > > The HTTP solr server really needs a socket close() method. > > On Thu, Jun 17, 2010 at 6:08 AM, <karl.wri...@nokia.com> wrote: >> Folks, >> >> I ran 20,000,000 records into Solr via the extractingUpdateRequestHandler >> under jetty. The previous problems with resources have apparently been >> resolved by using Http1.1 with keep-alive, rather than creating and >> destroying 20,000,000 sockets. ;-) However, after the client terminates, I >> still find the Solr process chewing away CPU – indeed, there were 5 threads >> doing this. >> >> A thread dump yields the following partial trace for all 5 threads: >> >> "btpool0-13" prio=10 tid=0x0000000041391000 nid=0xe7c runnable >> [0x00007f4a8c789000] >> java.lang.Thread.State: RUNNABLE >> at >> org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:925) >> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897) >> at >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) >> at >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:924) >> at >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:904) >> at org.apache.commons.fileupload.util.Streams.copy(Streams.java:119) >> at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) >> at >> org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362) >> at >> org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) >> at >> org.apache.solr.servlet.MultipartRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:343) >> at >> org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:396) >> at >> org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:114) >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) >> … >> >> I could be wrong, but it looks to me like either jetty or fileupload may >> have a problem here. I have not looked at the jetty source code, but >> infinitely spinning processes even after the socket has been abandoned do >> not seem reasonable to me. Thoughts? >> >> Karl >> >> > > > > -- > Lance Norskog > goks...@gmail.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > -- Lance Norskog goks...@gmail.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
