Re: Doppleganger threads after ingestion completed
I can't think of anything else. On 6/21/10, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Some answers below. (1) netstat -an shows no sockets at all. Remember, the client process is gone, dead, shut down. (2) This is Solr 1.5 from approximately mid-March. (3) Autocommit was on, using the standard configuration present in the example. This could well be a jetty bug and, no, I have not tried tomcat yet. Karl From: ext Lance Norskog [goks...@gmail.com] Sent: Sunday, June 20, 2010 10:47 PM To: dev@lucene.apache.org Subject: Re: Doppleganger threads after ingestion completed Does 'netstat -an' show incoming sockets for these threads? What Solr release is this? Is this one long upload of 20m documents without committing? Are you doing periodic commits, or automatic commits (in solrconfig.xml)? How large are the documents? Could this be a jetty bug? Have you tried this on tomcat? Lance On Sun, Jun 20, 2010 at 4:50 PM, karl.wri...@nokia.com wrote: So far they've stayed around for 72 hours and counting. Also, I don't care what the stack trace says - CPU is listing as 500%. So it may be momentarily blocked but then it must loop. Karl From: ext Lance Norskog [goks...@gmail.com] Sent: Saturday, June 19, 2010 8:51 PM To: dev@lucene.apache.org Subject: Re: Doppleganger threads after ingestion completed Chewing up cpu or blocked. The stack trace says it's blocked. The sockets are abandoned by the program, yes, but TCP/IP itself has a complex sequence for shutting down sockets that takes a few minutes. If these sockets stay around for hours, then there's a real problem. (In fact, there is a bug in the TCP/IP specification, 40 years old, that causes zombie sockets that never shut down.) The HTTP solr server really needs a socket close() method. On Thu, Jun 17, 2010 at 6:08 AM, karl.wri...@nokia.com wrote: Folks, I ran 20,000,000 records into Solr via the extractingUpdateRequestHandler under jetty. The previous problems with resources have apparently been resolved by using Http1.1 with keep-alive, rather than creating and destroying 20,000,000 sockets. ;-) However, after the client terminates, I still find the Solr process chewing away CPU – indeed, there were 5 threads doing this. A thread dump yields the following partial trace for all 5 threads: btpool0-13 prio=10 tid=0x41391000 nid=0xe7c runnable [0x7f4a8c789000] java.lang.Thread.State: RUNNABLE at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:925) at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:924) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:904) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:119) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at org.apache.solr.servlet.MultipartRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:343) at org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:396) at org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:114) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) … I could be wrong, but it looks to me like either jetty or fileupload may have a problem here. I have not looked at the jetty source code, but infinitely spinning processes even after the socket has been abandoned do not seem reasonable to me. Thoughts? Karl -- Lance Norskog goks...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org -- Lance Norskog goks...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
RE: Doppleganger threads after ingestion completed
Some answers below. (1) netstat -an shows no sockets at all. Remember, the client process is gone, dead, shut down. (2) This is Solr 1.5 from approximately mid-March. (3) Autocommit was on, using the standard configuration present in the example. This could well be a jetty bug and, no, I have not tried tomcat yet. Karl From: ext Lance Norskog [goks...@gmail.com] Sent: Sunday, June 20, 2010 10:47 PM To: dev@lucene.apache.org Subject: Re: Doppleganger threads after ingestion completed Does 'netstat -an' show incoming sockets for these threads? What Solr release is this? Is this one long upload of 20m documents without committing? Are you doing periodic commits, or automatic commits (in solrconfig.xml)? How large are the documents? Could this be a jetty bug? Have you tried this on tomcat? Lance On Sun, Jun 20, 2010 at 4:50 PM, karl.wri...@nokia.com wrote: So far they've stayed around for 72 hours and counting. Also, I don't care what the stack trace says - CPU is listing as 500%. So it may be momentarily blocked but then it must loop. Karl From: ext Lance Norskog [goks...@gmail.com] Sent: Saturday, June 19, 2010 8:51 PM To: dev@lucene.apache.org Subject: Re: Doppleganger threads after ingestion completed Chewing up cpu or blocked. The stack trace says it's blocked. The sockets are abandoned by the program, yes, but TCP/IP itself has a complex sequence for shutting down sockets that takes a few minutes. If these sockets stay around for hours, then there's a real problem. (In fact, there is a bug in the TCP/IP specification, 40 years old, that causes zombie sockets that never shut down.) The HTTP solr server really needs a socket close() method. On Thu, Jun 17, 2010 at 6:08 AM, karl.wri...@nokia.com wrote: Folks, I ran 20,000,000 records into Solr via the extractingUpdateRequestHandler under jetty. The previous problems with resources have apparently been resolved by using Http1.1 with keep-alive, rather than creating and destroying 20,000,000 sockets. ;-) However, after the client terminates, I still find the Solr process chewing away CPU – indeed, there were 5 threads doing this. A thread dump yields the following partial trace for all 5 threads: btpool0-13 prio=10 tid=0x41391000 nid=0xe7c runnable [0x7f4a8c789000] java.lang.Thread.State: RUNNABLE at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:925) at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:924) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:904) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:119) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at org.apache.solr.servlet.MultipartRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:343) at org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:396) at org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:114) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) … I could be wrong, but it looks to me like either jetty or fileupload may have a problem here. I have not looked at the jetty source code, but infinitely spinning processes even after the socket has been abandoned do not seem reasonable to me. Thoughts? Karl -- Lance Norskog goks...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org -- Lance Norskog goks...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Doppleganger threads after ingestion completed
Does 'netstat -an' show incoming sockets for these threads? What Solr release is this? Is this one long upload of 20m documents without committing? Are you doing periodic commits, or automatic commits (in solrconfig.xml)? How large are the documents? Could this be a jetty bug? Have you tried this on tomcat? Lance On Sun, Jun 20, 2010 at 4:50 PM, karl.wri...@nokia.com wrote: So far they've stayed around for 72 hours and counting. Also, I don't care what the stack trace says - CPU is listing as 500%. So it may be momentarily blocked but then it must loop. Karl From: ext Lance Norskog [goks...@gmail.com] Sent: Saturday, June 19, 2010 8:51 PM To: dev@lucene.apache.org Subject: Re: Doppleganger threads after ingestion completed Chewing up cpu or blocked. The stack trace says it's blocked. The sockets are abandoned by the program, yes, but TCP/IP itself has a complex sequence for shutting down sockets that takes a few minutes. If these sockets stay around for hours, then there's a real problem. (In fact, there is a bug in the TCP/IP specification, 40 years old, that causes zombie sockets that never shut down.) The HTTP solr server really needs a socket close() method. On Thu, Jun 17, 2010 at 6:08 AM, karl.wri...@nokia.com wrote: Folks, I ran 20,000,000 records into Solr via the extractingUpdateRequestHandler under jetty. The previous problems with resources have apparently been resolved by using Http1.1 with keep-alive, rather than creating and destroying 20,000,000 sockets. ;-) However, after the client terminates, I still find the Solr process chewing away CPU – indeed, there were 5 threads doing this. A thread dump yields the following partial trace for all 5 threads: btpool0-13 prio=10 tid=0x41391000 nid=0xe7c runnable [0x7f4a8c789000] java.lang.Thread.State: RUNNABLE at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:925) at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:924) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:904) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:119) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at org.apache.solr.servlet.MultipartRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:343) at org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:396) at org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:114) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) … I could be wrong, but it looks to me like either jetty or fileupload may have a problem here. I have not looked at the jetty source code, but infinitely spinning processes even after the socket has been abandoned do not seem reasonable to me. Thoughts? Karl -- Lance Norskog goks...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org -- Lance Norskog goks...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Doppleganger threads after ingestion completed
Chewing up cpu or blocked. The stack trace says it's blocked. The sockets are abandoned by the program, yes, but TCP/IP itself has a complex sequence for shutting down sockets that takes a few minutes. If these sockets stay around for hours, then there's a real problem. (In fact, there is a bug in the TCP/IP specification, 40 years old, that causes zombie sockets that never shut down.) The HTTP solr server really needs a socket close() method. On Thu, Jun 17, 2010 at 6:08 AM, karl.wri...@nokia.com wrote: Folks, I ran 20,000,000 records into Solr via the extractingUpdateRequestHandler under jetty. The previous problems with resources have apparently been resolved by using Http1.1 with keep-alive, rather than creating and destroying 20,000,000 sockets. ;-) However, after the client terminates, I still find the Solr process chewing away CPU – indeed, there were 5 threads doing this. A thread dump yields the following partial trace for all 5 threads: btpool0-13 prio=10 tid=0x41391000 nid=0xe7c runnable [0x7f4a8c789000] java.lang.Thread.State: RUNNABLE at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:925) at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:924) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:904) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:119) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at org.apache.solr.servlet.MultipartRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:343) at org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:396) at org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:114) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) … I could be wrong, but it looks to me like either jetty or fileupload may have a problem here. I have not looked at the jetty source code, but infinitely spinning processes even after the socket has been abandoned do not seem reasonable to me. Thoughts? Karl -- Lance Norskog goks...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
