Oh, and as an aside, ^C out of Solr at this point takes *forever* to run the shutdown hook. Well, I don't know for sure if it's forever, but it's been running for more than 5 minutes...
Karl -----Original Message----- From: Wright Karl (Nokia-S/Cambridge) Sent: Monday, June 14, 2010 8:21 AM To: [email protected] Subject: RE: Repeat to the right list: Solr spewage and possible re-entrancy problem? Good catch! r...@duck6:~# netstat -an | fgrep :8983 | wc 28223 169338 2257840 r...@duck6:~# ... and here's an example: tcp6 0 0 127.0.0.1:8983 127.0.0.1:44058 TIME_WAIT So, once again, what causes this behavior? How can I wind up with 28,000 socket connections hanging around, if both my client and Solr are behaving properly and are closing connections properly? (I suspect that the answer to my somewhat rhetorical question is, "this should not happen". But then the question becomes, "why IS it happening?") Karl -----Original Message----- From: Wright Karl (Nokia-S/Cambridge) Sent: Sunday, June 13, 2010 7:52 AM To: [email protected] Subject: RE: Repeat to the right list: Solr spewage and possible re-entrancy problem? Good idea. How would you prevent such a thing from occurring on the server? Or would this be the result of the client not doing something properly? Karl ________________________________________ From: ext Lance Norskog [[email protected]] Sent: Saturday, June 12, 2010 11:55 PM To: [email protected] Subject: Re: Repeat to the right list: Solr spewage and possible re-entrancy problem? There are situations where zombie sockets pile up at the server and keep zombie threads open. When this happens, check the total number of threads in the server JVM, and the total number of open or TIME_WAIT sockets. 'netstat -an | fgrep :8983' may find 2000 entries. Lance On Mon, Jun 7, 2010 at 7:35 AM, <[email protected]> wrote: > Hi folks, > > This morning I was experimenting with using multiple threads while indexing > some 20,000,000 records worth of content. In fact, my test spun up some 50 > threads, and happily chugged away for a couple of hours before I saw the > following output from my test code: > >>>>>>> > Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to > index record 6469124 > Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to > index record 6469551 > Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to > index record 6470592 > Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to > index record 6472454 > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:168) > at HttpPoster.getResponse(HttpPoster.java:280) > at HttpPoster.indexPost(HttpPoster.java:191) > at ParseAndLoad$PostThread.run(ParseAndLoad.java:638) > <<<<<< > > Looking at the solr-side output, I see nothing interesting at all: > >>>>>>> > Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute > INFO: [] webapp=/solr path=/update/extract > params={literal.nokia_longitude=9.78518981933594&literal.nokia_phone=%2B497971910474&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_district=Münster&literal.nokia_placerating=0&literal.id=6472724&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=1&literal.nokia_ppid=276u0wyw-c8cb7f4d6cd84a639a4e7d3570bf8814&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9985514322917&literal.nokia_postalcode=74405&literal.nokia_street=WeinhaldenstraÃe&literal.nokia_title=Dorfgemeinschaft+Münster+e.V.&literal.nokia_category=261} > status=0 QTime=1 > Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute > INFO: [] webapp=/solr path=/update/extract > params={literal.nokia_longitude=9.76717020670573&literal.nokia_phone=%2B497971950725&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_placerating=0&literal.id=6472737&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=13&literal.nokia_ppid=276u0wyw-d3bed6449fcb41b0adc50ae08e041f8d&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9974405924479&literal.nokia_fax=%2B497971950712&literal.nokia_postalcode=74405&literal.nokia_street=KochstraÃe&literal.nokia_title=BayWa+AG+Bau-+%26+Gartenmarkt&literal.nokia_category=194} > status=0 QTime=0 > Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute > INFO: [] webapp=/solr path=/update/extract > params={literal.nokia_longitude=9.77591044108073&literal.nokia_phone=%2B49797124009&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_district=Unterrot&literal.nokia_placerating=0&literal.id=6472739&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=28&literal.nokia_ppid=276u0wyw-d534d7a9235a4edf878d5e32a34bad8b&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9791788736979&literal.nokia_fax=%2B49797123431&literal.nokia_postalcode=74405&literal.nokia_street=HauptstraÃe&literal.nokia_title=Gastel+R.&literal.nokia_category=5} > status=0 QTime=1 > Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute > INFO: [] webapp=/solr path=/update/extract > params={literal.nokia_longitude=9.76935&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_placerating=5&literal.id=6472698&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=15&literal.nokia_ppid=276u0wyw-9544100e68d74162aff54783b9376134&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9981&literal.nokia_postalcode=74405&literal.nokia_street=KanzleistraÃe&literal.nokia_tag=Steuerberater&literal.nokia_tag=Business+%26+Service&literal.nokia_title=Consultis+GmbH&literal.nokia_category=215} > status=0 QTime=92 > Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute > INFO: [] webapp=/solr path=/update/extract > params={literal.nokia_longitude=9.77173970540364&literal.nokia_phone=%2B4979713238&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_placerating=0&literal.id=6472699&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=37&literal.nokia_ppid=276u0wyw-9600016fd0d248c9b442111838350f64&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9987182617188&literal.nokia_fax=%2B497971911639&literal.nokia_postalcode=74405&literal.nokia_street=KarlstraÃe&literal.nokia_title=Videothek,+5th+avenue+Peltekis+Apostolos&literal.nokia_category=5} > status=0 QTime=93 > <<<<<< > > It is unlikely (but, of course, not out of the question) that this hiccup is > due to some reentrancy problem in my test code. It is much more likely to > be some kind of a Solr multi-threaded race condition - especially since it > looks like a number of requests all failed at precisely the same time. This > is a Solr 1.5 build from mid-late March, FWIW. Does anyone know of an > extractingUpdateRequestHandler re-entrancy bug of this kind? > > Thanks, > Karl > > > > > -- Lance Norskog [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
