Nutch bean does have a close.. It just does not kill all the threads.. Each creation of a Nutch Bean kicks off a new thread and that thread never dies.. So... In my case.. I do so many queries I throw an exception because I exhaust my allotment of threads.. Yeah I call that many queries.. I have patched this code and it works great now.. I will email you a copy.. If that is okay.. And give you a hand getting it working.. If you want.. Just let me know.. ray
-----Original Message----- From: Raymond Balmès [mailto:[email protected]] Sent: Friday, April 24, 2009 2:51 AM To: [email protected] Subject: Re: Hadoop thread seems to remain alive Actually I have my files locked when exiting from Tomcat, no matter how I exit gracefull or not, probably due to some lost threads. Since the servlet uses the same NutchBean looks like a similar issue as yours. Maybe there is no nutchBean.close() being called I will look for it when I have more time for this. -The other Ray- 2009/4/23 Lukas, Ray <[email protected]> > I'm sorry guys.. I made a mistake.. This is not coming out of hadoop.. This > thread is coming out of nutch bean. Sorry.. I should have looked more > carefully.. I am still learning this stuff.. > > Here is my code as it is.. Let me look into this some more.. > > NativeCrawler nativeCrawler = new NativeCrawler(" > www.dcu.com", "dcu-index", 2, 5); > int maxHits = 1000; > --> NutchBean nutchBean = new > NutchBean(nativeCrawler.getConfig(), nativeCrawler.getIndexPath()); > Query nutchQuery = Query.parse("credit", > nativeCrawler.getConfig()); > Hits nutchHits = nutchBean.search(nutchQuery, maxHits); > > nutchQuery = null; > nutchBean.close(); > nutchBean = null; > > Native Crawler is my version of java.Crawl code.. Which works great.. I am > not closing down the query part of my system correctly and will now go and > read on that.. Please forgive my taking your time.. I should have been a > little more precise in my work.. Sigh.. It happens when you are rushing on > to many projects.. Sorry guys and thanks so much for the help that you guys > gave me.. I will post the solution to this for us. > > ray > > -----Original Message----- > From: Lukas, Ray [mailto:[email protected]] > Sent: Thursday, April 23, 2009 9:21 AM > To: [email protected] > Subject: RE: Hadoop thread seems to remain alive > > Hey Ray.. Great name you have there.. HA.. > > I don't actually care about deleting these files.. That is not the issue.. > See I have embedded Nutch in my application. That application calls nutch > over and over again to do crawling and index creation.. This thread that > stays alive.. It eventually exceeds some limit (native thread) in Java and > crashes my application.. So that is why I need to find and properly close > down that service or whatever. I noticed that Hadopp files are still locked > and so I am thinking that as a hint that it is hadopp.. > > Bottom line is > > When you run Crawl in the java directory, some thread stays open.. That > thread is killing me.. What is it that stays alive past the completion of > the Crawl.java code... > If you run org.apache.nutch.crawl.Crawl from within java/eclispe something > stays alive.. How to clise that is the issue.. > > See what I am asking.. > > Ray, the other ray.. > > -----Original Message----- > From: Raymond Balmès [mailto:[email protected]] > Sent: Thursday, April 23, 2009 8:23 AM > To: [email protected] > Subject: Re: Hadoop thread seems to remain alive > > Same problems... even rebooting the PC does not always solve the issue, > files remain locked. > > I have gone the brutal way and use unlocker.exe but I mean to find out > what's going wrong so I will keep posted on this one. > > -Ray- > > 2009/4/23 Lukas, Ray <[email protected]> > > > Question: > > What is the proper accepted and safe way to shut down nutch (hadoop) > > after I am done with it? > > > > Hadoop.getFileSystem().closeAll() ?? > > I did try this and no luck. Anyone else having this problem? > > > > Thanks guys.. Thanks, if/when I find it I will post it for everyone. > > Ray > > >
