What does that thread do.. Well you guessed it (and this is a hint why I first thought it was in a problem in Hadoop) it opens up and gathers a Hadoop segments.. Take a peek at FetchedSegments, there is a Thread in there (class) called SegmentUpdater.. It never dies.. You see, "while (true)".. That is your/our problem.. I addded a membervariable that is set inside the FetchedSegments.close method, which is called by the NutchBean.close method to set that member variable. Once set. The loop exits and then thread expires, maybe after it comes out of sleeping, but it does expire.... You see what I am saying..
-----Original Message----- From: Raymond Balmès [mailto:[email protected]] Sent: Friday, April 24, 2009 2:51 AM To: [email protected] Subject: Re: Hadoop thread seems to remain alive Actually I have my files locked when exiting from Tomcat, no matter how I exit gracefull or not, probably due to some lost threads. Since the servlet uses the same NutchBean looks like a similar issue as yours. Maybe there is no nutchBean.close() being called I will look for it when I have more time for this. -The other Ray- 2009/4/23 Lukas, Ray <[email protected]> > I'm sorry guys.. I made a mistake.. This is not coming out of hadoop.. This > thread is coming out of nutch bean. Sorry.. I should have looked more > carefully.. I am still learning this stuff.. > > Here is my code as it is.. Let me look into this some more.. > > NativeCrawler nativeCrawler = new NativeCrawler(" > www.dcu.com", "dcu-index", 2, 5); > int maxHits = 1000; > --> NutchBean nutchBean = new > NutchBean(nativeCrawler.getConfig(), nativeCrawler.getIndexPath()); > Query nutchQuery = Query.parse("credit", > nativeCrawler.getConfig()); > Hits nutchHits = nutchBean.search(nutchQuery, maxHits); > > nutchQuery = null; > nutchBean.close(); > nutchBean = null; > > Native Crawler is my version of java.Crawl code.. Which works great.. I am > not closing down the query part of my system correctly and will now go and > read on that.. Please forgive my taking your time.. I should have been a > little more precise in my work.. Sigh.. It happens when you are rushing on > to many projects.. Sorry guys and thanks so much for the help that you guys > gave me.. I will post the solution to this for us. > > ray > > -----Original Message----- > From: Lukas, Ray [mailto:[email protected]] > Sent: Thursday, April 23, 2009 9:21 AM > To: [email protected] > Subject: RE: Hadoop thread seems to remain alive > > Hey Ray.. Great name you have there.. HA.. > > I don't actually care about deleting these files.. That is not the issue.. > See I have embedded Nutch in my application. That application calls nutch > over and over again to do crawling and index creation.. This thread that > stays alive.. It eventually exceeds some limit (native thread) in Java and > crashes my application.. So that is why I need to find and properly close > down that service or whatever. I noticed that Hadopp files are still locked > and so I am thinking that as a hint that it is hadopp.. > > Bottom line is > > When you run Crawl in the java directory, some thread stays open.. That > thread is killing me.. What is it that stays alive past the completion of > the Crawl.java code... > If you run org.apache.nutch.crawl.Crawl from within java/eclispe something > stays alive.. How to clise that is the issue.. > > See what I am asking.. > > Ray, the other ray.. > > -----Original Message----- > From: Raymond Balmès [mailto:[email protected]] > Sent: Thursday, April 23, 2009 8:23 AM > To: [email protected] > Subject: Re: Hadoop thread seems to remain alive > > Same problems... even rebooting the PC does not always solve the issue, > files remain locked. > > I have gone the brutal way and use unlocker.exe but I mean to find out > what's going wrong so I will keep posted on this one. > > -Ray- > > 2009/4/23 Lukas, Ray <[email protected]> > > > Question: > > What is the proper accepted and safe way to shut down nutch (hadoop) > > after I am done with it? > > > > Hadoop.getFileSystem().closeAll() ?? > > I did try this and no luck. Anyone else having this problem? > > > > Thanks guys.. Thanks, if/when I find it I will post it for everyone. > > Ray > > >
