Cool// well at least you know that I was willing to give you a hand and some code.. You are not alone in this cruel world..
-----Original Message----- From: Raymond Balmès [mailto:[email protected]] Sent: Saturday, April 25, 2009 5:28 AM To: [email protected] Subject: Re: Hadoop thread seems to remain alive Hey ray, Actually found my problem, I wasn't stopping Tomcat at the right moment & the right way... so it kept some threads/locks. If I do it using the Windows proper service... works fine. -Ray- 2009/4/24 Lukas, Ray <[email protected]> > What does that thread do.. Well you guessed it (and this is a hint why I > first thought it was in a problem in Hadoop) it opens up and gathers a > Hadoop segments.. > Take a peek at FetchedSegments, there is a Thread in there (class) called > SegmentUpdater.. It never dies.. You see, "while (true)".. That is your/our > problem.. I addded a membervariable that is set inside the > FetchedSegments.close method, which is called by the NutchBean.close method > to set that member variable. Once set. The loop exits and then thread > expires, maybe after it comes out of sleeping, but it does expire.... > You see what I am saying.. > > -----Original Message----- > From: Raymond Balmès [mailto:[email protected]] > Sent: Friday, April 24, 2009 2:51 AM > To: [email protected] > Subject: Re: Hadoop thread seems to remain alive > > Actually I have my files locked when exiting from Tomcat, no matter how I > exit gracefull or not, probably due to some lost threads. > Since the servlet uses the same NutchBean looks like a similar issue as > yours. > > Maybe there is no nutchBean.close() being called I will look for it when I > have more time for this. > > -The other Ray- > > > 2009/4/23 Lukas, Ray <[email protected]> > > > I'm sorry guys.. I made a mistake.. This is not coming out of hadoop.. > This > > thread is coming out of nutch bean. Sorry.. I should have looked more > > carefully.. I am still learning this stuff.. > > > > Here is my code as it is.. Let me look into this some more.. > > > > NativeCrawler nativeCrawler = new NativeCrawler(" > > www.dcu.com", "dcu-index", 2, 5); > > int maxHits = 1000; > > --> NutchBean nutchBean = new > > NutchBean(nativeCrawler.getConfig(), nativeCrawler.getIndexPath()); > > Query nutchQuery = Query.parse("credit", > > nativeCrawler.getConfig()); > > Hits nutchHits = nutchBean.search(nutchQuery, maxHits); > > > > nutchQuery = null; > > nutchBean.close(); > > nutchBean = null; > > > > Native Crawler is my version of java.Crawl code.. Which works great.. I > am > > not closing down the query part of my system correctly and will now go > and > > read on that.. Please forgive my taking your time.. I should have been a > > little more precise in my work.. Sigh.. It happens when you are rushing > on > > to many projects.. Sorry guys and thanks so much for the help that you > guys > > gave me.. I will post the solution to this for us. > > > > ray > > > > -----Original Message----- > > From: Lukas, Ray [mailto:[email protected]] > > Sent: Thursday, April 23, 2009 9:21 AM > > To: [email protected] > > Subject: RE: Hadoop thread seems to remain alive > > > > Hey Ray.. Great name you have there.. HA.. > > > > I don't actually care about deleting these files.. That is not the > issue.. > > See I have embedded Nutch in my application. That application calls nutch > > over and over again to do crawling and index creation.. This thread that > > stays alive.. It eventually exceeds some limit (native thread) in Java > and > > crashes my application.. So that is why I need to find and properly close > > down that service or whatever. I noticed that Hadopp files are still > locked > > and so I am thinking that as a hint that it is hadopp.. > > > > Bottom line is > > > > When you run Crawl in the java directory, some thread stays open.. That > > thread is killing me.. What is it that stays alive past the completion of > > the Crawl.java code... > > If you run org.apache.nutch.crawl.Crawl from within java/eclispe > something > > stays alive.. How to clise that is the issue.. > > > > See what I am asking.. > > > > Ray, the other ray.. > > > > -----Original Message----- > > From: Raymond Balmès [mailto:[email protected]] > > Sent: Thursday, April 23, 2009 8:23 AM > > To: [email protected] > > Subject: Re: Hadoop thread seems to remain alive > > > > Same problems... even rebooting the PC does not always solve the issue, > > files remain locked. > > > > I have gone the brutal way and use unlocker.exe but I mean to find out > > what's going wrong so I will keep posted on this one. > > > > -Ray- > > > > 2009/4/23 Lukas, Ray <[email protected]> > > > > > Question: > > > What is the proper accepted and safe way to shut down nutch (hadoop) > > > after I am done with it? > > > > > > Hadoop.getFileSystem().closeAll() ?? > > > I did try this and no luck. Anyone else having this problem? > > > > > > Thanks guys.. Thanks, if/when I find it I will post it for everyone. > > > Ray > > > > > >
