I'm sorry guys.. I made a mistake.. This is not coming out of hadoop.. This
thread is coming out of nutch bean. Sorry.. I should have looked more
carefully.. I am still learning this stuff..
Here is my code as it is.. Let me look into this some more..
NativeCrawler nativeCrawler = new NativeCrawler("www.dcu.com",
"dcu-index", 2, 5);
int maxHits = 1000;
--> NutchBean nutchBean = new NutchBean(nativeCrawler.getConfig(),
nativeCrawler.getIndexPath());
Query nutchQuery = Query.parse("credit",
nativeCrawler.getConfig());
Hits nutchHits = nutchBean.search(nutchQuery, maxHits);
nutchQuery = null;
nutchBean.close();
nutchBean = null;
Native Crawler is my version of java.Crawl code.. Which works great.. I am not
closing down the query part of my system correctly and will now go and read on
that.. Please forgive my taking your time.. I should have been a little more
precise in my work.. Sigh.. It happens when you are rushing on to many
projects.. Sorry guys and thanks so much for the help that you guys gave me.. I
will post the solution to this for us.
ray
-----Original Message-----
From: Lukas, Ray [mailto:[email protected]]
Sent: Thursday, April 23, 2009 9:21 AM
To: [email protected]
Subject: RE: Hadoop thread seems to remain alive
Hey Ray.. Great name you have there.. HA..
I don't actually care about deleting these files.. That is not the issue.. See
I have embedded Nutch in my application. That application calls nutch over and
over again to do crawling and index creation.. This thread that stays alive..
It eventually exceeds some limit (native thread) in Java and crashes my
application.. So that is why I need to find and properly close down that
service or whatever. I noticed that Hadopp files are still locked and so I am
thinking that as a hint that it is hadopp..
Bottom line is
When you run Crawl in the java directory, some thread stays open.. That thread
is killing me.. What is it that stays alive past the completion of the
Crawl.java code...
If you run org.apache.nutch.crawl.Crawl from within java/eclispe something
stays alive.. How to clise that is the issue..
See what I am asking..
Ray, the other ray..
-----Original Message-----
From: Raymond Balmès [mailto:[email protected]]
Sent: Thursday, April 23, 2009 8:23 AM
To: [email protected]
Subject: Re: Hadoop thread seems to remain alive
Same problems... even rebooting the PC does not always solve the issue,
files remain locked.
I have gone the brutal way and use unlocker.exe but I mean to find out
what's going wrong so I will keep posted on this one.
-Ray-
2009/4/23 Lukas, Ray <[email protected]>
> Question:
> What is the proper accepted and safe way to shut down nutch (hadoop)
> after I am done with it?
>
> Hadoop.getFileSystem().closeAll() ??
> I did try this and no luck. Anyone else having this problem?
>
> Thanks guys.. Thanks, if/when I find it I will post it for everyone.
> Ray
>
