Hi:
Please attach the patch with a jira issue my mail account give me
trouble with attachment.
Kind regards
Zaheed
On 12/14/06, Doğacan Güney <[EMAIL PROTECTED]> wrote:
> Doğacan Güney wrote:
> > Hi,
> >
> > After hadoop-0.9.1, parsing and indexing doesn't seem to work.
> > If you parse while fetching then it is fine, but if you run parse as a
> > different job, it creates an essentially empty parse_data
> > directory(which has index files, but doesn't have data files). I am
> > looking into this, but so far, I couldn't find the source of error.
> >
> > Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The
> > parameter fs seems to be an instance of PhasedFileSystem which throws
> > exceptions on delete and {start,complete}LocalOutput. The following
> > patch should fix it, but may not be the best way of doing this.
> >
> > Index: src/java/org/apache/nutch/indexer/Indexer.java
> > ===================================================================
> > --- src/java/org/apache/nutch/indexer/Indexer.java (revision 487240)
> > +++ src/java/org/apache/nutch/indexer/Indexer.java (working copy)
> > @@ -94,11 +94,15 @@
> > final Path temp =
> > job.getLocalPath("index/_"+Integer.toString(new
> > Random().nextInt()));
> >
> > - fs.delete(perm); // delete old, if any
> > -
> > + final FileSystem dfs = FileSystem.get(job);
> > + + if (dfs.exists(perm)) {
> > + dfs.delete(perm); // delete old,
> > if any
> > + }
> > + final AnalyzerFactory factory = new AnalyzerFactory(job);
> > final IndexWriter writer = // build locally first
> > - new IndexWriter(fs.startLocalOutput(perm, temp).toString(),
> > + new IndexWriter(dfs.startLocalOutput(perm, temp).toString(),
> > new NutchDocumentAnalyzer(job), true);
> >
> > writer.setMergeFactor(job.getInt("indexer.mergeFactor", 10));
> > @@ -146,7 +150,7 @@
> > // optimize & close index
> > writer.optimize();
> > writer.close();
> > - fs.completeLocalOutput(perm, temp); // copy to dfs
> > + dfs.completeLocalOutput(perm, temp);
> > fs.createNewFile(new Path(perm, DONE_NAME));
> > } finally {
> > closed = true;
> >
> >
> >
> >
> Sorry about the patch, it got garbled somehow. I am attaching it, I hope
> mailing list doesn't drop attachments.
>
>
>
>
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
