Re: Indexing large files? - No answers yet...

2009-09-11 Thread Michael McCandless
On Fri, Sep 11, 2009 at 1:15 PM, wrote: > I've been testing out "paging" the document this past week. I'm > still working on getting a successful test and think I'm close. The > down side was a drastic slow down in indexing speed, and lots of > open files, but that was expected. You mean a sl

RE: Indexing large files? - No answers yet...

2009-09-11 Thread Paul_Murdoch
ne.apache.org [mailto:java-user-return-42283-paul_murdoch=emainc@lucene.apache.org] On Behalf Of Michael McCandless Sent: Friday, September 11, 2009 11:54 AM To: java-user@lucene.apache.org Subject: Re: Indexing large files? - No answers yet... To minimize Lucene's RAM usage during indexing, you sh

Re: Indexing large files? - No answers yet...

2009-09-11 Thread Michael McCandless
] On > Behalf Of Glen Newton > Sent: Friday, September 11, 2009 10:44 AM > To: java-user@lucene.apache.org > Subject: Re: Indexing large files? - No answers yet... > > Paul, > > I saw your last post and now understand the issues you face. > > I don't think there has

Re: Indexing large files? - No answers yet...

2009-09-11 Thread Brian Pinkerton
-user-return-42272- paul_murdoch=emainc@lucene.apache.org] On Behalf Of Glen Newton Sent: Friday, September 11, 2009 9:53 AM To: java-user@lucene.apache.org Subject: Re: Indexing large files? - No answers yet... In this project: http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.ht

RE: Indexing large files? - No answers yet...

2009-09-11 Thread Paul_Murdoch
java-user-return-42277-paul_murdoch=emainc@lucene.apache.org [mailto:java-user-return-42277-paul_murdoch=emainc@lucene.apache.org] On Behalf Of Glen Newton Sent: Friday, September 11, 2009 10:44 AM To: java-user@lucene.apache.org Subject: Re: Indexing large files? - No answers yet... Pau

Re: Indexing large files? - No answers yet...

2009-09-11 Thread Glen Newton
..@lucene.apache.org > [mailto:java-user-return-42272-paul_murdoch=emainc@lucene.apache.org] On > Behalf Of Glen Newton > Sent: Friday, September 11, 2009 9:53 AM > To: java-user@lucene.apache.org > Subject: Re: Indexing large files? - No answers yet... > > In this

RE: Indexing large files? - No answers yet...

2009-09-11 Thread Paul_Murdoch
y, September 11, 2009 9:53 AM To: java-user@lucene.apache.org Subject: Re: Indexing large files? - No answers yet... In this project: http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html I concatenate all the text of all of articles of a single journal into a single text

RE: Indexing large files? - No answers yet...

2009-09-11 Thread Paul_Murdoch
009 8:13 AM To: java-user@lucene.apache.org Subject: RE: Indexing large files? - No answers yet... Paul: My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with

Re: Indexing large files? - No answers yet...

2009-09-11 Thread Glen Newton
o look at your index? > If this is a large index, have you optimized it recently? > Are there any searches going on while you are indexing? > > > Regards, > Dan > > > -Original Message- > From: paul_murd...@emainc.com [mailto:paul_murd...@emainc.com] > Sent: Friday,

RE: Indexing large files? - No answers yet...

2009-09-11 Thread Dan OConnor
Subject: RE: Indexing large files? - No answers yet... This issue is still open. Any suggestions/help with this would be greatly appreciated. Thanks, Paul -Original Message- From: java-user-return-42080-paul_murdoch=emainc@lucene.apache.org [mailto:java-user-return-42080-paul_murdoch

RE: Indexing large files? - No answers yet...

2009-09-11 Thread Paul_Murdoch
paul_murd...@emainc.com Sent: Monday, August 31, 2009 10:28 AM To: java-user@lucene.apache.org Subject: Indexing large files? Hi, I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm consistently receiving "OutOfMemoryError: Java heap space", when trying to in

Indexing large files?

2009-08-31 Thread Paul_Murdoch
uld cause the error as it seems correlative. I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best practice for indexing large files? Here is a code snippet that I'm using: // Index the content of a text file. private Boolean saveTXTFile(File text

Re: Indexing large files

2006-12-12 Thread Otis Gospodnetic
Hi, Yes, you can't get to the "stored" field content in your Hits because you are using a (File)Reader. Otis - Original Message From: abdul aleem <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Tuesday, December 12, 2006 8:22:54 AM Subject: Indexing larg

Indexing large files

2006-12-12 Thread abdul aleem
Hi There, I have been working with Lucene API for the past 1 day we are in the process of building a log viewer tool, this is how the log file looks [2006-12-11 01:52:40.179] [lon0571xus] [DEBUG] [TIE heartbeat monitor (monitor.heartbeat.fxstreamrates)] [unknown] [] [] ActiveRateServerIdList -