Re: about increment update
Thank you Ian, it works. Kerr.
- Original Message -
From: "Ian Lea" <[EMAIL PROTECTED]>
To: "kerr" <[EMAIL PROTECTED]>
Cc: "Lucene Users List" <[EMAIL PROTECTED]>
Sent: Friday, April 04, 2003 4:55 AM
Subject: Re: about increment update
> Try this:
>
> 1. Open reader.
> 2. removeModifiedFiles(reader)
> 3. reader.close()
> 4. Open writer.
> 5. updateIndexDocs()
> 6. writer.close();
>
> i.e. don't have both reader and writer open at the same time.
>
> btw I suspect you might be removing index entries for files that
> have been modified, but adding all files. Another "index keeps
> growing" problem! Could be wrong.
>
>
> --
> Ian.
>
> > [EMAIL PROTECTED] (kerr) wrote
> >
> > Thank you Otis,
> > Yes, reader should be closed. But it isn't the reason of this Exception.
> > the errors happen before deleting file.
> >Kerr.
> > close()
> > Closes files associated with this index. Also saves any new deletions to disk. No
> > other methods should be called after this has been called.
> >
> > ----- Original Message -
> > From: "Otis Gospodnetic" <[EMAIL PROTECTED]>
> > To: "Lucene Users List" <[EMAIL PROTECTED]>
> > Sent: Thursday, April 03, 2003 12:14 PM
> > Subject: Re: about increment update
> >
> >
> > > Maybe this is missing?
> > > http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReader.html#close()
> > >
> > > Otis
> > >
> > > --- kerr <[EMAIL PROTECTED]> wrote:
> > > > Hello everyone,
> > > > Here I try to increment update index file and follow the idea to
> > > > delete modified file first and re-add it. Here is the source.
> > > > But when I execute it, the index directory create a file(write.lock)
> > > > when execute the line
> > > > reader.delete(i);,
> > > > and caught a class java.io.IOException with message: Index locked
> > > > for write.
> > > > After that, when I execute the line
> > > > IndexWriter writer = new IndexWriter("index", new
> > > > StandardAnalyzer(), false);
> > > > caught a class java.io.IOException with message: Index locked for
> > > > write
> > > > if I delete the file(write.lock), the error will re-happen.
> > > > anyone can help and thanks.
> > > >Kerr.
> > > >
> > > >
> > > > import org.apache.lucene.analysis.standard.StandardAnalyzer;
> > > > import org.apache.lucene.index.IndexWriter;
> > > > import org.apache.lucene.document.Document;
> > > > import org.apache.lucene.document.Field;
> > > > import org.apache.lucene.store.Directory;
> > > > import org.apache.lucene.store.FSDirectory;
> > > > import org.apache.lucene.index.IndexReader;
> > > > import org.apache.lucene.index.Term;
> > > >
> > > > import java.io.File;
> > > > import java.util.Date;
> > > >
> > > >
> > > > public class UpdateIndexFiles {
> > > > public static void main(String[] args) {
> > > > try {
> > > > Date start = new Date();
> > > >
> > > > Directory directory = FSDirectory.getDirectory("index", false);
> > > > IndexReader reader = IndexReader.open(directory);
> > > > System.out.println(reader.isLocked(directory));
> > > > //reader.unlock(directory);
> > > > IndexWriter writer = new IndexWriter("index", new
> > > > StandardAnalyzer(), false);
> > > >
> > > > String base = "";
> > > > if (args.length == 0){
> > > > base = "D:\\Tomcat\\webapps\\ROOT\\test";
> > > > } else {
> > > > base = args[0];
> > > > }
> > > > removeModifiedFiles(reader);
> > > > updateIndexDocs(reader, writer, new File(base));
> > > >
> > > > writer.optimize();
> > > > writer.close();
> > > >
> > > > Date end = new Date();
> > > >
> > > > System.out.print(end.getTime() - start.getTime());
> > > > System.out.println(" total milliseconds");
> > > >
> > > > } catch (Exception e) {
> > > >
Re: about increment update
Try this:
1. Open reader.
2. removeModifiedFiles(reader)
3. reader.close()
4. Open writer.
5. updateIndexDocs()
6. writer.close();
i.e. don't have both reader and writer open at the same time.
btw I suspect you might be removing index entries for files that
have been modified, but adding all files. Another "index keeps
growing" problem! Could be wrong.
--
Ian.
> [EMAIL PROTECTED] (kerr) wrote
>
> Thank you Otis,
> Yes, reader should be closed. But it isn't the reason of this Exception.
> the errors happen before deleting file.
>Kerr.
> close()
> Closes files associated with this index. Also saves any new deletions to disk. No
> other methods should be called after this has been called.
>
> - Original Message -
> From: "Otis Gospodnetic" <[EMAIL PROTECTED]>
> To: "Lucene Users List" <[EMAIL PROTECTED]>
> Sent: Thursday, April 03, 2003 12:14 PM
> Subject: Re: about increment update
>
>
> > Maybe this is missing?
> > http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReader.html#close()
> >
> > Otis
> >
> > --- kerr <[EMAIL PROTECTED]> wrote:
> > > Hello everyone,
> > > Here I try to increment update index file and follow the idea to
> > > delete modified file first and re-add it. Here is the source.
> > > But when I execute it, the index directory create a file(write.lock)
> > > when execute the line
> > > reader.delete(i);,
> > > and caught a class java.io.IOException with message: Index locked
> > > for write.
> > > After that, when I execute the line
> > > IndexWriter writer = new IndexWriter("index", new
> > > StandardAnalyzer(), false);
> > > caught a class java.io.IOException with message: Index locked for
> > > write
> > > if I delete the file(write.lock), the error will re-happen.
> > > anyone can help and thanks.
> > >Kerr.
> > >
> > >
> > > import org.apache.lucene.analysis.standard.StandardAnalyzer;
> > > import org.apache.lucene.index.IndexWriter;
> > > import org.apache.lucene.document.Document;
> > > import org.apache.lucene.document.Field;
> > > import org.apache.lucene.store.Directory;
> > > import org.apache.lucene.store.FSDirectory;
> > > import org.apache.lucene.index.IndexReader;
> > > import org.apache.lucene.index.Term;
> > >
> > > import java.io.File;
> > > import java.util.Date;
> > >
> > >
> > > public class UpdateIndexFiles {
> > > public static void main(String[] args) {
> > > try {
> > > Date start = new Date();
> > >
> > > Directory directory = FSDirectory.getDirectory("index", false);
> > > IndexReader reader = IndexReader.open(directory);
> > > System.out.println(reader.isLocked(directory));
> > > //reader.unlock(directory);
> > > IndexWriter writer = new IndexWriter("index", new
> > > StandardAnalyzer(), false);
> > >
> > > String base = "";
> > > if (args.length == 0){
> > > base = "D:\\Tomcat\\webapps\\ROOT\\test";
> > > } else {
> > > base = args[0];
> > > }
> > > removeModifiedFiles(reader);
> > > updateIndexDocs(reader, writer, new File(base));
> > >
> > > writer.optimize();
> > > writer.close();
> > >
> > > Date end = new Date();
> > >
> > > System.out.print(end.getTime() - start.getTime());
> > > System.out.println(" total milliseconds");
> > >
> > > } catch (Exception e) {
> > > System.out.println(" caught a " + e.getClass() +
> > >"\n with message: " + e.getMessage());
> > > e.printStackTrace();
> > > }
> > > }
> > >
> > > public static void removeModifiedFiles(IndexReader reader) throws
> > > Exception {
> > > Document adoc;
> > > String path;
> > > File aFile;
> > > for (int i=0; i > > adoc = reader.document(i);
> > > path = adoc.get("path");
> > > aFile = new File(path);
> > > if (reader.lastModified(path) < aFile.lastModified()){
> > > System.out.println(reader.isLocked(path));
> > > reader.delete(i);
> > > }
> > > }
> > > }
> > >
> > > public static void updateIndexDocs(IndexReader reader, IndexWriter
> > > writer, File file)
> > >throws Exception {
> > >
> > > if (file.isDirectory()) {
> > > String[] files = file.list();
> > > for (int i = 0; i < files.length; i++)
> > > updateIndexDocs(reader, writer, new File(file, files[i]));
> > > } else {
> > > if (!reader.indexExists(file)){
> > > System.out.println("adding " + file);
> > > writer.addDocument(FileDocument.Document(file));
> > > } else {}
> > > }
> > > }
> > > }
--
Searchable personal storage and archiving from http://www.digimem.net/
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: about increment update
Thank you Otis,
Yes, reader should be closed. But it isn't the reason of this Exception.
the errors happen before deleting file.
Kerr.
close()
Closes files associated with this index. Also saves any new deletions to disk. No
other methods should be called after this has been called.
- Original Message -
From: "Otis Gospodnetic" <[EMAIL PROTECTED]>
To: "Lucene Users List" <[EMAIL PROTECTED]>
Sent: Thursday, April 03, 2003 12:14 PM
Subject: Re: about increment update
> Maybe this is missing?
> http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReader.html#close()
>
> Otis
>
> --- kerr <[EMAIL PROTECTED]> wrote:
> > Hello everyone,
> > Here I try to increment update index file and follow the idea to
> > delete modified file first and re-add it. Here is the source.
> > But when I execute it, the index directory create a file(write.lock)
> > when execute the line
> > reader.delete(i);,
> > and caught a class java.io.IOException with message: Index locked
> > for write.
> > After that, when I execute the line
> > IndexWriter writer = new IndexWriter("index", new
> > StandardAnalyzer(), false);
> > caught a class java.io.IOException with message: Index locked for
> > write
> > if I delete the file(write.lock), the error will re-happen.
> > anyone can help and thanks.
> >Kerr.
> >
> >
> > import org.apache.lucene.analysis.standard.StandardAnalyzer;
> > import org.apache.lucene.index.IndexWriter;
> > import org.apache.lucene.document.Document;
> > import org.apache.lucene.document.Field;
> > import org.apache.lucene.store.Directory;
> > import org.apache.lucene.store.FSDirectory;
> > import org.apache.lucene.index.IndexReader;
> > import org.apache.lucene.index.Term;
> >
> > import java.io.File;
> > import java.util.Date;
> >
> >
> > public class UpdateIndexFiles {
> > public static void main(String[] args) {
> > try {
> > Date start = new Date();
> >
> > Directory directory = FSDirectory.getDirectory("index", false);
> > IndexReader reader = IndexReader.open(directory);
> > System.out.println(reader.isLocked(directory));
> > //reader.unlock(directory);
> > IndexWriter writer = new IndexWriter("index", new
> > StandardAnalyzer(), false);
> >
> > String base = "";
> > if (args.length == 0){
> > base = "D:\\Tomcat\\webapps\\ROOT\\test";
> > } else {
> > base = args[0];
> > }
> > removeModifiedFiles(reader);
> > updateIndexDocs(reader, writer, new File(base));
> >
> > writer.optimize();
> > writer.close();
> >
> > Date end = new Date();
> >
> > System.out.print(end.getTime() - start.getTime());
> > System.out.println(" total milliseconds");
> >
> > } catch (Exception e) {
> > System.out.println(" caught a " + e.getClass() +
> >"\n with message: " + e.getMessage());
> > e.printStackTrace();
> > }
> > }
> >
> > public static void removeModifiedFiles(IndexReader reader) throws
> > Exception {
> > Document adoc;
> > String path;
> > File aFile;
> > for (int i=0; i > adoc = reader.document(i);
> > path = adoc.get("path");
> > aFile = new File(path);
> > if (reader.lastModified(path) < aFile.lastModified()){
> > System.out.println(reader.isLocked(path));
> > reader.delete(i);
> > }
> > }
> > }
> >
> > public static void updateIndexDocs(IndexReader reader, IndexWriter
> > writer, File file)
> >throws Exception {
> >
> > if (file.isDirectory()) {
> > String[] files = file.list();
> > for (int i = 0; i < files.length; i++)
> > updateIndexDocs(reader, writer, new File(file, files[i]));
> > } else {
> > if (!reader.indexExists(file)){
> > System.out.println("adding " + file);
> > writer.addDocument(FileDocument.Document(file));
> > } else {}
> > }
> > }
> > }
>
>
> __
> Do you Yahoo!?
> Yahoo! Tax Center - File online, calculators, forms, and more
> http://tax.yahoo.com
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
Re: about increment update
Maybe this is missing?
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReader.html#close()
Otis
--- kerr <[EMAIL PROTECTED]> wrote:
> Hello everyone,
> Here I try to increment update index file and follow the idea to
> delete modified file first and re-add it. Here is the source.
> But when I execute it, the index directory create a file(write.lock)
> when execute the line
> reader.delete(i);,
> and caught a class java.io.IOException with message: Index locked
> for write.
> After that, when I execute the line
> IndexWriter writer = new IndexWriter("index", new
> StandardAnalyzer(), false);
> caught a class java.io.IOException with message: Index locked for
> write
> if I delete the file(write.lock), the error will re-happen.
> anyone can help and thanks.
>Kerr.
>
>
> import org.apache.lucene.analysis.standard.StandardAnalyzer;
> import org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.document.Document;
> import org.apache.lucene.document.Field;
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.FSDirectory;
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.index.Term;
>
> import java.io.File;
> import java.util.Date;
>
>
> public class UpdateIndexFiles {
> public static void main(String[] args) {
> try {
> Date start = new Date();
>
> Directory directory = FSDirectory.getDirectory("index", false);
> IndexReader reader = IndexReader.open(directory);
> System.out.println(reader.isLocked(directory));
> //reader.unlock(directory);
> IndexWriter writer = new IndexWriter("index", new
> StandardAnalyzer(), false);
>
> String base = "";
> if (args.length == 0){
> base = "D:\\Tomcat\\webapps\\ROOT\\test";
> } else {
> base = args[0];
> }
> removeModifiedFiles(reader);
> updateIndexDocs(reader, writer, new File(base));
>
> writer.optimize();
> writer.close();
>
> Date end = new Date();
>
> System.out.print(end.getTime() - start.getTime());
> System.out.println(" total milliseconds");
>
> } catch (Exception e) {
> System.out.println(" caught a " + e.getClass() +
>"\n with message: " + e.getMessage());
> e.printStackTrace();
> }
> }
>
> public static void removeModifiedFiles(IndexReader reader) throws
> Exception {
> Document adoc;
> String path;
> File aFile;
> for (int i=0; i adoc = reader.document(i);
> path = adoc.get("path");
> aFile = new File(path);
> if (reader.lastModified(path) < aFile.lastModified()){
> System.out.println(reader.isLocked(path));
> reader.delete(i);
> }
> }
> }
>
> public static void updateIndexDocs(IndexReader reader, IndexWriter
> writer, File file)
>throws Exception {
>
> if (file.isDirectory()) {
> String[] files = file.list();
> for (int i = 0; i < files.length; i++)
> updateIndexDocs(reader, writer, new File(file, files[i]));
> } else {
> if (!reader.indexExists(file)){
> System.out.println("adding " + file);
> writer.addDocument(FileDocument.Document(file));
> } else {}
> }
> }
> }
__
Do you Yahoo!?
Yahoo! Tax Center - File online, calculators, forms, and more
http://tax.yahoo.com
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
about increment update
Hello everyone,
Here I try to increment update index file and follow the idea to delete modified file
first and re-add it. Here is the source.
But when I execute it, the index directory create a file(write.lock) when execute the
line
reader.delete(i);,
and caught a class java.io.IOException with message: Index locked for write.
After that, when I execute the line
IndexWriter writer = new IndexWriter("index", new StandardAnalyzer(), false);
caught a class java.io.IOException with message: Index locked for write
if I delete the file(write.lock), the error will re-happen.
anyone can help and thanks.
Kerr.
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.Term;
import java.io.File;
import java.util.Date;
public class UpdateIndexFiles {
public static void main(String[] args) {
try {
Date start = new Date();
Directory directory = FSDirectory.getDirectory("index", false);
IndexReader reader = IndexReader.open(directory);
System.out.println(reader.isLocked(directory));
//reader.unlock(directory);
IndexWriter writer = new IndexWriter("index", new StandardAnalyzer(), false);
String base = "";
if (args.length == 0){
base = "D:\\Tomcat\\webapps\\ROOT\\test";
} else {
base = args[0];
}
removeModifiedFiles(reader);
updateIndexDocs(reader, writer, new File(base));
writer.optimize();
writer.close();
Date end = new Date();
System.out.print(end.getTime() - start.getTime());
System.out.println(" total milliseconds");
} catch (Exception e) {
System.out.println(" caught a " + e.getClass() +
"\n with message: " + e.getMessage());
e.printStackTrace();
}
}
public static void removeModifiedFiles(IndexReader reader) throws Exception {
Document adoc;
String path;
File aFile;
for (int i=0; i
