Re: Binary fields and data compression

2004-08-30 Thread Andrzej Bialecki
Robert Engels wrote: My estimates are based on our own projects where we see that adding a DeflatorInputStream around an InputStream takes about 20% of the CPU time, so whether to actually use it or not will depend on if the IndexReader is performance bound by the CPU or IO. The problem with the "a

RE: Binary fields and data compression

2004-08-30 Thread Otis Gospodnetic
--- Robert Engels <[EMAIL PROTECTED]> wrote: .. > ... thus my request that any compression support be optional. I think this goes without say. Say say say... Otis > -Original Message- > From: David Spencer [mailto:[EMAIL PROTECTED] > Sent: Monday, August 30, 2004 5:33 PM > To: L

RE: Binary fields and data compression

2004-08-30 Thread Robert Engels
My estimates are based on our own projects where we see that adding a DeflatorInputStream around an InputStream takes about 20% of the CPU time, so whether to actually use it or not will depend on if the IndexReader is performance bound by the CPU or IO. The problem with the "after the read" decom

Re: Binary fields and data compression

2004-08-30 Thread David Spencer
Robert Engels wrote: The data size savings is almost certainly not worth the probable 20-40% increase in CPU usage in most cases no? I think it should be optional for those who have extremely large indices and want to save some space (seems not necessary these days), and those who want to maximize

Re: API cleanup for Field

2004-08-30 Thread Daniel Naber
On Monday 30 August 2004 18:34, Doug Cutting wrote: > If they're confusing and have a less-confusing alternative then we > should eventually remove them from the API, so we should deprecate them > now. ÂWe should move entirely to the new enumeration-based contructors. > Â Everything else should be

Re: Binary fields and data compression

2004-08-30 Thread Otis Gospodnetic
Bernhard, Sounds good to me. I would, however, also be interested in the performance impact of text-field compression. While adapting Drew's patch, it may be nice to make the compression mechanism pluggable. Otis --- Bernhard Messer <[EMAIL PROTECTED]> wrote: > hi developers, > > a few month

RE: Binary fields and data compression

2004-08-30 Thread Robert Engels
The data size savings is almost certainly not worth the probable 20-40% increase in CPU usage in most cases no? I think it should be optional for those who have extremely large indices and want to save some space (seems not necessary these days), and those who want to maximize performance. -

Binary fields and data compression

2004-08-30 Thread Bernhard Messer
hi developers, a few month ago, there was a very interesting discussion about field compression and the possibility to store binary field values within a lucene document. Regarding to this topic, Drew Farris came up with a patch to add the necessary functionality. I ran all the necessary tests

[Jakarta Lucene Wiki] Updated: Lucene2Whiteboard

2004-08-30 Thread lucene-cvs
Date: 2004-08-30T14:17:50 Editor: DougCutting <[EMAIL PROTECTED]> Wiki: Jakarta Lucene Wiki Page: Lucene2Whiteboard URL: http://wiki.apache.org/jakarta-lucene/Lucene2Whiteboard no comment Change Log: --

[Jakarta Lucene Wiki] Updated: Lucene2Whiteboard

2004-08-30 Thread lucene-cvs
Date: 2004-08-30T14:06:15 Editor: DanielNaber <[EMAIL PROTECTED]> Wiki: Jakarta Lucene Wiki Page: Lucene2Whiteboard URL: http://wiki.apache.org/jakarta-lucene/Lucene2Whiteboard no comment Change Log: --

cvs commit: jakarta-lucene/src/test/org/apache/lucene/document TestDocument.java

2004-08-30 Thread dnaber
dnaber 2004/08/30 14:03:40 Modified:src/test/org/apache/lucene/document TestDocument.java Log: add test for runtime exceptions Revision ChangesPath 1.6 +20 -1 jakarta-lucene/src/test/org/apache/lucene/document/TestDocument.java Index: TestDocument.java

cvs commit: jakarta-lucene/src/test/org/apache/lucene/document TestDocument.java

2004-08-30 Thread dnaber
dnaber 2004/08/30 13:56:04 Modified:src/test/org/apache/lucene/document TestDocument.java Log: stop using the deprecated Field constructor; small doc cleanup to avoid warnings Revision ChangesPath 1.5 +7 -9 jakarta-lucene/src/test/org/apache/lucene/document/T

cvs commit: jakarta-lucene/src/test/org/apache/lucene/search TestSort.java TestPhraseQuery.java

2004-08-30 Thread dnaber
dnaber 2004/08/30 13:52:16 Modified:src/test/org/apache/lucene/search TestSort.java TestPhraseQuery.java Log: stop using the deprecated Field constructor Revision ChangesPath 1.8 +8 -8 jakarta-lucene/src/test/org/apache/lucene/search/Te

cvs commit: jakarta-lucene/src/test/org/apache/lucene/search TestBooleanPrefixQuery.java

2004-08-30 Thread dnaber
dnaber 2004/08/30 13:49:34 Modified:src/test/org/apache/lucene/search TestBooleanPrefixQuery.java Log: don't use the deprecated API for Booleanquery.add() anymore Revision ChangesPath 1.3 +2 -2 jakarta-lucene/src/test/org/apache/lucene

cvs commit: jakarta-lucene/src/java/org/apache/lucene/document Field.java

2004-08-30 Thread dnaber
dnaber 2004/08/30 13:48:22 Modified:src/java/org/apache/lucene/document Field.java Log: throw NullPointerException instead of IllegalArgumentException if appropriate Revision ChangesPath 1.18 +6 -6 jakarta-lucene/src/java/org/apache/lucene/document/Field.java

cvs commit: jakarta-lucene/src/java/org/apache/lucene/document Field.java

2004-08-30 Thread dnaber
dnaber 2004/08/30 13:45:02 Modified:src/java/org/apache/lucene/document Field.java Log: deprecate the constructor that uses three resp. four booleans, use two resp. three enumerations instead Revision ChangesPath 1.17 +136 -3jakarta-lucene/src/java/org/apache/l