Re: A codec moment or pickle

2015-02-13 Thread Robert Muir
heh, i just don't think thats the typical case. Its definitely extreme. Even still, in many cases using the filesystem (properly warmed) with compression might still be better. It depends how you are measuring latency. storing your whole index in gigabytes of heap ram without any compression on a

Re: A codec moment or pickle

2015-02-12 Thread Benson Margulies
Based on reading the same comments you read, I'm pretty doubtful that Codec.getDefault() is going to work. It seems to me that this situation renders the FilterCodec a bit hard to to use, at least given the 'every release deprecates a codec' sort of pattern. On Thu, Feb 12, 2015 at 3:20 AM, Uwe

Re: A codec moment or pickle

2015-02-12 Thread Benson Margulies
Robert, Let me lay out the scenario. Hardware has .5T of Index is relatively small. Application profiling shows a significant amount of time spent codec-ing. Options as I see them: 1. Use DPF complete with the irritation of having to have this spurious codec name in the on-disk format that has

Re: A codec moment or pickle

2015-02-12 Thread Benson Margulies
WHOOPS. First sentence was, until just before I clicked 'send', Hardware has .5T of RAM. Index is relatively small (20g) ... On Thu, Feb 12, 2015 at 4:51 PM, Benson Margulies ben...@basistech.com wrote: Robert, Let me lay out the scenario. Hardware has .5T of Index is relatively small.

RE: A codec moment or pickle

2015-02-12 Thread Uwe Schindler
Hi, How about Codec.getDefault()? It does indeed not necessarily return the newest one (if somebody changes the default using Codec.setDefault()), but for your use case wrapping the current default one, it should be fine? I have not tried this yet, but there might be a chicken-egg problem: -

Re: A codec moment or pickle

2015-02-12 Thread Robert Muir
Honestly i dont agree. I don't know what you are trying to do, but if you want file format backwards compat working, then you need a different FilterCodec to match each lucene codec. Otherwise your codec is broken from a back compat standpoint. Wrapping the latest is an antipattern here. On

Re: A codec moment or pickle

2015-02-12 Thread Robert Muir
On Thu, Feb 12, 2015 at 8:51 AM, Benson Margulies ben...@basistech.com wrote: On Thu, Feb 12, 2015 at 8:43 AM, Robert Muir rcm...@gmail.com wrote: Honestly i dont agree. I don't know what you are trying to do, but if you want file format backwards compat working, then you need a different

RE: A codec moment or pickle

2015-02-12 Thread Uwe Schindler
[mailto:ben...@basistech.com] Sent: Thursday, February 12, 2015 11:34 AM To: java-user@lucene apache. org Subject: Re: A codec moment or pickle Based on reading the same comments you read, I'm pretty doubtful that Codec.getDefault() is going to work. It seems to me that this situation renders

Re: A codec moment or pickle

2015-02-12 Thread Benson Margulies
On Thu, Feb 12, 2015 at 8:43 AM, Robert Muir rcm...@gmail.com wrote: Honestly i dont agree. I don't know what you are trying to do, but if you want file format backwards compat working, then you need a different FilterCodec to match each lucene codec. Otherwise your codec is broken from a