Re: Filesystem based bitset

2009-01-19 Thread eks dev
at: http://www.iis.uni-stuttgart.de/intset/ not meant for on disk storage, but the idea is quite similar. cheers, eks From: Paul Elschot paul.elsc...@xs4all.nl To: java-dev@lucene.apache.org Sent: Sunday, 18 January, 2009 23:51:36 Subject: Re: Filesystem based

Re: Filesystem based bitset

2009-01-19 Thread Michael McCandless
Paul Elschot wrote: Since this started by thinking out loud, I'd like to continue doing that. I've been thinking about how to add a decent skipTo() to something that compresses better than an (Open)BitSet, and this turns out to be an integer set implemented as a B plus tree (all leafs on

Re: Filesystem based bitset

2009-01-19 Thread Paul Elschot
On Monday 19 January 2009 11:32:17 Michael McCandless wrote: Paul Elschot wrote: Since this started by thinking out loud, I'd like to continue doing that. I've been thinking about how to add a decent skipTo() to something that compresses better than an (Open)BitSet, and this

Re: Filesystem based bitset

2009-01-10 Thread Grant Ingersoll
On Jan 9, 2009, at 8:06 PM, robert engels wrote: Luckily there are entrepreneurs and other managers/owners that value quality first, and let feelings get repaired over beers or not at all. Sure, but let me ask you, do you like working with those people who are jerks all the time?

Re: Filesystem based bitset

2009-01-10 Thread robert engels
You are completely off-base in regards to my Columbia reference. It is sorrowful when anyone dies (others would dispute this, executions of murderers, etc.), but people die all the time - it doesn't make it a tragedy. What makes the Columbia truly a tragedy is that they died due to

Re: Filesystem based bitset

2009-01-10 Thread robert engels
Also, ideally my coworker would be both. But given that people are of differing ability levels, my coworker has a problem. If he is smarter than me, wasting his time explaining things over and over to me does little good - unless I take the time to learn from it - and that is not always

Re: Filesystem based bitset

2009-01-10 Thread Grant Ingersoll
Robert, if you wish to continue on this list I suggest you stop. Either contribute peacefully and positively to this list or you will be removed. We've all had it with your name calling and constant derision backed up by a complete lack of substance in actually doing any of the work,

Re: Filesystem based bitset

2009-01-10 Thread Yonik Seeley
Can we please let this thread die. -Yonik - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org

Filesystem based bitset

2009-01-09 Thread Karl Wettin
Thinking out loud, SSD is pretty close to RAM when it comes to seeking. Wouldn't that mean that a bitset stored on an SSD would be more or less as fast as a bitset in RAM? So how about storing all permutations of filters one use on SSD? Perhaps loading them to RAM in case they are

Re: Filesystem based bitset

2009-01-09 Thread Michael McCandless
While SSDs are delightfully fast compared to mechanical drives, I think they are still quite a bit slower than RAM for truly random access. EG Intel's X25-E (apparently the leader at the moment) lists a 75us read latency, whereas RAM latency is maybe 50-100 ns. Though since Lucene accesses the

Re: Filesystem based bitset

2009-01-09 Thread Marvin Humphrey
On Fri, Jan 09, 2009 at 08:11:31PM +0100, Karl Wettin wrote: SSD is pretty close to RAM when it comes to seeking. Wouldn't that mean that a bitset stored on an SSD would be more or less as fast as a bitset in RAM? Provided that your index can fit in the system i/o cache and stay there,

Re: Filesystem based bitset

2009-01-09 Thread robert engels
If your index can fit in the IO cache, you should using a completely different implementation... You should be writing a sequential transaction log for add/update/ delete operations, and storing the entire index in memory (RAMDirectory) - with periodic background flushes of the log. If

Re: Filesystem based bitset

2009-01-09 Thread Marvin Humphrey
On Fri, Jan 09, 2009 at 03:42:35PM -0600, robert engels wrote: If your index can fit in the IO cache, you should using a completely different implementation... You should be writing a sequential transaction log for add/update/ delete operations, and storing the entire index in memory

Re: Filesystem based bitset

2009-01-09 Thread robert engels
I have better things to do than read a 10,000 word incident that discusses about 100 different topics under the generic heading "Further steps towards flexible indexing" in order to answer a simple question.You are a moron.  And I don't mean that in a offensive way - I am using the secondary

Re: Filesystem based bitset

2009-01-09 Thread Doug Cutting
robert engels wrote: You are a moron. And I don't mean that in a offensive way - I am using the secondary definition. *2**:* a very stupid person That's still offensive and totally unacceptable here. Please refrain from making ad-hominem remarks and stick to discussing the issues.

Re: Filesystem based bitset

2009-01-09 Thread robert engels
Can something be offensive if its a statement of fact ?  If you believe it is (under definition #3), then his remarks to me were just as offensive - as they caused me much displeasure and resentment. So please dress him down as well.Main Entry: 1of·fen·sive  Pronunciation: \ə-ˈfen(t)-siv,

Re: Filesystem based bitset

2009-01-09 Thread Doug Cutting
robert engels wrote: Can something be offensive if its a statement of fact ? If you believe it is (under definition #3), then his remarks to me were just as offensive - as they caused me much displeasure and resentment. So please dress him down as well. His comments were on-topic. The

Re: Filesystem based bitset

2009-01-09 Thread robert engels
Your exactly right. Playing well with others has trumped actual production and quality. You can see the mess that's gotten us in all sorts of areas. Luckily there are entrepreneurs and other managers/owners that value quality first, and let feelings get repaired over beers or not at all.

Re: Filesystem based bitset

2009-01-09 Thread robert engels
It was not ad hominem. It was a indirect critique of the value of the answer provided. Ad hominem would be if I called him ugly. On Jan 9, 2009, at 6:34 PM, Doug Cutting wrote: robert engels wrote: Can something be offensive if its a statement of fact ? If you believe it is (under