While SSDs are delightfully fast compared to mechanical drives, I
think they are still quite a bit slower than RAM for truly random
access.
EG Intel's X25-E (apparently the leader at the moment) lists a 75us
read latency, whereas RAM latency is maybe 50-100 ns.
Though since Lucene accesses the filter via sequential scan, the read
latency may not matter (unless the file is heavily fragmented, in
which case lots of "seeks" are being done as you scan through it).
And if you have enough RAM, the OS will cache the files in its IO
cache anyway.
I do think as "we" switch over to SSDs, it will change how we optimize
Lucene. EG I think suddenly the CPU cost of searching will matter
much more, the ability to make use of concurrency while searching will
be important, etc.
Mike
Karl Wettin wrote:
Thinking out loud,
SSD is pretty close to RAM when it comes to seeking. Wouldn't that
mean that a bitset stored on an SSD would be more or less as fast as
a bitset in RAM? So how about storing all permutations of filters
one use on SSD? Perhaps loading them to RAM in case they are
frequently used? To me it sounds like a great idea.
Not sure if one should focus at OpenBitSet or a fixed size BitSet,
I'd really need to do some real tests to tell. Still, I'm rather
convinced the bang for the buck ratio is quite a bit more using SSD
than RAM given IO throughput (compare an index in RAM vs on SSD vs
on HDD) isn't an issue.
The only real issue I can this of is the lack of
DocSetIterator#close()..
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
