On Thu, Apr 8, 2010 at 6:21 PM, Earwin Burrfoot wrote:
>> But, IW doesn't let you "hold on to" checkpoints... only to commits.
>>
>> Ie SnapshotDP will only "see" actual commit/close calls, not
>> intermediate checkpoints like a random segment merge completing, a
>> flush happening, etc.
>>
>> Or.
> But, IW doesn't let you "hold on to" checkpoints... only to commits.
>
> Ie SnapshotDP will only "see" actual commit/close calls, not
> intermediate checkpoints like a random segment merge completing, a
> flush happening, etc.
>
> Or... maybe you would in fact call commit frequently from the main
On Wed, Apr 7, 2010 at 3:27 PM, Earwin Burrfoot wrote:
>> No, this doesn't make sense. The OS detects a disk full on accepting
>> the write into the write cache, not [later] on flushing the write
>> cache to disk. If the OS accepts the write, then disk is not full (ie
>> flushing the cache will
> No, this doesn't make sense. The OS detects a disk full on accepting
> the write into the write cache, not [later] on flushing the write
> cache to disk. If the OS accepts the write, then disk is not full (ie
> flushing the cache will succeed, unless some other not-disk-full
> problem happens).
I don't have the system at hand now, but if I remember right fsync
took like 100-200ms.
2010/4/7 Shai Erera :
> Earwin - do you have some numbers to share on the running time of the
> indexing application? You've mentioned that if you take out fsync into a BG
> thread, the running time improves, b
On Tue, Apr 6, 2010 at 7:26 PM, Earwin Burrfoot wrote:
>> Running out of disk space with fsync disabled won't lead to corruption.
>> Even kill -9 the JRE process with fsync disabled won't corrupt.
>> In these cases index just falls back to last successful commit.
>>
>> It's "only" power loss / OS
Earwin - do you have some numbers to share on the running time of the
indexing application? You've mentioned that if you take out fsync into a BG
thread, the running time improves, but I'm curious to know by how much.
Shai
On Wed, Apr 7, 2010 at 2:26 AM, Earwin Burrfoot wrote:
> > Running out o
> Running out of disk space with fsync disabled won't lead to corruption.
> Even kill -9 the JRE process with fsync disabled won't corrupt.
> In these cases index just falls back to last successful commit.
>
> It's "only" power loss / OS / machine crash where you need fsync to
> avoid possible corr
On Tue, Apr 6, 2010 at 10:11 AM, Earwin Burrfoot wrote:
> So, I want to pump my IndexWriter hard and fast with documents.
Nice.
> Removing fsync from FSDirectory helps. But for that I pay with possibility of
> index corruption, not only if my node suddenly loses
> power/kernelpanics, but also i
How often is fsync called? If it's just during calls to commit, then is that
that expensive? I mean, how often do you call commit?
If that's that expensive (do you have some numbers to share) then I think
that's be a neat idea. Though "losing a few minutes worth of updates" may
sometimes be unreco
So, I want to pump my IndexWriter hard and fast with documents.
Removing fsync from FSDirectory helps. But for that I pay with possibility of
index corruption, not only if my node suddenly loses
power/kernelpanics, but also if it
runs out of disk space (which happens more frequently).
I invented
11 matches
Mail list logo
