[jira] Created: (LUCENE-2299) if you open an NRT reader while addIndexes* is running it may miss segments

if you open an NRT reader while addIndexes* is running it may miss segments --- Key: LUCENE-2299 URL: https://issues.apache.org/jira/browse/LUCENE-2299 Project: Lucene - Java

Re: addIndexes* blocks addDocuments calls

On Tue, Jul 21, 2009 at 7:26 PM, Jason Rutherglen wrote: >> EG you could imagine an addIndexes* call getting started, >> completing a few merges. Then, concurrently, CMS picks & chooses >> some of those added external segments to merge with some of the >> original segme

Re: addIndexes* blocks addDocuments calls

> EG you could imagine an addIndexes* call getting started, > completing a few merges. Then, concurrently, CMS picks & chooses > some of those added external segments to merge with some of the > original segments. Then addIndexes hits an exception. What do we > do? An exception

Re: addIndexes* blocks addDocuments calls

> EG you could imagine an addIndexes* call getting started, completing a few merges. Then, concurrently, CMS picks & chooses some of those added external segments to merge with some of the original segments. Then addIndexes hits an exception. What do we do? Right because we're roll

Re: addIndexes* blocks addDocuments calls

I agree, there's no real reason why addIndexes can run concurrently with other things. It's just software ;) One challenge is the transactional guarantee that addIndexes provide, ie, it's all or none. If there's an exception while adding, then nothing was added. But, t

addIndexes* blocks addDocuments calls

For replicating and general system performance, it would be good to offer a way to addIndexes* without blocking the addition of more docs. This seems doable somehow?

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

0537 > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 > Project: Lucene - Java

[jira] Resolved: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

[ https://issues.apache.org/jira/browse/LUCENE-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-1335. Resolution: Fixed > Correctly handle concurrent calls to addIndexes, optim

Re: [jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

> > Michael McCandless commented on LUCENE-1335: > > > Ning (or anyone), any more feedback on this one? Else I plan to commit > soon... > >> Correctly handle

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

more feedback on this one? Else I plan to commit soon... > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

it is replaced with the "real" wait(), to catch any missing notifyAll()'s. > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

closer look at and possibly simplify the concurrency control in IndexWriter, especially when autoCommit is disabled? > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-13

[jira] Updated: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

erges were already in progress in CMS. This way the disk full error is hit in the FG and the transaction (in addIndexesNoOptimize) unwinds. > Correctly handle concurrent calls to addIndexes, optimize, commit > - > >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

- we can run all merges in the BG. In fact I think it's no longer necessary to even compute & record isExternal (this was its only use). Hmmm, except when I take this out I'm seeing testAddIndexOnDiskFull hang. I'll dig. > Correctly handle con

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

this difference? > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 > Proj

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

to run concurrently? {quote} > This is to make sure any just-started addIndexes cleanly finish or > abort before we enter the wait loop. I was seeing cases where the > wait loop would think no more merges were pending, but in fact an > addIndexes was just getting underway and was abou

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

t() from multiple threads at the same > time. Is it better to simplify things by serializing all commit()/commit(long) calls? > This is to make sure any just-started addIndexes cleanly finish or > abort before we enter the wait loop. I was seeing cases where the > wait loop would think

[jira] Updated: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

Ning!). > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

lose(waitForMerges=false) and rollback() make an ongoing addIndexes[NoOptimize](dirs) abort, but wait for addIndexes(readers) to finish. It'd be nice if they make any addIndexes* abort for a quick shutdown, but that's for later. {quote} True, agreed. {quote} commit() and commit(long) use

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

bout what it means when the methods (commit, close, rollback, optimize, addIndexes) are called concurrently from different threads. The discussion below is on their current behavior. > Only one addIndexes can run at once, so call to 2nd or more > addIndexes just blocks until the one is done

[jira] Updated: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

tly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 > Project: Lucene - Java >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

y it. Thanks! > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

day or two. > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

this patch? > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 >

[jira] Updated: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

& copyExternalSegments. > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

;ll add a comment there, and any other places where I call doWait that look similarly confusing... > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

very welcome. > Correctly handle concurrent calls to addIndexes, optimize, commit > - > > Key: LUCENE-1335 > URL: https://issues.apache.org/jira/browse/LUCENE-1335 >

[jira] Commented: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

API promises about what it means when these methods (commit, close, rollback, optimize, addIndexes) are called concurrently from different threads, except that the methods all work correctly, IndexWriter won't throw an errant exception, and your index won't become corrupt.{quote} Agree..

[jira] Updated: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

y one addIndexes can run at once, so call to 2nd or more addIndexes just blocks until the one is done. * close() and rollback() wait for any running addIndexes to finish and then blocks new addIndexes calls * commit() waits for any running addIndexes, or any already running commi

[jira] Created: (LUCENE-1335) Correctly handle concurrent calls to addIndexes, optimize, commit

Correctly handle concurrent calls to addIndexes, optimize, commit - Key: LUCENE-1335 URL: https://issues.apache.org/jira/browse/LUCENE-1335 Project: Lucene - Java Issue Type

Re: Commit while addIndexes is in progress

Ning Li wrote: I think there're similar problems with calling optimize() while addIndexes is in progress... I think we should disallow that? Optimize waits for addIndexes to finish? I think it's useful to allow addIndexes during maybeMerge and optimize, no? OK I agree it wou

Re: Commit while addIndexes is in progress

Yonik Seeley wrote: On Fri, Jul 11, 2008 at 2:38 PM, Michael McCandless <[EMAIL PROTECTED]> wrote: Hmm, I think we should. What should it "mean" when you call commit(), while another thread is in the middle of addIndexes? Seems like either all or none of the segmen

Re: Commit while addIndexes is in progress

Yonik Seeley wrote: On Fri, Jul 11, 2008 at 3:27 PM, Ning Li <[EMAIL PROTECTED]> wrote: We should also disallow concurrent addIndexes, right? Hmmm, the current implementation looks like it won't currently won't work correctly (docWriter.resumeAllThreads() being called whil

Re: Commit while addIndexes is in progress

On Fri, Jul 11, 2008 at 3:27 PM, Ning Li <[EMAIL PROTECTED]> wrote: > We should also disallow concurrent addIndexes, right? Hmmm, the current implementation looks like it won't currently won't work correctly (docWriter.resumeAllThreads() being called while another thread is

Re: Commit while addIndexes is in progress

On Fri, Jul 11, 2008 at 2:38 PM, Michael McCandless <[EMAIL PROTECTED]> wrote: > > Hmm, I think we should. > > What should it "mean" when you call commit(), while another thread is in the > middle of addIndexes? Seems like either all or none of the segments in

Re: Commit while addIndexes is in progress

> What should it "mean" when you call commit(), while another thread is in the > middle of addIndexes? > > We could 1) disallow it (throw an exception if you try), 2) allow it but > block until addIndexes is done, 3) allow it but commit all changes up until > when

Re: Commit while addIndexes is in progress

Hmm, I think we should. What should it "mean" when you call commit(), while another thread is in the middle of addIndexes? We could 1) disallow it (throw an exception if you try), 2) allow it but block until addIndexes is done, 3) allow it but commit all changes up until when

Commit while addIndexes is in progress

Hi, Should we guard against the case when commit() is called during addIndexes? Otherwise, errors such as a file does not exist could happen during commit. Cheers, Ning Li - To unsubscribe, e-mail: [EMAIL PROTECTED] For

RE: addIndexes()

On Thu, 31 May 2007, Steven Parkes wrote: Hmmm ... something's not meshing for me here. If I understood what you've said, you have a DbD index to which you are addIndexes'ing a memory index? I must have missed something, because addIndexes pre- and post-optimizes the target (

RE: addIndexes()

Hmmm ... something's not meshing for me here. If I understood what you've said, you have a DbD index to which you are addIndexes'ing a memory index? I must have missed something, because addIndexes pre- and post-optimizes the target (Dbd) index, not the operand (mem) index. -O

Re: addIndexes()

On Thu, 31 May 2007, Doug Cutting wrote: Steven Parkes wrote: Is there any particular reason that the version that takes a Directory[] optimizes first? There was, but unfortunately I can't recall it now. Index merging has changed substantially since then, so, whatever it was, it may no lon

Re: addIndexes()

Steven Parkes wrote: Is there any particular reason that the version that takes a Directory[] optimizes first? There was, but unfortunately I can't recall it now. Index merging has changed substantially since then, so, whatever it was, it may no longer apply. If no one can think of a good r

addIndexes()

I'm cleaning up the patch for LUCENE-847 (factored merge policy) and noticed a couple of things about the addIndexes methods. Is there any particular reason that the version that takes a Directory[] optimizes first? The later merge is going to use the normal logarithmic stepping; is th

[jira] Closed: (LUCENE-592) Create compound file after addIndexes but before rewrite of segments

with SegmentInfo.isCompoundFile == true. So even if the writing of the cfs file fails for some reason and there is a corrupted cfs file on disk, it is not referenced by the segments file and thus an IndexReader/writer would not try to open it. > Create compound file after addIndexes but before

[jira] Resolved: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

[ http://issues.apache.org/jira/browse/LUCENE-702?page=all ] Michael McCandless resolved LUCENE-702. --- Fix Version/s: 2.1 Resolution: Fixed > Disk full during addIndexes(Directory[]) can corrupt in

[jira] Updated: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

ritten files (since no segments_N references them). > Disk full during addIndexes(Directory[]) can corrupt index > -- > > Key: LUCENE-702 > URL: http://issues.apache.org/jira/browse/LUCENE-702

[jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

se write to the same segment_N filename, right? The "write once" semantics is not kept for segment names or .delN. This is ok because no reader will read the old versions. > Disk full during addIndexes(Directory[]) can corrupt index >

[jira] Updated: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

call in optimize (as described above). * Fixed javadocs for addIndexes(*): we actually require up to 2X the total input size of all indices. Fixed unit test to assert this. * Fixed javadocs in IndexWriter's optimize, addIndexes(*), addDocument to describe disk usage and index stat

[jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

ent not in compound format (I will call this out in the javadocs). OK I've added a unit-test that explicitly tests max temp space required by just optimize and asserts that it's at most 1X starting index size. Will attach a patch shortly! > Disk full during addIndexes(Directory

[jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

ses. > Disk full during addIndexes(Directory[]) can corrupt index > -- > > Key: LUCENE-702 > URL: http://issues.apache.org/jira/browse/LUCENE-702 > Project: Lucene - Java >

[jira] Updated: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

[ http://issues.apache.org/jira/browse/LUCENE-702?page=all ] Michael McCandless updated LUCENE-702: -- Lucene Fields: [New, Patch Available] (was: [New]) > Disk full during addIndexes(Directory[]) can corrupt in

[jira] Updated: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

unit tests pass. This patch fixes addIndexes to 1) not corrupt the index on exception, 2) be transactional (either all or nothing was actually added), and 3) to leave the writer instance in an consistent state (meaning, on an exception, the segmentInfos state is restored to its starting point, so th

[jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

[ http://issues.apache.org/jira/browse/LUCENE-702?page=comments#action_12448006 ] Ning Li commented on LUCENE-702: > I think we should try to make all of the addIndexes calls (and more > generally any call to Lucene) "transactional&

[jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

[ http://issues.apache.org/jira/browse/LUCENE-702?page=comments#action_12447968 ] Michael McCandless commented on LUCENE-702: --- I think we should try to make all of the addIndexes calls (and more generally any call to Lucene

[jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

info file) those segments that were carried over from the input readers/directories? This would make the operation transactional, so that if we crashed part way through, then the index rolls back to where it was before the addIndexes call. And user could free up disk space and try again, without cre

[jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

a segment info is d, and only write if it is. Suggestions? The following is my comment on this issue from the mailing list documenting how Lucene could produce an inconsistent index if addIndexes(Directory[]) does not run to its completion. "This makes me notice a bug in current addIn

[jira] Created: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index

Disk full during addIndexes(Directory[]) can corrupt index -- Key: LUCENE-702 URL: http://issues.apache.org/jira/browse/LUCENE-702 Project: Lucene - Java Issue Type: Bug

[jira] Updated: (LUCENE-592) Create compound file after addIndexes but before rewrite of segments

smthg is wrong. > Create compound file after addIndexes but before rewrite of segments > > > Key: LUCENE-592 > URL: http://issues.apache.org/jira/browse/LUCENE-592 > Project: Lucene - J

[jira] Created: (LUCENE-592) Create compound file after addIndexes but before rewrite of segments

Create compound file after addIndexes but before rewrite of segments - Key: LUCENE-592 URL: http://issues.apache.org/jira/browse/LUCENE-592 Project: Lucene - Java Type: Improvement Components

DO NOT REPLY [Bug 28339] - addIndexes unexpectedly closes index

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG· RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT . ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND· INSERTED IN THE BUG DATABASE. http://issues.apache.org/bugzilla/show_bu

DO NOT REPLY [Bug 28339] - addIndexes unexpectedly closes index

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG· RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT . ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND· INSERTED IN THE BUG DATABASE. http://issues.apache.org/bugzilla/show_bu