[jira] Resolved: (LUCENE-1018) intermittant exceptions in TestConcurrentMergeScheduler
[ https://issues.apache.org/jira/browse/LUCENE-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-1018. Resolution: Fixed I just committed this. > intermittant exceptions in TestConcurrentMergeScheduler > --- > > Key: LUCENE-1018 > URL: https://issues.apache.org/jira/browse/LUCENE-1018 > Project: Lucene - Java > Issue Type: Bug > Components: Index >Affects Versions: 2.3 >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.3 > > Attachments: LUCENE-1018.patch > > > The TestConcurrentMergeScheduler throws intermittant exceptions that > do not result in a test failure. > The exception happens in the "testNoWaitClose()" test, which repeated > tests closing an IndexWriter with "false", meaning abort any > still-running merges. When a merge is aborted it can hit various > exceptions because the files it is reading and/or writing have been > deleted, so we ignore these exceptions. > The bug was just that we were failing to properly check whether the > running merge was actually aborted because of a scoping issue of the > "merge" variable in ConcurrentMergeScheduler. So the exceptions are > actually "harmless". Thanks to Ning for spotting it! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1018) intermittant exceptions in TestConcurrentMergeScheduler
[ https://issues.apache.org/jira/browse/LUCENE-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1018: --- Attachment: LUCENE-1018.patch Attached patch, fixing the scoping issue Ning found and also a few other small issues and adding more verbosity when infoStream is set. I plan to commit in a few days. > intermittant exceptions in TestConcurrentMergeScheduler > --- > > Key: LUCENE-1018 > URL: https://issues.apache.org/jira/browse/LUCENE-1018 > Project: Lucene - Java > Issue Type: Bug > Components: Index >Affects Versions: 2.3 >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.3 > > Attachments: LUCENE-1018.patch > > > The TestConcurrentMergeScheduler throws intermittant exceptions that > do not result in a test failure. > The exception happens in the "testNoWaitClose()" test, which repeated > tests closing an IndexWriter with "false", meaning abort any > still-running merges. When a merge is aborted it can hit various > exceptions because the files it is reading and/or writing have been > deleted, so we ignore these exceptions. > The bug was just that we were failing to properly check whether the > running merge was actually aborted because of a scoping issue of the > "merge" variable in ConcurrentMergeScheduler. So the exceptions are > actually "harmless". Thanks to Ning for spotting it! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Created: (LUCENE-1018) intermittant exceptions in TestConcurrentMergeScheduler
intermittant exceptions in TestConcurrentMergeScheduler --- Key: LUCENE-1018 URL: https://issues.apache.org/jira/browse/LUCENE-1018 Project: Lucene - Java Issue Type: Bug Components: Index Affects Versions: 2.3 Reporter: Michael McCandless Assignee: Michael McCandless Priority: Minor Fix For: 2.3 Attachments: LUCENE-1018.patch The TestConcurrentMergeScheduler throws intermittant exceptions that do not result in a test failure. The exception happens in the "testNoWaitClose()" test, which repeated tests closing an IndexWriter with "false", meaning abort any still-running merges. When a merge is aborted it can hit various exceptions because the files it is reading and/or writing have been deleted, so we ignore these exceptions. The bug was just that we were failing to properly check whether the running merge was actually aborted because of a scoping issue of the "merge" variable in ConcurrentMergeScheduler. So the exceptions are actually "harmless". Thanks to Ning for spotting it! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
Indeed this will work! I was unsure whether you could fail() a test
inside tearDown() but it seems to work fine -- I just tested it.
Only downside is a mass replacement of all "extends TestCase" with
"extends LuceneTestCase" (but that's a one time cost, now), and making
sure tearDown() always calls super.tearDown().
OK I like this solution better; I plan to go with that. I'll open a
separate issue for it. Thanks Erik :)
Mike
"Erik Hatcher" <[EMAIL PROTECTED]> wrote:
> Mike,
>
> Would it work to have a common LuceneTestCase base class that could
> do that check and fail() in tearDown?
>
> Erik
>
>
> On Oct 4, 2007, at 5:31 AM, Michael McCandless wrote:
>
> >
> > OK, I think I found one possibility here. With ant's junit task, you
> > can define a custom formatter implementing this interface:
> >
> > org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
> >
> > That interface has a method endTestSuite that is invoked once at the
> > end of all the test cases. So I can define a customer formatter, and
> > in this method, I can check with ConcurrentMergeScheduler and if any
> > unhandled exceptions has occurred, I can throw an exception and the
> > suite/testcase is marked as failed. It seems to work.
> >
> > This is a nice solution in that we don't have to modify every unit
> > test to do its own checking. However, it's not really a "normal" use
> > case because formatters are supposed to just "format" the test result
> > output. It also adds a dependence from Lucene's unit test sources to
> > ant. But at least it does work ("progress not perfection").
> >
> > And objections to this approach? Is there a better approach?
> >
> > Mike
> >
> > "Michael McCandless" <[EMAIL PROTECTED]> wrote:
> >>
> >> "Chris Hostetter" <[EMAIL PROTECTED]> wrote:
> >>>
> >>> : But it'd be nice to do this across the board, ie, for any junit
> >>> test
> >>> : if one of CMS's threads (or, threads launched elsewhere) hits an
> >>> : unhandled exception, fail the testcase that's currently running.
> >>> : I'll dig and see if there's some central way to do this with
> >>> junit...
> >>>
> >>> FYI: i did some casual investigation of this and the only thing that
> >>> jumped out at me is the static
> >>> Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for
> >>> 1.4
> >>> there doesn't seem to be a generic way to notice an uncaught
> >>> exception
> >>> from any thread.
> >>
> >> Thanks Hoss.
> >>
> >> Catching the exception is actually not the hard part because I "own"
> >> all the threads spawned by ConcurrentMergeScheduler. What's
> >> tricky is
> >> finding a way to force the currently running JUnit testcase or suite
> >> to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
> >> to see if there's some hook somewhere where we could insert a check,
> >> just before the suite finishes, to assert that no exceptions were
> >> hit.
> >> Or, if I can somehow "look up" the current Test that's running, I
> >> could add an error to it.
> >>
> >> If there are any JUnit and ant experts out there (I'm not!) please
> >> chime in!
> >>
> >> Mike
> >>
> >> -
> >> To unsubscribe, e-mail: [EMAIL PROTECTED]
> >> For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >
> > -
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
>
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
Mike,
Would it work to have a common LuceneTestCase base class that could
do that check and fail() in tearDown?
Erik
On Oct 4, 2007, at 5:31 AM, Michael McCandless wrote:
OK, I think I found one possibility here. With ant's junit task, you
can define a custom formatter implementing this interface:
org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
That interface has a method endTestSuite that is invoked once at the
end of all the test cases. So I can define a customer formatter, and
in this method, I can check with ConcurrentMergeScheduler and if any
unhandled exceptions has occurred, I can throw an exception and the
suite/testcase is marked as failed. It seems to work.
This is a nice solution in that we don't have to modify every unit
test to do its own checking. However, it's not really a "normal" use
case because formatters are supposed to just "format" the test result
output. It also adds a dependence from Lucene's unit test sources to
ant. But at least it does work ("progress not perfection").
And objections to this approach? Is there a better approach?
Mike
"Michael McCandless" <[EMAIL PROTECTED]> wrote:
"Chris Hostetter" <[EMAIL PROTECTED]> wrote:
: But it'd be nice to do this across the board, ie, for any junit
test
: if one of CMS's threads (or, threads launched elsewhere) hits an
: unhandled exception, fail the testcase that's currently running.
: I'll dig and see if there's some central way to do this with
junit...
FYI: i did some casual investigation of this and the only thing that
jumped out at me is the static
Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for
1.4
there doesn't seem to be a generic way to notice an uncaught
exception
from any thread.
Thanks Hoss.
Catching the exception is actually not the hard part because I "own"
all the threads spawned by ConcurrentMergeScheduler. What's
tricky is
finding a way to force the currently running JUnit testcase or suite
to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
to see if there's some hook somewhere where we could insert a check,
just before the suite finishes, to assert that no exceptions were
hit.
Or, if I can somehow "look up" the current Test that's running, I
could add an error to it.
If there are any JUnit and ant experts out there (I'm not!) please
chime in!
Mike
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
OK, I think I found one possibility here. With ant's junit task, you
can define a custom formatter implementing this interface:
org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
That interface has a method endTestSuite that is invoked once at the
end of all the test cases. So I can define a customer formatter, and
in this method, I can check with ConcurrentMergeScheduler and if any
unhandled exceptions has occurred, I can throw an exception and the
suite/testcase is marked as failed. It seems to work.
This is a nice solution in that we don't have to modify every unit
test to do its own checking. However, it's not really a "normal" use
case because formatters are supposed to just "format" the test result
output. It also adds a dependence from Lucene's unit test sources to
ant. But at least it does work ("progress not perfection").
And objections to this approach? Is there a better approach?
Mike
"Michael McCandless" <[EMAIL PROTECTED]> wrote:
>
> "Chris Hostetter" <[EMAIL PROTECTED]> wrote:
> >
> > : But it'd be nice to do this across the board, ie, for any junit test
> > : if one of CMS's threads (or, threads launched elsewhere) hits an
> > : unhandled exception, fail the testcase that's currently running.
> > : I'll dig and see if there's some central way to do this with junit...
> >
> > FYI: i did some casual investigation of this and the only thing that
> > jumped out at me is the static
> > Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4
> > there doesn't seem to be a generic way to notice an uncaught exception
> > from any thread.
>
> Thanks Hoss.
>
> Catching the exception is actually not the hard part because I "own"
> all the threads spawned by ConcurrentMergeScheduler. What's tricky is
> finding a way to force the currently running JUnit testcase or suite
> to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
> to see if there's some hook somewhere where we could insert a check,
> just before the suite finishes, to assert that no exceptions were hit.
> Or, if I can somehow "look up" the current Test that's running, I
> could add an error to it.
>
> If there are any JUnit and ant experts out there (I'm not!) please
> chime in!
>
> Mike
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
"Chris Hostetter" <[EMAIL PROTECTED]> wrote: > > : But it'd be nice to do this across the board, ie, for any junit test > : if one of CMS's threads (or, threads launched elsewhere) hits an > : unhandled exception, fail the testcase that's currently running. > : I'll dig and see if there's some central way to do this with junit... > > FYI: i did some casual investigation of this and the only thing that > jumped out at me is the static > Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4 > there doesn't seem to be a generic way to notice an uncaught exception > from any thread. Thanks Hoss. Catching the exception is actually not the hard part because I "own" all the threads spawned by ConcurrentMergeScheduler. What's tricky is finding a way to force the currently running JUnit testcase or suite to fail. I'm digging through JUnit and ant's JUnitTestRunner sources to see if there's some hook somewhere where we could insert a check, just before the suite finishes, to assert that no exceptions were hit. Or, if I can somehow "look up" the current Test that's running, I could add an error to it. If there are any JUnit and ant experts out there (I'm not!) please chime in! Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
: But it'd be nice to do this across the board, ie, for any junit test : if one of CMS's threads (or, threads launched elsewhere) hits an : unhandled exception, fail the testcase that's currently running. : I'll dig and see if there's some central way to do this with junit... FYI: i did some casual investigation of this and the only thing that jumped out at me is the static Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4 there doesn't seem to be a generic way to notice an uncaught exception from any thread. -Hoss - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
"Ning Li" <[EMAIL PROTECTED]> wrote: > The cause is that in MergeThread.run(), merge in the try block is a > local variable, while merge in the catch block is the class variable. > Merge in the try block could be one different from the original merge, > but the catch block always checks the abort flag of the original > merge. Aha! You are right :) Good catch. I will fix. Thanks Ning! Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
The cause is that in MergeThread.run(), merge in the try block is a local variable, while merge in the catch block is the class variable. Merge in the try block could be one different from the original merge, but the catch block always checks the abort flag of the original merge. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
> I can't get this to happen. Which OS/hardware are you seeing this on? > I seeing it on my Laptop: 2.0 GHz Centrino, 2 GB RAM, Windows XP, Sun JRE 1.5.0_06. I tried the testcase a couple more times and it's approximately failing in 1 to 2 runs out of 10. > But it'd be nice to do this across the board, ie, for any junit test > if one of CMS's threads (or, threads launched elsewhere) hits an > unhandled exception, fail the testcase that's currently running. > I'll dig and see if there's some central way to do this with junit... > Yeah, that'd be great! - Michael - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
"Michael Busch" <[EMAIL PROTECTED]> wrote: > Hi, > > I noticed the following exception in TestConcurrentMergeScheduler: > > [junit] Testsuite: org.apache.lucene.index.TestConcurrentMergeScheduler > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 17.765 > sec > [junit] - Standard Error - > [junit] Exception in thread "Thread-113" > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.NullPointerException > [junit] at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:263) > [junit] Caused by: java.lang.NullPointerException > [junit] at > org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:2744) > [junit] at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:238) > [junit] - --- I can't get this to happen. Which OS/hardware are you seeing this on? It looks like there is a concurrent merge that tried to initialize itself after its writer had closed. This is likely during the "testNoWaitClose" testcase which does iterations where it closes writer without waiting (forcing an abort of all running merges). I think somehow a merge is getting created but not properly aborted... I will try to find it. > I ran the test again a couple of times, but didn't see the exception > again. Also, these exceptions don't let the testcase fail, so even if > the exceptions occur in the nightly build we probably won't notice it. Good point. I can definitely add checking that no exceptions occurred in these test cases. But it'd be nice to do this across the board, ie, for any junit test if one of CMS's threads (or, threads launched elsewhere) hits an unhandled exception, fail the testcase that's currently running. I'll dig and see if there's some central way to do this with junit... Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Exceptions in TestConcurrentMergeScheduler
Hi, I noticed the following exception in TestConcurrentMergeScheduler: [junit] Testsuite: org.apache.lucene.index.TestConcurrentMergeScheduler [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 17.765 sec [junit] - Standard Error - [junit] Exception in thread "Thread-113" org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException [junit] at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:263) [junit] Caused by: java.lang.NullPointerException [junit] at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:2744) [junit] at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:238) [junit] - --- I ran the test again a couple of times, but didn't see the exception again. Also, these exceptions don't let the testcase fail, so even if the exceptions occur in the nightly build we probably won't notice it. - Michael - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
