RE: changes to surefire output with testng
Jason Chaffee wrote: I am pretty familiar with testng code, so I would like to look into this further. Is it a correct statement to say that the output we are seeing on the console is coming directly from testng? No... Console output in Surefire is a bit strange. :-) By default, Surefire forks a single Java process to run all tests and begins gathering console output from that process. If you configure Surefire's output using redirectTestOutputToFile, we'll print it all out to a file, UNLESS the line begins with a special marker indicating that the console output is intended for Surefire, in which case we'll display it in the clear on the console. Hence, TestNG can print out "foo" and that will go to the file, but Surefire itself can print out "Tests run: 1, Failures: 0, Errors: 0 [...]". If you don't redirectTestOutputToFile, console messages from the forked process appear directly on the console, but we also take extra steps to strip out the special markers. Currently, Surefire only knows to log when TestNG calls us back. We register as an org.testng.IResultListener, so we get notifications when the run starts and finishes, and when an individual method starts and finishes, but not when the entire class starts and finishes. If TestNG notified us at the right time, we could log the right stuff. But I think this won't be an easy fix on the TestNG side, because TestNG runs stuff in a crazy order, by design. -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: changes to surefire output with testng
I am pretty familiar with testng code, so I would like to look into this further. Is it a correct statement to say that the output we are seeing on the console is coming directly from testng? -Original Message- From: Dan Fabulich [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 12, 2008 3:36 PM To: Maven Developers List Cc: Maven Users List Subject: Re: changes to surefire output with testng Benjamin's links are the right place to start, though I might add that the head of that thread is a really long e-mail from me that doesn't directly address your question. The rest of the thread is about your question. Executive summary: TestNG support was broken in Surefire 2.3.x; it only worked in certain simple cases. In Surefire 2.4 we give more control over to TestNG, fixing the functional errors, but reducing our ability to log, because TestNG doesn't call us back at the right times. There is no option to switch back to the old mostly-broken behavior. If this feature matters (in the linked thread I argue that it doesn't matter very much) then users should pester the TestNG team to provide support for it, by creating an "onClassStart"/"onClassFinish" event that Surefire can use. -Dan Benjamin Bentmann wrote: >> Surefire-2.3.1 used to output which test was running and it would create >> an output file for each test. >> >> I am using testng. Can anyone tell me why this changed and if there is >> any way to configure it to behave in the same manner as previous >> releases? > > Nice to see I am not the only one who is missing that after trying Surefire > 2.4 out... You might want to look at > - http://jira.codehaus.org/browse/SUREFIRE-433 > - > http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport- to15076378.html#a15076378 > > As a workaround, I tried to set forkMode=always to get test-per-class > behavior back but ended again in some discussion: > - http://jira.codehaus.org/browse/SUREFIRE-446 > - > http://www.nabble.com/testng%2C-surefire%2C-and-forkMode%3Dalways-to1532 3782.html > > Regards, > > > Benjamin Bentmann > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: changes to surefire output with testng
Jason Chaffee wrote: Hmmm, the bug says it is fixed in 2.4.1 and it still produces a single suite file and single suite output to the console. I agree with you Benjamin, I don't think Dan understood the problem and thus didn't actually fix it. Instead, the fix was for the reporting. SUREFIRE-433 was a separate problem. I think I understood Benjamin's problem (I think, Benjamin can decide) but chose not to fix it in Surefire. [I think this is really a TestNG feature; if it matters at all, the TestNG team should implement it.] I've filed bug SUREFIRE-457 so people can vote for if if they like. It's currently a "Wish" marked Minor and targeted for post 2.x "Future", meaning "Maybe someday, maybe never". ;-) -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: changes to surefire output with testng
Benjamin's links are the right place to start, though I might add that the head of that thread is a really long e-mail from me that doesn't directly address your question. The rest of the thread is about your question. Executive summary: TestNG support was broken in Surefire 2.3.x; it only worked in certain simple cases. In Surefire 2.4 we give more control over to TestNG, fixing the functional errors, but reducing our ability to log, because TestNG doesn't call us back at the right times. There is no option to switch back to the old mostly-broken behavior. If this feature matters (in the linked thread I argue that it doesn't matter very much) then users should pester the TestNG team to provide support for it, by creating an "onClassStart"/"onClassFinish" event that Surefire can use. -Dan Benjamin Bentmann wrote: Surefire-2.3.1 used to output which test was running and it would create an output file for each test. I am using testng. Can anyone tell me why this changed and if there is any way to configure it to behave in the same manner as previous releases? Nice to see I am not the only one who is missing that after trying Surefire 2.4 out... You might want to look at - http://jira.codehaus.org/browse/SUREFIRE-433 - http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-to15076378.html#a15076378 As a workaround, I tried to set forkMode=always to get test-per-class behavior back but ended again in some discussion: - http://jira.codehaus.org/browse/SUREFIRE-446 - http://www.nabble.com/testng%2C-surefire%2C-and-forkMode%3Dalways-to15323782.html Regards, Benjamin Bentmann - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: changes to surefire output with testng
Hmmm, the bug says it is fixed in 2.4.1 and it still produces a single suite file and single suite output to the console. I agree with you Benjamin, I don't think Dan understood the problem and thus didn't actually fix it. Instead, the fix was for the reporting. -Original Message- From: Benjamin Bentmann [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 12, 2008 3:15 PM To: Maven Developers List; Maven Users List Subject: Re: changes to surefire output with testng Hi, > Surefire-2.3.1 used to output which test was running and it would create > an output file for each test. > > I am using testng. Can anyone tell me why this changed and if there is > any way to configure it to behave in the same manner as previous > releases? Nice to see I am not the only one who is missing that after trying Surefire 2.4 out... You might want to look at - http://jira.codehaus.org/browse/SUREFIRE-433 - http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport- to15076378.html#a15076378 As a workaround, I tried to set forkMode=always to get test-per-class behavior back but ended again in some discussion: - http://jira.codehaus.org/browse/SUREFIRE-446 - http://www.nabble.com/testng%2C-surefire%2C-and-forkMode%3Dalways-to1532 3782.html Regards, Benjamin Bentmann - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: changes to surefire output with testng
Hi, Surefire-2.3.1 used to output which test was running and it would create an output file for each test. I am using testng. Can anyone tell me why this changed and if there is any way to configure it to behave in the same manner as previous releases? Nice to see I am not the only one who is missing that after trying Surefire 2.4 out... You might want to look at - http://jira.codehaus.org/browse/SUREFIRE-433 - http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-to15076378.html#a15076378 As a workaround, I tried to set forkMode=always to get test-per-class behavior back but ended again in some discussion: - http://jira.codehaus.org/browse/SUREFIRE-446 - http://www.nabble.com/testng%2C-surefire%2C-and-forkMode%3Dalways-to15323782.html Regards, Benjamin Bentmann - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
changes to surefire output with testng
Surefire-2.3.1 used to output which test was running and it would create an output file for each test. For example, [INFO] [surefire:test] [INFO] Surefire report directory: /usr/local/hudson/workspace/common/common/management/target/surefire-rep orts --- T E S T S --- Running com.foo.management.MBeanTest Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.279 sec Running com.foo.TomcatServerTest Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.509 sec Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 0 Surefire 2.4 and 2.4.1 no longer do this, instead it does the following: [INFO] [surefire:test] [INFO] Surefire report directory: C:\workspace\tva\server\common\management\targ et\surefire-reports --- T E S T S --- Running TestSuite Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.39 sec Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 0 I am using testng. Can anyone tell me why this changed and if there is any way to configure it to behave in the same manner as previous releases?
