Hey Guys,
Austen and I found a CRITICAL bug in the Junit Sensor. Check the
following evidence.
#######################################
1) Here is a log of junit and the junit sensor execution from
StackyHack. Note, that i added a few fake temporary tests to
demonstrate the problem. Everything seems fine with the log.
C:\java\svn-csdl\StackyHack>ant -f junit.build.xml
Buildfile: junit.build.xml
compile:
[javac] Compiling 1 source file to
C:\java\svn-csdl\StackyHack\build\classes
junit.tool:
[junit] Running edu.hawaii.stack.TestClearStack
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.062 sec
[junit] Running edu.hawaii.stack.TestStack
[junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 0 sec
junit.report:
[junitreport] Transform time: 672ms
junit.sensor:
[hacky-junit] Sensor enabled?: true
[hacky-junit] Processing file:
C:\java\svn-csdl\StackyHack\build\junit\TEST-edu.hawaii.stack.TestClearStack.xml
[hacky-junit] Processing file:
C:\java\svn-csdl\StackyHack\build\junit\TEST-edu.hawaii.stack.TestStack.xml
[hacky-junit] Hackystat data on 9 JUnit tests sent to
http://hackystat.ics.hawaii.edu/ (0 secs.)
junit:
BUILD SUCCESSFUL
Total time: 4 seconds
C:\java\svn-csdl\StackyHack>
#######################################
2) Here is the junit.0.log file from the .hackystat/logs directory.
Again everything looks okay here.
Hackystat Version: 7.5.818 (August 18 2006 13:30:59)
SensorShell started at: 08/23/2006 18:07:21
Type 'help' for a list of commands.
Host: http://hackystat.ics.hawaii.edu/ is available and key is valid.
Defined shell command: Dependency
Defined shell command: Issue
Defined shell command: EvolSdt
Defined shell command: CodeIssue
Defined shell command: Perf
Defined shell command: FileMetric
Defined shell command: ReviewIssue
Defined shell command: Activity
Defined shell command: Cli
Defined shell command: ReviewActivity
Defined shell command: Coverage
Defined shell command: DevEvent
Defined shell command: UnitTest
Defined shell command: BuffTrans
Defined shell command: Commit
Defined shell command: Build
#> AutoSend [10]
AutoSend OK (set to 10 minutes)
AutoSend enabled every 10 minutes.
Checking for offline data to recover.
No offline data found.
#> UnitTest [set, tool=JUnit]
set OK
#> UnitTest [add,
name=edu.hawaii.stack.TestClearStack.testNormalOperation,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestClearStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00019testNormalOperation00008testName00031edu.hawaii.stack.TestClearStack]
UnitTest add OK (1 total)
#> UnitTest [add,
name=edu.hawaii.stack.TestStack.testNormalOperation,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00019testNormalOperation00008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (2 total)
#> UnitTest [add, name=edu.hawaii.stack.TestStack.testIllegalPop,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00014testIllegalPop00008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (3 total)
#> UnitTest [add, name=edu.hawaii.stack.TestStack.testTemp1,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00009testTemp100008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (4 total)
#> UnitTest [add, name=edu.hawaii.stack.TestStack.testTemp2,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00009testTemp200008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (5 total)
#> UnitTest [add, name=edu.hawaii.stack.TestStack.testTemp3,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00009testTemp300008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (6 total)
#> UnitTest [add, name=edu.hawaii.stack.TestStack.testTemp4,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00009testTemp400008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (7 total)
#> UnitTest [add, name=edu.hawaii.stack.TestStack.testTemp5,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00009testTemp500008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (8 total)
#> UnitTest [add, name=edu.hawaii.stack.TestStack.testTemp6,
path=C:\java\svn-csdl\StackyHack\src\edu\hawaii\stack\TestStack.java,
result=pass,
pMap=000600011elapsedTime00001000008testCase00009testTemp600008testName00026edu.hawaii.stack.TestStack]
UnitTest add OK (9 total)
#> send
Sending sensor data (08/23 18:07:22)
Ping: Ping OK (contacted server http://hackystat.ics.hawaii.edu/
with valid key.)
Dependency: Send OK (No entries to send.)
AutoSend: AutoSend OK ('send' command ignored)
Issue: Send OK (No entries to send.)
EvolSdt: Send OK (No entries to send.)
CodeIssue: Send OK (No entries to send.)
Perf: Send OK (No entries to send.)
FileMetric: Send OK (No entries to send.)
ReviewIssue: Send OK (No entries to send.)
Activity: Send OK (No entries to send.)
Cli: Send OK (No entries to send.)
ReviewActivity: Send OK (No entries to send.)
DevEvent: Send OK (No entries to send.)
Coverage: Send OK (No entries to send.)
Commit: Send OK (No entries to send.)
BuffTrans: Send OK (No entries to send.)
UnitTest: Send OK (9 entries)
Build: Send OK (No entries to send.)
>>
#######################################
3) Now lets take a look at the UnitTest data in the List Sensor Data
analysis: http://csdl.ics.hawaii.edu/~kagawaa/UnitTestSensorData.JPG
Note that the last three entries are associated with the execution in
the log and sensor log. So, you can see that we have only three
entries but we should have 9 entries!
I should have data for the following methods.
TestStack.testNormalOperation()
TestStack.testIllegalPop()
TestStack.testTemp1()
TestStack.testTemp2()
TestStack.testTemp3()
TestStack.testTemp4()
TestStack.testTemp5()
TestStack.testTemp6()
But, I only have these:
TestStack.testTemp5()
TestStack.testTemp6()
TestStack.testNormalOperation()
So, I thought there might be a bug in the List Sensor Data
analysis. But, I checked the Daily Project Details
(http://csdl.ics.hawaii.edu/~kagawaa/DailyProjectDetails-StackyHack.JPG)
and it shows only 6 unit tests but it should show 12 (3 + 9 from the
last run).
It seems that the maximum amount of UnitTest entries is two per Java
file. Meaning two different name values for the same path value.
This isn't the only example that I've seen. Austen and I seen this
problem in two different cases in our Company's data. And it is
reproducible with StackHack. So, it seems that there is a CRITICAL
bug in the Junit sensor or something that is receiving the data.
thanks, Aaron
