Hi Mauro,
Yes we're running in multi-threading mode with 10 threads for concurrent
story execution.
EmbedderControls[batch=false,skip=false,generateViewAfterStories=true,ignoreFailureInStories=true,ignoreFailureInView=false,verboseFailures=true,verboseFiltering=false,storyTimeoutInSecs=108000,failOnStoryTimeout=false,threads=10]
We're trying to speed up the test execution (Selenium-based tests) with
multiple threads (threads=10) targeting a Selenium Grid with 10 nodes.
I noticed that the reported duration for each story is also high and that is
why the sum of all story durations added up to a high total number. Does this
mean the duration at the story level has been affected with multi-threading
execution mode? Do we need to de-normalize (divided by 10 in this case) the
duration for each story to get its accurate execution time?
It would be great if the de-normalization for story duration can be at
addressed at the JBehave framework level to handle the multi-threading
execution mode.
Thanks,
Phong
From: Mauro Talevi [mailto:[email protected]]
Sent: Saturday, March 28, 2015 1:56 AM
To: [email protected]
Subject: Re: [jbehave-user] Story duration are not calculated correctly in
multi-thread execution
Hi Phong,
I would imagine you're running in multi-threading mode. The total is
calculated as simply be the sum of each duration. If you have a single thread
then it will coincide with the build duration (more or less), but you use
multiple threads it'll be off by a factor which should correspond to the number
of threads.
Arguably we could use the information of the threads to renormalise the total
by the number of threads. This formula would still work in the case of a
single thread.
Can you confirm that you're using multiple threads?
Cheers
On 25/03/2015 08:22, tran, phong wrote:
Hi Mauro,
Thanks a lot for updating the core example template! We can access and
integrate the durations from storyDurations.props into our custom HTML report
now. However, we just notice another issue. The total duration reported in
storyDurations.props for all stories is much higher (8 times) that what we
usually see in our test run. Basically we have a configuration in TeamCity to
run our Selenium-based tests and the test run took 5 hours and 25 minutes as
reported by TeamCity. This is the normal time range we expected for our test
run.
Results
Started
Changes
Started
Duration
#177
[cid:[email protected]] NGIS #: 6234 Beacon #: 3224 Pass Rate:
83.43949% passed: 262 failed: 52 pending: 51
[cid:[email protected]] View
Changes (7)
24 Mar 15 09:55
5h:25m
Beacon_Selenium_W2K8R2_13
None
However, the total duration aggregated for all stories as reported by JBehave
is unusually high (more than 48 hours). This duration is not correct!
Reports Overview
Stories
Scenarios
Steps
Download as CSV
Total
Excluded
Total
Successful
Pending
Failed
Excluded
Total
Successful
Pending
Failed
Not Performed
Ignorable
Duration (hh:mm:ss.sss)
279
0
345
210
82
53
0
9,195
3,751
230
66
3,699
1,449
48:39:21.000
storyDurations.props:
#org.jbehave.core.embedder.StoryManager
#Tue Mar 24 15:20:14 PDT 2015
total=175161000
Do you have any idea what could be causing the much higher story duration
reported in storyDurations.props? Is there any JBehave configuration I should
check in our setup?
Thanks,
Phong
From: Mauro Talevi [mailto:[email protected]]
Sent: Saturday, March 21, 2015 7:00 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: [jbehave-user] Story duration are not calculated correctly in
multi-thread execution
Hi Phong,
the core example template was out-of-date. I've pushed the updated one.
You'll see that the storyDurations are taken from a different template object
called "storyDurations".
There is a case for integrating this info into the Report and/or ReportTable
objects. Please raise a JIRA for this against version 4.x.
FYI, these objects are instantiated in the TemplateableViewGenerator.
Cheers
On 20/03/2015 21:50, tran, phong wrote:
Hi Mauro,
For example, take a look at the custom report packaged in JBehave core
example (/behave-3.9.5/examples/core/src/main/java/ftl/custom-reports.ftl). In
this FreeMarker template, a data object reportsTable is accessed and used to
populate the content for HTML reports.
<#assign reportNames = reportsTable.getReportNames()>
Through reportsTable data object, other stats information (including story
duration) is also accessed.
<#assign stats = report.getStats()>
<@renderMillis stats "duration"/>
Questions:
1. Where is reportsTable data model coming from? How is it created and
injected into this FreeMarker template file (.ftl)?
2. As the story duration is no longer available in the stats object
(story .stats file) with JBehave 3.9.4+
(https://jira.codehaus.org/browse/JBEHAVE-1041), and if I want to use and
access the story duration available in storyDurations.props file from the same
FreeMarker template file, how can I do this?
Thanks,
Phong
From: Mauro Talevi [mailto:[email protected]]
Sent: Friday, March 20, 2015 2:16 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: [jbehave-user] Story duration are not calculated correctly in
multi-thread execution
Not sure what you mean. Where do you want to make available the story
durations?
On 20/03/2015 00:17, tran, phong wrote:
Thanks Mauro and Brent for confirming the fix. It's actually my fault. I did
not update all other modules in the project with the latest JBehave 3.9.5. I
could see the storyDurations.props file created with expected story duration
time now. My next question is how I can take the information in
storyDurations.props file and make them available for access in the FreeMarker
templates (.ftl) to produce the HTML views?
Thanks,
Phong
From: Mauro Talevi [mailto:[email protected]]
Sent: Thursday, March 19, 2015 3:00 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: [jbehave-user] Story duration are not calculated correctly in
multi-thread execution
Can you provide a sample project reproducting this behaviour.
This problem should be fixed in latest release.
On 17/03/2015 22:39, tran, phong wrote:
When I ran stories with a single thread (serial), the story duration in reports
looked correct. The duration time is in minutes as expected. See the attached
report (single-thread-reports.png). However, when stories are executed with
multiple threads (currently), the story duration is not calculated correctly
(See Multiple-Thread-Reports.png). The duration time is in in sub-seconds and
is not what expected. I'm using JBehave 3.9.2 and thought this problem could be
related to https://jira.codehaus.org/browse/JBEHAVE-1041 but upgrading to the
latest JBehave 3.9.5 does not seem to help with this problem.
Here is the code fragment for building story report.
.useStoryReporterBuilder(new StoryReporterBuilder()
.withDefaultFormats()
.withFormats(CONSOLE, TXT, HTML_WEB)
.withFailureTrace(true)
.withFailureTraceCompression(false)
.withViewResources(viewResources)
.withCrossReference(xref)
);
Does anyone have an idea what's could be the problem here?
Thanks in advance,
Phong
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
