Hello jc,
sorry for maybe starting the obvious, but first thing I do in case
of
such errors, is to make sure that every component in the Grid is
running
the latest version, as well as upgrade the Selenium version in my
test
project.
Most of the time, this would solve these kind of issues for me.
I have also made good experience with restarting the Grid before
running
a major test suite during the night.
The "forkEvery" parameter in Gradle (and similar mechanisms) would
only
affect the machine running the test code, not the ones running the
browser code, so that should not have any effect on the Grid.
For the Grid it should only matter how many requests for browser
slots
are occurring simultaneously.
Another thing you could check, is whether the browsers get restarted
during the tests.
Maybe there is some sort of memory leak (in the webpage/Javascript
that
you are testing), and restarting could help with that. The browser
itself running out of memory seems suspicious. I have never seen
this
before, and I have been using Grid a lot.
A memory leak could also be a result of a version mismatch between
Selenium and the Grid, but I'm speculating here.
Depending how much you have customised your Grid installation, it
might
not be an option to do this, but I can also very much recommend to
use
Zalenium (https://opensource.zalando.com/zalenium/ [1]) as Grid
implementation.
It helped me save a lot of maintenance effort, and it comes with a
lot
of great features.
Best regards,
Thomas
Am 02.09.2020 16:19 schrieb jc:
Hi all,
I am coming here as a shot in the dark. We have a suite of about
600+
tests that we run on Chrome in desktop resolution and then we also
run
them in Tablet and Mobile resolution using Chrome's mobile
emulation.
We also run it on Firefox desktop resolution. So about 2500 tests
on
the same application running nightly through Jenkins.
Starting in early July we haven't had much for passing builds
because
at some point during the tests I get seemingly random
WebDriverExceptions. Most of the time I get errors such as these:
_- org.openqa.selenium.WebDriverException: Session
[a7d84a66-daaa-48ad-a0cd-c29dfb98d805] was terminated due to
BROWSER_TIMEOUT - __org.openqa.selenium.WebDriverException:
Session
[a7d84a66-daaa-48ad-a0cd-c29dfb98d805] was terminated due to
TIMEOUT_
_- __org.openqa.selenium.WebDriverException: Session
[a7d84a66-daaa-48ad-a0cd-c29dfb98d805] was terminated due to
SO_TIMEOUT_
_- __org.openqa.selenium.WebDriverException: Session
[a7d84a66-daaa-48ad-a0cd-c29dfb98d805] was terminated due to
ERROR_FORWARDING_TO_NODE _(don't see this one often)
Nothing has really changed in the application and thus the code in
the
test suite hasn't really changes all that much. The builds are ran
from Jenkins overnight in the order of Chrome Desktop > Chrome
Mobile
Chrome Tablet > Firefox Desktop.
The grid is set up with docker for the HUB only, though I am not
sure
what image it is. I just know it is the latest stable version of
3.141.59. The nodes are VMs with the same image on each one of
them.
There are 10 nodes connected to the hub with 5 sessions each of
Chrome, Firefox, and IE. Most nights everything will pass except
for 1
or 2 test classes that fail halfway through because of the
exceptions
and then it seems to get progressively worse down to where Firefox
which is last in line almost always fails somewhere.
I think what is happening is these tests are crashing either the
hubs
or the nodes but I don't know why. I run the tests in parallel
with
grade's MAXPARALLELFORKS = 5 which launches 1 session on 5
different
VMs. I have tried playing with this number, turning it down to 3
even
and that did not seem to help. I have FORKEVERY = 1 because I
believe
Firefox had a memory leak issue awhile back and have decided to
leave
it. I also have CACHEDRIVERPERTHREAD = TRUE in my
GebConfig.groovy. I
can't imagine 1 browser per machine is causing memory issues.
Just last night I decided to use a local windows laptop of mine
and a
coworkers to test out what might be happening. Chrome desktop ran
100%
fine, and then Chrome mobile ran good about halfway through until
every broke. Chrome tablet and Firefox desktop subsequently had
failures all over the board. When we both logged into our machine
my
coworker said the nodes said that Javascript ran out of memory. On
my
windows machine Chrome was stuck open and said "Out of memory"
also.
These machines have 16GB of ram each so that is surprising to me.
The
hub and Chrome node were actually closed as well, I'm assuming
they
crashed. My guess is this is basically what is happening on our
docker
hub but I don't know why.
I guess what I am trying to find out is if this is a Geb/Spock
issue
and if so, why? What other settings should I check in my
build.gradle
or my GebConfig.groovy? If it's a hub/node issue, what else can we
check? The VMs have 10GB of RAM on them which should be plenty.
Has anyone else had these issues? If I should ask this somewhere
else
please let me know where. Any help or troubleshooting tips is
appreciated.
--
You received this message because you are subscribed to the Google
Groups "Geb User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected].
To view this discussion on the web visit
