Agorguy opened a new issue, #2219:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2219
Hello,
We tried running your project and discovered that it contains some flaky
tests (i.e., tests that nondeterministically pass and fail). We found these
tests to fail more frequently when running them on certain machines of ours.
To prevent others from running this project and its tests in machines that
may result in flaky tests, we suggest adding information to the README.md file
indicating the minimum resource configuration for running the tests of this
project as to prevent observation of test flakiness:
If we run this project in a machine with 1cpu and 1gb ram, we observe flaky
tests. We found that the tests in this project did not have any flaky tests
when we ran it on machines with 2cpu and 2gb ram.
Here is a list of the tests we have identified and their likelihood of
failure on a system with less than the recommended 2 CPUs and 2 GB RAM.
1.
org.apache.shardingsphere.elasticjob.error.handler.wechat.WechatJobErrorHandlerTest#assertHandleExceptionWithNotifySuccessful
( 4 out 10 )
2.
org.apache.shardingsphere.elasticjob.cloud.scheduler.mesos.MesosStateServiceTest#assertExecutors
(1 out 10)
org.apache.shardingsphere.elasticjob.error.handler.wechat.WechatJobErrorHandlerTest#assertHandleExceptionWithWrongToken
(1 out 10)
3.
org.apache.shardingsphere.elasticjob.http.executor.HttpJobExecutorTest#assertProcessWithGet
(1 out 10)
4.
org.apache.shardingsphere.elasticjob.lite.spring.namespace.job.OneOffJobSpringNamespaceWithTypeTest#jobScriptWithJobTypeTest
(1 out 10)
Please let me know if you would like us to create a pull request on this
matter (possibly to the readme of this project).
## Reproducing
~~~Dockerfile
FROM maven:3.5.4-jdk-11
WORKDIR /home/
RUN git clone https://github.com/apache/shardingsphere-elasticjob && \
cd shardingsphere-elasticjob && \
git checkout 2bfce1ebc39475e1b8eda456b673ab9431c0f270
WORKDIR /home/shardingsphere-elasticjob
RUN mvn install -DskipTests
ENTRYPOINT ["mvn", "test", "-fn"]
~~~
Build the image:
~~~bash
$> mkdir tmp
$> cp Dockerfile tmp
$> cd tmp
$> docker build -t elasticjob . # estimated time of build 3m
~~~
Running:
this configuration likely prevents flakiness (no flakiness in 10 runs)
~~~bash
$> docker run --rm --memory=2g --cpus=2 --memory-swap=-1 elasticjob | tee
output.txt
$> grep "Failures:" output.txt # checking results
~~~
this other configuration –similar to the previous– can’t prevent flaky
tests (observation in 10 runs)
~~~bash
$> docker run --rm --memory=1g --cpus=1 --memory-swap=-1 elasticjob | tee
output2.txt
$> grep "Failures:" output2.txt # checking results
~~~
Thank you for your attention to this matter. We hope that our
recommendations will be helpful in improving the quality and performance of
your project, especially for others to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
