It seems that we do not set MR_AM_STAGING_DIR option when starting the mini MR cluster. The default value is '/tmp/hadoop-yarn/staging'.
2017-04-27 7:06 GMT+08:00 Apekshit Sharma <[email protected]>: > So the issue here is, somehow Test*ExportSnapshot (4 tests classes) tests > are stepping on each other. > When we run locally, tests are run sequentially, hence they always pass. > But in precommit/flaky-tests-job, there are multiple forks and multiple > tests can run in parallel. That's when they fail. > If you look at this failure [1], the following exception is weird because > these tests should be using /tmp/* dir, but one of our > <module>/target/test-data/xxxx-yyyy random dirs. > > 2017-04-21 21:15:48,787 WARN [IPC Server handler 3 on 46240] > localizer.ResourceLocalizationService$LocalizerRunner(1015): { > file:/tmp/hadoop-yarn/staging/jenkins/.staging/job_ > 1492809330233_0001/job.split, > 1492809346000, FILE, null } failed: File > file:/tmp/hadoop-yarn/staging/jenkins/.staging/job_ > 1492809330233_0001/job.split > does not exist > java.io.FileNotFoundException: File > file:/tmp/hadoop-yarn/staging/jenkins/.staging/job_ > 1492809330233_0001/job.split > does not exist > at org.apache.hadoop.fs.RawLocalFileSystem. > deprecatedGetFileStatus(RawLocalFileSystem.java:606) > at org.apache.hadoop.fs.RawLocalFileSystem. > getFileLinkStatusInternal(RawLocalFileSystem.java:819) > at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus( > RawLocalFileSystem.java:596) > at org.apache.hadoop.fs.FilterFileSystem.getFileStatus( > FilterFileSystem.java:421) > at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload. > java:253) > at org.apache.hadoop.yarn.util.FSDownload.access$000( > FSDownload.java:63) > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload. > java:361) > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload. > java:359) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1657) > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload. > java:359) > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > [1] > https://builds.apache.org/job/HBASE-Flaky-Tests/15333/ > artifact/hbase-server/target/surefire-reports/org.apache. > hadoop.hbase.snapshot.TestMobExportSnapshot-output.txt/*view*/ > > On Wed, Apr 26, 2017 at 2:39 PM, Stephen Jiang <[email protected]> > wrote: > > > It is not flaky in my local machine :-(. Today I can 100% repro in 7 > runs. > > "org.apache.hadoop.hbase.snapshot.TestExportSnapshot" also always > fails. > > > > Thanks > > Stephen > > > > On Wed, Apr 26, 2017 at 2:32 PM, Ted Yu <[email protected]> wrote: > > > > > You can take a look at: > > > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/ > > > lastSuccessfulBuild/artifact/dashboard.html > > > > > > It seems the test is flaky. > > > > > > FYI > > > > > > On Wed, Apr 26, 2017 at 2:25 PM, Stephen Jiang < > [email protected]> > > > wrote: > > > > > > > I am running "org.apache.hadoop.hbase.snapshot. > > TestSecureExportSnapshot" > > > > test in my local branch-1 today and it failed. I just wonder anyone > > > else > > > > see the same failure? > > > > > > > > ------------------------------------------------------- > > > > T E S T S > > > > ------------------------------------------------------- > > > > Running org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot > > > > Tests run: 9, Failures: 7, Errors: 0, Skipped: 0, Time elapsed: > 163.004 > > > sec > > > > <<< FAILURE! - in org.apache.hadoop.hbase.snapshot. > > > > TestSecureExportSnapshot > > > > testExportWithTargetName(org.apache.hadoop.hbase.snapshot. > > > > TestSecureExportSnapshot) > > > > Time elapsed: 14.708 sec <<< FAILURE! > > > > java.lang.AssertionError: expected:<0> but was:<1> > > > > at org.junit.Assert.fail(Assert.java:88) > > > > at org.junit.Assert.failNotEquals(Assert.java:834) > > > > at org.junit.Assert.assertEquals(Assert.java:645) > > > > at org.junit.Assert.assertEquals(Assert.java:631) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:294) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:266) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportWithTargetName(TestExportSnapshot.java:223) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > > > FrameworkMethod.java:50) > > > > at > > > > org.junit.internal.runners.model.ReflectiveCallable.run( > > > > ReflectiveCallable.java:12) > > > > at > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > > > FrameworkMethod.java:47) > > > > at > > > > org.junit.internal.runners.statements.InvokeMethod. > > > > evaluate(InvokeMethod.java:17) > > > > at > > > > org.junit.internal.runners.statements.RunBefores. > > > > evaluate(RunBefores.java:26) > > > > at > > > > org.junit.internal.runners.statements.RunAfters.evaluate( > > > > RunAfters.java:27) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:298) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:292) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > testExportFileSystemState(org.apache.hadoop.hbase.snapshot. > > > > TestSecureExportSnapshot) > > > > Time elapsed: 13.045 sec <<< FAILURE! > > > > java.lang.AssertionError: expected:<0> but was:<1> > > > > at org.junit.Assert.fail(Assert.java:88) > > > > at org.junit.Assert.failNotEquals(Assert.java:834) > > > > at org.junit.Assert.assertEquals(Assert.java:645) > > > > at org.junit.Assert.assertEquals(Assert.java:631) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:294) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:266) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:198) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > > > FrameworkMethod.java:50) > > > > at > > > > org.junit.internal.runners.model.ReflectiveCallable.run( > > > > ReflectiveCallable.java:12) > > > > at > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > > > FrameworkMethod.java:47) > > > > at > > > > org.junit.internal.runners.statements.InvokeMethod. > > > > evaluate(InvokeMethod.java:17) > > > > at > > > > org.junit.internal.runners.statements.RunBefores. > > > > evaluate(RunBefores.java:26) > > > > at > > > > org.junit.internal.runners.statements.RunAfters.evaluate( > > > > RunAfters.java:27) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:298) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:292) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > testConsecutiveExports(org.apache.hadoop.hbase.snapshot. > > > > TestSecureExportSnapshot) > > > > Time elapsed: 12.967 sec <<< FAILURE! > > > > java.lang.AssertionError: expected:<0> but was:<1> > > > > at org.junit.Assert.fail(Assert.java:88) > > > > at org.junit.Assert.failNotEquals(Assert.java:834) > > > > at org.junit.Assert.assertEquals(Assert.java:645) > > > > at org.junit.Assert.assertEquals(Assert.java:631) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:294) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testConsecutiveExports(TestExportSnapshot.java:215) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > > > FrameworkMethod.java:50) > > > > at > > > > org.junit.internal.runners.model.ReflectiveCallable.run( > > > > ReflectiveCallable.java:12) > > > > at > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > > > FrameworkMethod.java:47) > > > > at > > > > org.junit.internal.runners.statements.InvokeMethod. > > > > evaluate(InvokeMethod.java:17) > > > > at > > > > org.junit.internal.runners.statements.RunBefores. > > > > evaluate(RunBefores.java:26) > > > > at > > > > org.junit.internal.runners.statements.RunAfters.evaluate( > > > > RunAfters.java:27) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:298) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:292) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > testEmptyExportFileSystemState(org.apache.hadoop.hbase.snapshot. > > > > TestSecureExportSnapshot) > > > > Time elapsed: 12.913 sec <<< FAILURE! > > > > java.lang.AssertionError: expected:<0> but was:<1> > > > > at org.junit.Assert.fail(Assert.java:88) > > > > at org.junit.Assert.failNotEquals(Assert.java:834) > > > > at org.junit.Assert.assertEquals(Assert.java:645) > > > > at org.junit.Assert.assertEquals(Assert.java:631) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:294) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:266) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testEmptyExportFileSystemState(TestExportSnapshot.java:209) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > > > FrameworkMethod.java:50) > > > > at > > > > org.junit.internal.runners.model.ReflectiveCallable.run( > > > > ReflectiveCallable.java:12) > > > > at > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > > > FrameworkMethod.java:47) > > > > at > > > > org.junit.internal.runners.statements.InvokeMethod. > > > > evaluate(InvokeMethod.java:17) > > > > at > > > > org.junit.internal.runners.statements.RunBefores. > > > > evaluate(RunBefores.java:26) > > > > at > > > > org.junit.internal.runners.statements.RunAfters.evaluate( > > > > RunAfters.java:27) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:298) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:292) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > testSnapshotWithRefsExportFileSystemState(org.apache.hadoop. > > > > hbase.snapshot.TestSecureExportSnapshot) > > > > Time elapsed: 13.342 sec <<< FAILURE! > > > > java.lang.AssertionError: expected:<0> but was:<1> > > > > at org.junit.Assert.fail(Assert.java:88) > > > > at org.junit.Assert.failNotEquals(Assert.java:834) > > > > at org.junit.Assert.assertEquals(Assert.java:645) > > > > at org.junit.Assert.assertEquals(Assert.java:631) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:294) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:266) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testSnapshotWithRefsExportFileSystemState( > TestExportSnapshot.java:260) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testSnapshotWithRefsExportFileSystemState( > TestExportSnapshot.java:240) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > > > FrameworkMethod.java:50) > > > > at > > > > org.junit.internal.runners.model.ReflectiveCallable.run( > > > > ReflectiveCallable.java:12) > > > > at > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > > > FrameworkMethod.java:47) > > > > at > > > > org.junit.internal.runners.statements.InvokeMethod. > > > > evaluate(InvokeMethod.java:17) > > > > at > > > > org.junit.internal.runners.statements.RunBefores. > > > > evaluate(RunBefores.java:26) > > > > at > > > > org.junit.internal.runners.statements.RunAfters.evaluate( > > > > RunAfters.java:27) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:298) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:292) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > testExportFileSystemStateWithSkipTmp(org.apache.hadoop. > hbase.snapshot. > > > > TestSecureExportSnapshot) > > > > Time elapsed: 12.136 sec <<< FAILURE! > > > > java.lang.AssertionError: expected:<0> but was:<1> > > > > at org.junit.Assert.fail(Assert.java:88) > > > > at org.junit.Assert.failNotEquals(Assert.java:834) > > > > at org.junit.Assert.assertEquals(Assert.java:645) > > > > at org.junit.Assert.assertEquals(Assert.java:631) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:294) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemState(TestExportSnapshot.java:266) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot. > > > > testExportFileSystemStateWithSkipTmp(TestExportSnapshot.java:204) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > > > FrameworkMethod.java:50) > > > > at > > > > org.junit.internal.runners.model.ReflectiveCallable.run( > > > > ReflectiveCallable.java:12) > > > > at > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > > > FrameworkMethod.java:47) > > > > at > > > > org.junit.internal.runners.statements.InvokeMethod. > > > > evaluate(InvokeMethod.java:17) > > > > at > > > > org.junit.internal.runners.statements.RunBefores. > > > > evaluate(RunBefores.java:26) > > > > at > > > > org.junit.internal.runners.statements.RunAfters.evaluate( > > > > RunAfters.java:27) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:298) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:292) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > testExportRetry(org.apache.hadoop.hbase.snapshot. > > > TestSecureExportSnapshot) > > > > Time elapsed: 12.903 sec <<< FAILURE! > > > > java.lang.AssertionError: expected:<0> but was:<1> > > > > at org.junit.Assert.fail(Assert.java:88) > > > > at org.junit.Assert.failNotEquals(Assert.java:834) > > > > at org.junit.Assert.assertEquals(Assert.java:645) > > > > at org.junit.Assert.assertEquals(Assert.java:631) > > > > at > > > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportRetry( > > > > TestExportSnapshot.java:329) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > > > FrameworkMethod.java:50) > > > > at > > > > org.junit.internal.runners.model.ReflectiveCallable.run( > > > > ReflectiveCallable.java:12) > > > > at > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > > > FrameworkMethod.java:47) > > > > at > > > > org.junit.internal.runners.statements.InvokeMethod. > > > > evaluate(InvokeMethod.java:17) > > > > at > > > > org.junit.internal.runners.statements.RunBefores. > > > > evaluate(RunBefores.java:26) > > > > at > > > > org.junit.internal.runners.statements.RunAfters.evaluate( > > > > RunAfters.java:27) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:298) > > > > at > > > > org.junit.internal.runners.statements.FailOnTimeout$ > > > > CallableStatement.call(FailOnTimeout.java:292) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > > > Results : > > > > > > > > Failed tests: > > > > > > > > TestSecureExportSnapshot>TestExportSnapshot. > > testConsecutiveExports:215-> > > > > TestExportSnapshot.testExportFileSystemState:294 > > > > expected:<0> but was:<1> > > > > > > > > TestSecureExportSnapshot>TestExportSnapshot. > > > testEmptyExportFileSystemState > > > > :209->TestExportSnapshot.testExportFileSystemState:266- > > > > >TestExportSnapshot.testExportFileSystemState:294 > > > > expected:<0> but was:<1> > > > > > > > > TestSecureExportSnapshot>TestExportSnapshot. > > > testExportFileSystemState:198- > > > > >TestExportSnapshot.testExportFileSystemState:266- > >TestExportSnapshot. > > > > testExportFileSystemState:294 > > > > expected:<0> but was:<1> > > > > > > > > TestSecureExportSnapshot>TestExportSnapshot. > > > testExportFileSystemStateWithS > > > > kipTmp:204->TestExportSnapshot.testExportFileSystemState:266- > > > > >TestExportSnapshot.testExportFileSystemState:294 > > > > expected:<0> but was:<1> > > > > TestSecureExportSnapshot>TestExportSnapshot.testExportRetry:329 > > > > expected:<0> but was:<1> > > > > > > > > TestSecureExportSnapshot>TestExportSnapshot. > > > testExportWithTargetName:223-> > > > > TestExportSnapshot.testExportFileSystemState:266- > >TestExportSnapshot. > > > > testExportFileSystemState:294 > > > > expected:<0> but was:<1> > > > > > > > > TestSecureExportSnapshot>TestExportSnapshot. > > > testSnapshotWithRefsExportFile > > > > SystemState:240->TestExportSnapshot.testSnapshotWithRefsExportFile > > > > SystemState:260->TestExportSnapshot.testExportFileSystemState:266- > > > > >TestExportSnapshot.testExportFileSystemState:294 > > > > expected:<0> but was:<1> > > > > > > > > Tests run: 9, Failures: 7, Errors: 0, Skipped: 0 > > > > > > > > > > > > > -- > > -- Appy >
