I don’t disagree. There’s no clear signal to be had from a flaky test, so we’ve already made strides with some of the worst offenders. The existing timeout strategy
We’re actively working on strategies to make the test suite more robust (moving pexcept tests to a different strategy, for one, as we have for one of the most troublesome examples.) I expect this to be part of our quality push for the next year. We have existing radars tracking some of the expected work on the suite, but just in case this isn’t being tracked I’ve filed: <rdar://problem/27930082> Replace the timeout strategy used in our test suite with something more robust Kate Stone k8st...@apple.com <mailto:k8st...@apple.com> Xcode Low Level Tools > On Aug 19, 2016, at 2:10 PM, Daniel Dunbar via swift-lldb-dev > <swift-lldb-...@swift.org> wrote: > > The amount of flakiness in the LLDB tests is disturbing... is there a > high-level bug tracking improving this? It seems like it might be worth > running a stress test of the tests on a loaded machine to try and shake out > such problems. > > - Daniel > >> On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev >> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >> >>> >>> On Aug 19, 2016, at 1:42 PM, Enrico Granata <egran...@apple.com >>> <mailto:egran...@apple.com>> wrote: >>> >>>> >>>> On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev >>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>> >>>> It looks like this may be the packaging build: >>>> >>>>>> oss-swift-package-linux-ubuntu-15_10 >>>> >>>> It was this build: >>>> >>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7 >>>> >>>> <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7> >>>> >>>> It has since gone blue. >>>> >>>> I don’t think I recall ever seeing that test fail before in an >>>> intermittent way, but that might be what happened here. >>>> >>> >>> Given the failure mode, it does sound likely: >>> >>> File "/usr/lib/python2.7/socket.py", line 206, in accept >>> sock, addr = self._sock.accept() >>> timeout: timed out >>> >>> >> >> If we see that again, we may be able to tweak the socket timeout to allow >> for heavily loaded test CI. >> >>>> -Todd >>>> >>>>> On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev >>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>> >>>>> Where is this failure being reported? I’m not seeing anything on >>>>> swift-3.0-branch or master on ci.swift.org <http://ci.swift.org/>. >>>>> >>>>> Kate Stone k8st...@apple.com <mailto:k8st...@apple.com> >>>>> Xcode Low Level Tools >>>>> >>>>>> On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev >>>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>>> >>>>>> + swift-lldb-dev group. >>>>>> >>>>>>> On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dun...@apple.com >>>>>>> <mailto:daniel_dun...@apple.com>> wrote: >>>>>>> >>>>>>> Failing in lldb: >>>>>>> Command invoked: >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>> --executable >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>> --rerun-all-issues -C >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>> TestStubReverseConnect.py >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>> --event-add-entries worker_index=14:int >>>>>>> >>>>>>> Configuration: arch=x86_64 >>>>>>> compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>> ---------------------------------------------------------------------- >>>>>>> Collected 2 tests >>>>>>> >>>>>>> ====================================================================== >>>>>>> ERROR: test_reverse_connect_works_llgs >>>>>>> (TestStubReverseConnect.TestStubReverseConnect) >>>>>>> ---------------------------------------------------------------------- >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>> line 121, in wrapper >>>>>>> func(*args, **kwargs) >>>>>>> File >>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>> line 121, in wrapper >>>>>>> func(*args, **kwargs) >>>>>>> File >>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>> line 89, in test_reverse_connect_works_llgs >>>>>>> self.reverse_connect_works() >>>>>>> File >>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>> line 67, in reverse_connect_works >>>>>>> (stub_socket, address) = self.listener_socket.accept() >>>>>>> File "/usr/lib/python2.7/socket.py", line 206, in accept >>>>>>> sock, addr = self._sock.accept() >>>>>>> timeout: timed out >>>>>>> Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9 >>>>>>> ---------------------------------------------------------------------- >>>>>>> Ran 2 tests in 22.006s >>>>>>> >>>>>>> RESULT: FAILED (0 passes, 0 failures, 1 errors, 1 skipped, 0 expected >>>>>>> failures, 0 unexpected successes) >>>>>>> Session logs for test failures/errors/unexpected successes can be found >>>>>>> in directory '2016-08-18-15_15_07' >>>>>>> >>>>>>> <>[TestStubReverseConnect.py FAILED] >>>>>>> Command invoked: /usr/bin/python >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>> --executable >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>> --rerun-all-issues -C >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>> TestStubReverseConnect.py >>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>> --event-add-entries worker_index=14:int >>>>>>> >>>>>>> - Daniel >>>>>>> >>>>>>>> On Aug 18, 2016, at 9:18 PM, no-re...@swift.org >>>>>>>> <mailto:no-re...@swift.org> wrote: >>>>>>>> >>>>>>>> [FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930] >>>>>>>> >>>>>>>> Build URL: >>>>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/ >>>>>>>> <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/> >>>>>>>> Project: oss-swift-package-linux-ubuntu-15_10 >>>>>>>> Date of build: Thu, 18 Aug 2016 20:26:52 -0700 >>>>>>>> Build duration: 51 min >>>>>>>> Identified problems: >>>>>>>> >>>>>>>> Regression test failed: This build failed because a regression test in >>>>>>>> the test suite FAILed. Below is a list of all errors: >>>>>>>> Indication 1 >>>>>>>> <https://ci.swift.org//job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7> >>>>>>>> Changes >>>>>>>> >>>>>>>> Commit 2ebab526bfd7a5ce6ef460bebee0399fe669612d by xi_ge: >>>>>>>> [FixCode] Apply coercion fixits on return statement and initialization >>>>>>>> >>>>>>>> edit: test/FixCode/fixits-apply-objc.swift >>>>>>>> edit: test/FixCode/fixits-apply-objc.swift.result >>>>>>>> edit: lib/Sema/CSDiag.cpp >>>>>>>> edit: test/FixCode/fixits-apply.swift.result >>>>>>>> >>>>>>>> Commit 6447a2d16edcfbceab940d58a321b3ef70525f1a by daniel_dunbar: >>>>>>>> [Basic] Update Thread shim to match changes on Linux side. >>>>>>>> >>>>>>>> edit: Sources/Basic/Thread.swift >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> swift-lldb-dev mailing list >>>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> >>>>> >>>>> _______________________________________________ >>>>> swift-lldb-dev mailing list >>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> >>>> >>>> _______________________________________________ >>>> swift-lldb-dev mailing list >>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> >>> >>> >>> Thanks, >>> - Enrico >>> 📩 egranata@ <about:blank> <about:blank>.com <about:blank> ☎️ 27683 >> >> _______________________________________________ >> swift-lldb-dev mailing list >> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> > _______________________________________________ > swift-lldb-dev mailing list > swift-lldb-...@swift.org > https://lists.swift.org/mailman/listinfo/swift-lldb-dev
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
