> On Aug 19, 2016, at 3:02 PM, Enrico Granata via swift-lldb-dev > <swift-lldb-...@swift.org> wrote: > > >> On Aug 19, 2016, at 2:44 PM, Jim Ingham via swift-lldb-dev >> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >> >>> >>> On Aug 19, 2016, at 2:26 PM, Daniel Dunbar via swift-lldb-dev >>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>> >>> >>>> On Aug 19, 2016, at 2:22 PM, Todd Fiala <todd.fi...@gmail.com >>>> <mailto:todd.fi...@gmail.com>> wrote: >>>> >>>> First off, I think the whole LLDB team can agree on the simple principle >>>> that tests that sometimes fail is not something we’re shooting for, so >>>> we’re on the same page with the general sentiment. >>>> >>>> But, rather than talking in sweeping generalizations, let’s take this >>>> exact failure as an example. It was a socket attempting to connect. It >>>> uses general socket timeouts. If it’s using a default, it will typically >>>> be something like 30 seconds. Per my previous comment, this is not >>>> something we see fail with any kind of frequency. We just had a failure on >>>> it now. Is the solution to say we want 35 seconds? 1 minute? 2 minutes? >>>> We’re using end-to-end testing here, and therefore we are subject to how >>>> a real system behaves. If we crank that up, there could be a time where >>>> we overload the server further, and the socket facility just fails. What >>>> do you do then, claim the test as unworthy? >>>> >>>> The challenge we run into on LLDB is that, while we do have a growing >>>> number of unit tests that are strict input/output verification, far too >>>> much of a functioning debugger’s operations are not covered adequately by >>>> that. So we have a large body of end to end tests. Sometimes those fail >>>> under heavy load. Sometimes those issues are ours - we have hidden races >>>> that are only exposed when the machine is grinding to a halt. Sometimes >>>> it is that the OS fails to handle requests for resources that the debugger >>>> needs. >>>> >>>> So it isn’t as simple as saying “file a bug to make the LLDB test suite >>>> more reliable.” (I wish it were!) >>> >>> This is specifically why I proposed a concrete strategy like running a >>> stress test of the test suite. That is actionable, and should shake some >>> bugs I would hope? >>> >>> Another actionable strategies (although one I'm not a big fan of) is >>> enhance the test suite to automatically re-attempt flaky tests. >> >> The test suite already does this. > > But only for tests that are explicitly marked as flakey - which this test > wasn't because - as Todd points out - it is not one of the usual known > culprits
I’m running the tests with —rerun-all-issues (per the swift build script). That actually will rerun all issues, not just ones marked flaky. Most of the upstream LLVM.org <http://llvm.org/> buidbots run in the mode where only flaky tests are rerun, but we’re not there yet across all our platforms on the GitHub side. > >> >> Jim >> >>> >>>> >>>> On the positive side, we have more resources going into improving our >>>> quality in the upcoming months, and we now have a dedicated quality >>>> engineer working with us. I expect this will help things as we move >>>> forward. And we’re looking to sink some of the system testing that >>>> currently exists end-to-end in LLDB to a more input-output style testing >>>> that we can do without needing the whole of LLDB (e.g. DWARF verification, >>>> DWARF type round-tripping through the compiler, etc. - Greg and Sean can >>>> speak more to what we’re doing there). >>> >>> Great! >>> >>> - Daniel >>> >>>> >>>> The net effect is that we should be hitting less of this noise. >>>> >>>> -Todd >>>> On August 19, 2016 at 2:10:14 PM, Daniel Dunbar via swift-lldb-dev >>>> (swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>) wrote: >>>> >>>>> The amount of flakiness in the LLDB tests is disturbing... is there a >>>>> high-level bug tracking improving this? It seems like it might be worth >>>>> running a stress test of the tests on a loaded machine to try and shake >>>>> out such problems. >>>>> >>>>> - Daniel >>>>> >>>>>> On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev >>>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>>> >>>>>>> >>>>>>> On Aug 19, 2016, at 1:42 PM, Enrico Granata <egran...@apple.com >>>>>>> <mailto:egran...@apple.com>> wrote: >>>>>>> >>>>>>>> >>>>>>>> On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev >>>>>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>>>>> >>>>>>>> It looks like this may be the packaging build: >>>>>>>> >>>>>>>>>> >>>>>>>>>> oss-swift-package-linux-ubuntu-15_10 >>>>>>>>>> >>>>>>>> >>>>>>>> It was this build: >>>>>>>> >>>>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7 >>>>>>>> >>>>>>>> <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7> >>>>>>>> >>>>>>>> It has since gone blue. >>>>>>>> >>>>>>>> I don’t think I recall ever seeing that test fail before in an >>>>>>>> intermittent way, but that might be what happened here. >>>>>>>> >>>>>>> >>>>>>> Given the failure mode, it does sound likely: >>>>>>> >>>>>>> >>>>>>> File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr >>>>>>> = self._sock.accept() timeout: timed out >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> If we see that again, we may be able to tweak the socket timeout to >>>>>> allow for heavily loaded test CI. >>>>>> >>>>>>>> -Todd >>>>>>>> >>>>>>>>> On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev >>>>>>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>>>>>> >>>>>>>>> Where is this failure being reported? I’m not seeing anything on >>>>>>>>> swift-3.0-branch or master on ci.swift.org <http://ci.swift.org/>. >>>>>>>>> >>>>>>>>> Kate Stone k8st...@apple.com <mailto:k8st...@apple.com> >>>>>>>>> Xcode Low Level Tools >>>>>>>>> >>>>>>>>>> On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev >>>>>>>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>>>>>>> >>>>>>>>>> + swift-lldb-dev group. >>>>>>>>>> >>>>>>>>>>> On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dun...@apple.com >>>>>>>>>>> <mailto:daniel_dun...@apple.com>> wrote: >>>>>>>>>>> >>>>>>>>>>> Failing in lldb: >>>>>>>>>>> >>>>>>>>>>> Command invoked: >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>>>>> --executable >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>>>>> --rerun-all-issues -C >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>>>>> TestStubReverseConnect.py >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>>>>> --event-add-entries worker_index=14:int Configuration: arch=x86_64 >>>>>>>>>>> compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>>>> Collected 2 tests >>>>>>>>>>> ====================================================================== >>>>>>>>>>> ERROR: test_reverse_connect_works_llgs >>>>>>>>>>> (TestStubReverseConnect.TestStubReverseConnect) >>>>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>>>> Traceback (most recent call last): File >>>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>>>>> line 89, in test_reverse_connect_works_llgs >>>>>>>>>>> self.reverse_connect_works() File >>>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>>>>> line 67, in reverse_connect_works (stub_socket, address) = >>>>>>>>>>> self.listener_socket.accept() File "/usr/lib/python2.7/socket.py", >>>>>>>>>>> line 206, in accept sock, addr = self._sock.accept() timeout: timed >>>>>>>>>>> out >>>>>>>>>>> Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9 >>>>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>>>> Ran 2 tests in 22.006s RESULT: FAILED (0 passes, 0 failures, 1 >>>>>>>>>>> errors, 1 skipped, 0 expected failures, 0 unexpected successes) >>>>>>>>>>> Session logs for test failures/errors/unexpected successes can be >>>>>>>>>>> found in directory '2016-08-18-15_15_07' >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [TestStubReverseConnect.py >>>>>>>>>>> FAILED] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Command invoked: /usr/bin/python >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>>>>> --executable >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>>>>> --rerun-all-issues -C >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>>>>> TestStubReverseConnect.py >>>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>>>>> --event-add-entries worker_index=14:int >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - Daniel >>>>>>>>>>> >>>>>>>>>>>> On Aug 18, 2016, at 9:18 PM, no-re...@swift.org >>>>>>>>>>>> <mailto:no-re...@swift.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>> [FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930] >>>>>>>>>>>> >>>>>>>>>>>> Build URL: >>>>>>>>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/ >>>>>>>>>>>> >>>>>>>>>>>> <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/> >>>>>>>>>>>> Project: oss-swift-package-linux-ubuntu-15_10 >>>>>>>>>>>> Date of build: Thu, 18 Aug 2016 20:26:52 -0700 >>>>>>>>>>>> Build duration: 51 min >>>>>>>>>>>> Identified problems: >>>>>>>>>>>> >>>>>>>>>>>> • Regression test failed: This build failed because a >>>>>>>>>>>> regression test in the test suite FAILed. Below is a list of all >>>>>>>>>>>> errors: >>>>>>>>>>>> • Indication 1 >>>>>>>>>>>> >>>>>>>>>>>> Changes >>>>>>>>>>>> >>>>>>>>>>>> • Commit 2ebab526bfd7a5ce6ef460bebee0399fe669612d by xi_ge: >>>>>>>>>>>> [FixCode] Apply >>>>>>>>>>>> coercion fixits on return statement and initialization >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> • edit: test/FixCode/fixits-apply-objc.swift >>>>>>>>>>>> • edit: test/FixCode/fixits-apply-objc.swift.result >>>>>>>>>>>> • edit: lib/Sema/CSDiag.cpp >>>>>>>>>>>> • edit: test/FixCode/fixits-apply.swift.result >>>>>>>>>>>> >>>>>>>>>>>> • Commit 6447a2d16edcfbceab940d58a321b3ef70525f1a by >>>>>>>>>>>> daniel_dunbar: >>>>>>>>>>>> [Basic] Update >>>>>>>>>>>> Thread shim to match changes on Linux side. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> • edit: Sources/Basic/Thread.swift >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> swift-lldb-dev mailing list >>>>>>>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> swift-lldb-dev mailing list >>>>>>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> swift-lldb-dev mailing list >>>>>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> - Enrico >>>>>>> 📩 egranata@.com <about:blank> ☎️ 27683 >>>>>> >>>>>> _______________________________________________ >>>>>> swift-lldb-dev mailing list >>>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>> >>>>> _______________________________________________ >>>>> swift-lldb-dev mailing list >>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>> >>> _______________________________________________ >>> swift-lldb-dev mailing list >>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >> >> _______________________________________________ >> swift-lldb-dev mailing list >> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> > > Thanks, > - Enrico > 📩 egranata@.com ☎️ 27683 > > _______________________________________________ > swift-lldb-dev mailing list > swift-lldb-...@swift.org > https://lists.swift.org/mailman/listinfo/swift-lldb-dev
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
