> On Aug 19, 2016, at 2:26 PM, Daniel Dunbar via swift-lldb-dev > <swift-lldb-...@swift.org> wrote: > > >> On Aug 19, 2016, at 2:22 PM, Todd Fiala <todd.fi...@gmail.com> wrote: >> >> First off, I think the whole LLDB team can agree on the simple principle >> that tests that sometimes fail is not something we’re shooting for, so we’re >> on the same page with the general sentiment. >> >> But, rather than talking in sweeping generalizations, let’s take this exact >> failure as an example. It was a socket attempting to connect. It uses >> general socket timeouts. If it’s using a default, it will typically be >> something like 30 seconds. Per my previous comment, this is not something >> we see fail with any kind of frequency. We just had a failure on it now. >> Is the solution to say we want 35 seconds? 1 minute? 2 minutes? We’re >> using end-to-end testing here, and therefore we are subject to how a real >> system behaves. If we crank that up, there could be a time where we >> overload the server further, and the socket facility just fails. What do >> you do then, claim the test as unworthy? >> >> The challenge we run into on LLDB is that, while we do have a growing number >> of unit tests that are strict input/output verification, far too much of a >> functioning debugger’s operations are not covered adequately by that. So we >> have a large body of end to end tests. Sometimes those fail under heavy >> load. Sometimes those issues are ours - we have hidden races that are only >> exposed when the machine is grinding to a halt. Sometimes it is that the OS >> fails to handle requests for resources that the debugger needs. >> >> So it isn’t as simple as saying “file a bug to make the LLDB test suite more >> reliable.” (I wish it were!) > > This is specifically why I proposed a concrete strategy like running a stress > test of the test suite. That is actionable, and should shake some bugs I > would hope? > > Another actionable strategies (although one I'm not a big fan of) is enhance > the test suite to automatically re-attempt flaky tests.
The test suite already does this. Jim > >> >> On the positive side, we have more resources going into improving our >> quality in the upcoming months, and we now have a dedicated quality engineer >> working with us. I expect this will help things as we move forward. And >> we’re looking to sink some of the system testing that currently exists >> end-to-end in LLDB to a more input-output style testing that we can do >> without needing the whole of LLDB (e.g. DWARF verification, DWARF type >> round-tripping through the compiler, etc. - Greg and Sean can speak more to >> what we’re doing there). > > Great! > > - Daniel > >> >> The net effect is that we should be hitting less of this noise. >> >> -Todd >> On August 19, 2016 at 2:10:14 PM, Daniel Dunbar via swift-lldb-dev >> (swift-lldb-...@swift.org) wrote: >> >>> The amount of flakiness in the LLDB tests is disturbing... is there a >>> high-level bug tracking improving this? It seems like it might be worth >>> running a stress test of the tests on a loaded machine to try and shake out >>> such problems. >>> >>> - Daniel >>> >>>> On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev >>>> <swift-lldb-...@swift.org> wrote: >>>> >>>>> >>>>> On Aug 19, 2016, at 1:42 PM, Enrico Granata <egran...@apple.com> wrote: >>>>> >>>>>> >>>>>> On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev >>>>>> <swift-lldb-...@swift.org> wrote: >>>>>> >>>>>> It looks like this may be the packaging build: >>>>>> >>>>>>>> >>>>>>>> oss-swift-package-linux-ubuntu-15_10 >>>>>>>> >>>>>> >>>>>> It was this build: >>>>>> >>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7 >>>>>> >>>>>> It has since gone blue. >>>>>> >>>>>> I don’t think I recall ever seeing that test fail before in an >>>>>> intermittent way, but that might be what happened here. >>>>>> >>>>> >>>>> Given the failure mode, it does sound likely: >>>>> >>>>> >>>>> File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr >>>>> = self._sock.accept() timeout: timed out >>>>> >>>>> >>>>> >>>>> >>>> >>>> If we see that again, we may be able to tweak the socket timeout to allow >>>> for heavily loaded test CI. >>>> >>>>>> -Todd >>>>>> >>>>>>> On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev >>>>>>> <swift-lldb-...@swift.org> wrote: >>>>>>> >>>>>>> Where is this failure being reported? I’m not seeing anything on >>>>>>> swift-3.0-branch or master on ci.swift.org. >>>>>>> >>>>>>> Kate Stone k8st...@apple.com >>>>>>> Xcode Low Level Tools >>>>>>> >>>>>>>> On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev >>>>>>>> <swift-lldb-...@swift.org> wrote: >>>>>>>> >>>>>>>> + swift-lldb-dev group. >>>>>>>> >>>>>>>>> On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dun...@apple.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Failing in lldb: >>>>>>>>> >>>>>>>>> Command invoked: >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>>> --executable >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>>> --rerun-all-issues -C >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>>> TestStubReverseConnect.py >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>>> --event-add-entries worker_index=14:int Configuration: arch=x86_64 >>>>>>>>> compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>> Collected 2 tests >>>>>>>>> ====================================================================== >>>>>>>>> ERROR: test_reverse_connect_works_llgs >>>>>>>>> (TestStubReverseConnect.TestStubReverseConnect) >>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>> Traceback (most recent call last): File >>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>>> line 89, in test_reverse_connect_works_llgs >>>>>>>>> self.reverse_connect_works() File >>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>>> line 67, in reverse_connect_works (stub_socket, address) = >>>>>>>>> self.listener_socket.accept() File "/usr/lib/python2.7/socket.py", >>>>>>>>> line 206, in accept sock, addr = self._sock.accept() timeout: timed >>>>>>>>> out >>>>>>>>> Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9 >>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>> Ran 2 tests in 22.006s RESULT: FAILED (0 passes, 0 failures, 1 >>>>>>>>> errors, 1 skipped, 0 expected failures, 0 unexpected successes) >>>>>>>>> Session logs for test failures/errors/unexpected successes can be >>>>>>>>> found in directory '2016-08-18-15_15_07' >>>>>>>>> >>>>>>>>> >>>>>>>>> [TestStubReverseConnect.py >>>>>>>>> FAILED] >>>>>>>>> >>>>>>>>> >>>>>>>>> Command invoked: /usr/bin/python >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>>> --executable >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>>> --rerun-all-issues -C >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>>> TestStubReverseConnect.py >>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>>> --event-add-entries worker_index=14:int >>>>>>>>> >>>>>>>>> >>>>>>>>> - Daniel >>>>>>>>> >>>>>>>>>> On Aug 18, 2016, at 9:18 PM, no-re...@swift.org wrote: >>>>>>>>>> >>>>>>>>>> [FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930] >>>>>>>>>> >>>>>>>>>> Build URL: >>>>>>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/ >>>>>>>>>> Project: oss-swift-package-linux-ubuntu-15_10 >>>>>>>>>> Date of build: Thu, 18 Aug 2016 20:26:52 -0700 >>>>>>>>>> Build duration: 51 min >>>>>>>>>> Identified problems: >>>>>>>>>> >>>>>>>>>> • Regression test failed: This build failed because a >>>>>>>>>> regression test in the test suite FAILed. Below is a list of all >>>>>>>>>> errors: >>>>>>>>>> • Indication 1 >>>>>>>>>> >>>>>>>>>> Changes >>>>>>>>>> >>>>>>>>>> • Commit 2ebab526bfd7a5ce6ef460bebee0399fe669612d by xi_ge: >>>>>>>>>> [FixCode] Apply >>>>>>>>>> coercion fixits on return statement and initialization >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> • edit: test/FixCode/fixits-apply-objc.swift >>>>>>>>>> • edit: test/FixCode/fixits-apply-objc.swift.result >>>>>>>>>> • edit: lib/Sema/CSDiag.cpp >>>>>>>>>> • edit: test/FixCode/fixits-apply.swift.result >>>>>>>>>> >>>>>>>>>> • Commit 6447a2d16edcfbceab940d58a321b3ef70525f1a by >>>>>>>>>> daniel_dunbar: >>>>>>>>>> [Basic] Update >>>>>>>>>> Thread shim to match changes on Linux side. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> • edit: Sources/Basic/Thread.swift >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> swift-lldb-dev mailing list >>>>>>>> swift-lldb-...@swift.org >>>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>>> >>>>>>> _______________________________________________ >>>>>>> swift-lldb-dev mailing list >>>>>>> swift-lldb-...@swift.org >>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>> >>>>>> _______________________________________________ >>>>>> swift-lldb-dev mailing list >>>>>> swift-lldb-...@swift.org >>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>> >>>>> >>>>> Thanks, >>>>> - Enrico >>>>> 📩 egranata@.com ☎️ 27683 >>>> >>>> _______________________________________________ >>>> swift-lldb-dev mailing list >>>> swift-lldb-...@swift.org >>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>> >>> _______________________________________________ >>> swift-lldb-dev mailing list >>> swift-lldb-...@swift.org >>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev > > _______________________________________________ > swift-lldb-dev mailing list > swift-lldb-...@swift.org > https://lists.swift.org/mailman/listinfo/swift-lldb-dev _______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
