> On Aug 19, 2016, at 2:44 PM, Jim Ingham via swift-lldb-dev > <swift-lldb-...@swift.org> wrote: > >> >> On Aug 19, 2016, at 2:26 PM, Daniel Dunbar via swift-lldb-dev >> <swift-lldb-...@swift.org> wrote: >> >> >>> On Aug 19, 2016, at 2:22 PM, Todd Fiala <todd.fi...@gmail.com> wrote: >>> >>> First off, I think the whole LLDB team can agree on the simple principle >>> that tests that sometimes fail is not something we’re shooting for, so >>> we’re on the same page with the general sentiment. >>> >>> But, rather than talking in sweeping generalizations, let’s take this exact >>> failure as an example. It was a socket attempting to connect. It uses >>> general socket timeouts. If it’s using a default, it will typically be >>> something like 30 seconds. Per my previous comment, this is not something >>> we see fail with any kind of frequency. We just had a failure on it now. >>> Is the solution to say we want 35 seconds? 1 minute? 2 minutes? We’re >>> using end-to-end testing here, and therefore we are subject to how a real >>> system behaves. If we crank that up, there could be a time where we >>> overload the server further, and the socket facility just fails. What do >>> you do then, claim the test as unworthy? >>> >>> The challenge we run into on LLDB is that, while we do have a growing >>> number of unit tests that are strict input/output verification, far too >>> much of a functioning debugger’s operations are not covered adequately by >>> that. So we have a large body of end to end tests. Sometimes those fail >>> under heavy load. Sometimes those issues are ours - we have hidden races >>> that are only exposed when the machine is grinding to a halt. Sometimes it >>> is that the OS fails to handle requests for resources that the debugger >>> needs. >>> >>> So it isn’t as simple as saying “file a bug to make the LLDB test suite >>> more reliable.” (I wish it were!) >> >> This is specifically why I proposed a concrete strategy like running a >> stress test of the test suite. That is actionable, and should shake some >> bugs I would hope? >> >> Another actionable strategies (although one I'm not a big fan of) is enhance >> the test suite to automatically re-attempt flaky tests. > > The test suite already does this.
But only for tests that are explicitly marked as flakey - which this test wasn't because - as Todd points out - it is not one of the usual known culprits > > Jim > >> >>> >>> On the positive side, we have more resources going into improving our >>> quality in the upcoming months, and we now have a dedicated quality >>> engineer working with us. I expect this will help things as we move >>> forward. And we’re looking to sink some of the system testing that >>> currently exists end-to-end in LLDB to a more input-output style testing >>> that we can do without needing the whole of LLDB (e.g. DWARF verification, >>> DWARF type round-tripping through the compiler, etc. - Greg and Sean can >>> speak more to what we’re doing there). >> >> Great! >> >> - Daniel >> >>> >>> The net effect is that we should be hitting less of this noise. >>> >>> -Todd >>> On August 19, 2016 at 2:10:14 PM, Daniel Dunbar via swift-lldb-dev >>> (swift-lldb-...@swift.org) wrote: >>> >>>> The amount of flakiness in the LLDB tests is disturbing... is there a >>>> high-level bug tracking improving this? It seems like it might be worth >>>> running a stress test of the tests on a loaded machine to try and shake >>>> out such problems. >>>> >>>> - Daniel >>>> >>>>> On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev >>>>> <swift-lldb-...@swift.org> wrote: >>>>> >>>>>> >>>>>> On Aug 19, 2016, at 1:42 PM, Enrico Granata <egran...@apple.com> wrote: >>>>>> >>>>>>> >>>>>>> On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev >>>>>>> <swift-lldb-...@swift.org> wrote: >>>>>>> >>>>>>> It looks like this may be the packaging build: >>>>>>> >>>>>>>>> >>>>>>>>> oss-swift-package-linux-ubuntu-15_10 >>>>>>>>> >>>>>>> >>>>>>> It was this build: >>>>>>> >>>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7 >>>>>>> >>>>>>> It has since gone blue. >>>>>>> >>>>>>> I don’t think I recall ever seeing that test fail before in an >>>>>>> intermittent way, but that might be what happened here. >>>>>>> >>>>>> >>>>>> Given the failure mode, it does sound likely: >>>>>> >>>>>> >>>>>> File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr >>>>>> = self._sock.accept() timeout: timed out >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> If we see that again, we may be able to tweak the socket timeout to allow >>>>> for heavily loaded test CI. >>>>> >>>>>>> -Todd >>>>>>> >>>>>>>> On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev >>>>>>>> <swift-lldb-...@swift.org> wrote: >>>>>>>> >>>>>>>> Where is this failure being reported? I’m not seeing anything on >>>>>>>> swift-3.0-branch or master on ci.swift.org. >>>>>>>> >>>>>>>> Kate Stone k8st...@apple.com >>>>>>>> Xcode Low Level Tools >>>>>>>> >>>>>>>>> On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev >>>>>>>>> <swift-lldb-...@swift.org> wrote: >>>>>>>>> >>>>>>>>> + swift-lldb-dev group. >>>>>>>>> >>>>>>>>>> On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dun...@apple.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Failing in lldb: >>>>>>>>>> >>>>>>>>>> Command invoked: >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>>>> --executable >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>>>> --rerun-all-issues -C >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>>>> TestStubReverseConnect.py >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>>>> --event-add-entries worker_index=14:int Configuration: arch=x86_64 >>>>>>>>>> compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>>> Collected 2 tests >>>>>>>>>> ====================================================================== >>>>>>>>>> ERROR: test_reverse_connect_works_llgs >>>>>>>>>> (TestStubReverseConnect.TestStubReverseConnect) >>>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>>> Traceback (most recent call last): File >>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>>>> line 89, in test_reverse_connect_works_llgs >>>>>>>>>> self.reverse_connect_works() File >>>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>>>> line 67, in reverse_connect_works (stub_socket, address) = >>>>>>>>>> self.listener_socket.accept() File "/usr/lib/python2.7/socket.py", >>>>>>>>>> line 206, in accept sock, addr = self._sock.accept() timeout: timed >>>>>>>>>> out >>>>>>>>>> Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9 >>>>>>>>>> ---------------------------------------------------------------------- >>>>>>>>>> Ran 2 tests in 22.006s RESULT: FAILED (0 passes, 0 failures, 1 >>>>>>>>>> errors, 1 skipped, 0 expected failures, 0 unexpected successes) >>>>>>>>>> Session logs for test failures/errors/unexpected successes can be >>>>>>>>>> found in directory '2016-08-18-15_15_07' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [TestStubReverseConnect.py >>>>>>>>>> FAILED] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Command invoked: /usr/bin/python >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>>>> --executable >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>>>> --rerun-all-issues -C >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>>>> TestStubReverseConnect.py >>>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>>>> --event-add-entries worker_index=14:int >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - Daniel >>>>>>>>>> >>>>>>>>>>> On Aug 18, 2016, at 9:18 PM, no-re...@swift.org wrote: >>>>>>>>>>> >>>>>>>>>>> [FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930] >>>>>>>>>>> >>>>>>>>>>> Build URL: >>>>>>>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/ >>>>>>>>>>> Project: oss-swift-package-linux-ubuntu-15_10 >>>>>>>>>>> Date of build: Thu, 18 Aug 2016 20:26:52 -0700 >>>>>>>>>>> Build duration: 51 min >>>>>>>>>>> Identified problems: >>>>>>>>>>> >>>>>>>>>>> • Regression test failed: This build failed because a >>>>>>>>>>> regression test in the test suite FAILed. Below is a list of all >>>>>>>>>>> errors: >>>>>>>>>>> • Indication 1 >>>>>>>>>>> >>>>>>>>>>> Changes >>>>>>>>>>> >>>>>>>>>>> • Commit 2ebab526bfd7a5ce6ef460bebee0399fe669612d by xi_ge: >>>>>>>>>>> [FixCode] Apply >>>>>>>>>>> coercion fixits on return statement and initialization >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> • edit: test/FixCode/fixits-apply-objc.swift >>>>>>>>>>> • edit: test/FixCode/fixits-apply-objc.swift.result >>>>>>>>>>> • edit: lib/Sema/CSDiag.cpp >>>>>>>>>>> • edit: test/FixCode/fixits-apply.swift.result >>>>>>>>>>> >>>>>>>>>>> • Commit 6447a2d16edcfbceab940d58a321b3ef70525f1a by >>>>>>>>>>> daniel_dunbar: >>>>>>>>>>> [Basic] Update >>>>>>>>>>> Thread shim to match changes on Linux side. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> • edit: Sources/Basic/Thread.swift >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> swift-lldb-dev mailing list >>>>>>>>> swift-lldb-...@swift.org >>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> swift-lldb-dev mailing list >>>>>>>> swift-lldb-...@swift.org >>>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>>> >>>>>>> _______________________________________________ >>>>>>> swift-lldb-dev mailing list >>>>>>> swift-lldb-...@swift.org >>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>> >>>>>> >>>>>> Thanks, >>>>>> - Enrico >>>>>> 📩 egranata@.com ☎️ 27683 >>>>> >>>>> _______________________________________________ >>>>> swift-lldb-dev mailing list >>>>> swift-lldb-...@swift.org >>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>> >>>> _______________________________________________ >>>> swift-lldb-dev mailing list >>>> swift-lldb-...@swift.org >>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >> >> _______________________________________________ >> swift-lldb-dev mailing list >> swift-lldb-...@swift.org >> https://lists.swift.org/mailman/listinfo/swift-lldb-dev > > _______________________________________________ > swift-lldb-dev mailing list > swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> > https://lists.swift.org/mailman/listinfo/swift-lldb-dev > <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> Thanks, - Enrico 📩 egranata@.com ☎️ 27683
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
