On Thu, Dec 8, 2016 at 12:11 PM, Martin Polednik <mpoled...@redhat.com>
wrote:
> On 08/12/16 11:26 +0200, Edward Haas wrote:
>
>> On Thu, Dec 8, 2016 at 10:12 AM, Martin Polednik <mpoled...@redhat.com>
>> wrote:
>>
>> On 08/12/16 09:28 +0200, Edward Haas wrote:
>>>
>>> On Wed, Dec 7, 2016 at 11:54 PM, Nir Soffer <nsof...@redhat.com> wrote:
>>>>
>>>> broken_on_ci is uses default name="OVIRT_CI", to mark it also for
>>>>
>>>>> travis, we need another broken_on_ci with name="TRAVIS_CI".
>>>>>
>>>>> Maybe this test should run only if nm is active on the machine?
>>>>>
>>>>>
>>>>> We need the test to always run when expected.
>>>> If NM is not running, the test will not run (silently) and we will never
>>>> know if there is a problem or not.
>>>>
>>>> It is not convenient to mark each CI type as broken, why the test code
>>>> needs to know we have multiple
>>>> CI/s?
>>>>
>>>>
>>> I believe this is great point - we should just mark the test as broken
>>> on *any* CI to create a pressure to get it fixed.
>>>
>>> Slight off-topic addition: I don't understand why patch marking a test
>>> as broken on CI takes more than 5 minutes to get merged in when given
>>> pointer to the failure.
>>>
>>
>>
>> Because it is a wrong approach. :)
>> If a test fails, it is a smell that something bad happens and we may have
>> a
>> production problem.
>> So before marking and excluding the test, one should feel very guilty that
>> this check is no longer covered and better understand why it fails.
>>
>
> Those are 1-in-N case breakages. The fact the test is unstable should
> be noted by a maintainer, but shouldn't block any other patches or
> series (which is what often happens).
>
> I don't feel any guilt marking bad test as bad.
How do you know if it is a bad test, bad production code or perhaps even a
bad
3rd party library/driver etc...?
In most cases, the test just checks that what we expect from the code
indeed happens.
If our expectations are not valid, we have a bad test, but if our
expectations are valid, killing
the test just hides the problem.
Regarding the blocking part, I agree, it is indeed a problem. Except
splitting a project into
smaller pieces I don't know of a good solution for that. (introducing other
problems)
(When you are small, you can run faster. I think this is the micro-services
way)
>
>
>
>>
>>>
>>> Currently, we run on CI tests that are not marked as 'functional'.
>>>
>>>> Perhaps we need another test type that can be mark not to run on simple
>>>> CI.
>>>> "power-integration", "super-integration"?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Wed, Dec 7, 2016 at 11:23 PM, Dan Kenigsberg <dan...@redhat.com>
>>>>> wrote:
>>>>> > On Wed, Dec 7, 2016 at 2:03 PM, Nir Soffer <nsof...@redhat.com>
>>>>> wrote:
>>>>> >> Looks like we need @brokentest("reason...", name="TRAVIC_CI") on
>>>>> this:
>>>>> >
>>>>> > Odd, the code already has
>>>>> >
>>>>> > @broken_on_ci('NetworkManager should not be started on CI nodes')
>>>>> >
>>>>> >
>>>>> >>
>>>>> >> See https://travis-ci.org/oVirt/vdsm/jobs/181933329
>>>>> >>
>>>>> >> ============================================================
>>>>> ==========
>>>>> >>
>>>>> >> ERROR: test suite for <module 'network.nmdbus_test' from
>>>>> >> '/vdsm/tests/network/nmdbus_test.py'>
>>>>> >>
>>>>> >> ------------------------------------------------------------
>>>>> ----------
>>>>> >>
>>>>> >> Traceback (most recent call last):
>>>>> >>
>>>>> >> File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209,
>>>>> in
>>>>> run
>>>>> >>
>>>>> >> self.setUp()
>>>>> >>
>>>>> >> File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292,
>>>>> in
>>>>> setUp
>>>>> >>
>>>>> >> self.setupContext(ancestor)
>>>>> >>
>>>>> >> File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315,
>>>>> in
>>>>> >> setupContext
>>>>> >>
>>>>> >> try_run(context, names)
>>>>> >>
>>>>> >> File "/usr/lib/python2.7/site-packages/nose/util.py", line 471,
>>>>> in
>>>>> try_run
>>>>> >>
>>>>> >> return func()
>>>>> >>
>>>>> >> File "/vdsm/tests/testValidation.py", line 191, in wrapper
>>>>> >>
>>>>> >> return f(*args, **kwargs)
>>>>> >>
>>>>> >> File "/vdsm/tests/testValidation.py", line 97, in wrapper
>>>>> >>
>>>>> >> return f(*args, **kwargs)
>>>>> >>
>>>>> >> File "/vdsm/tests/network/nmdbus_test.py", line 48, in
>>>>> setup_module
>>>>> >>
>>>>> >> NMDbus.init()
>>>>> >>
>>>>> >> File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in
>>>>> init
>>>>> >>
>>>>> >> NMDbus.bus = dbus.SystemBus()
>>>>> >>
>>>>> >> File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line
>>>>> 194,
>>>>> in __new__
>>>>> >>
>>>>> >> private=private)
>>>>> >>
>>>>> >> File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line
>>>>> 100,
>>>>> in __new__
>>>>> >>
>>>>> >> bus = BusConnection.__new__(subclass, bus_type,
>>>>> mainloop=mainloop)
>>>>> >>
>>>>> >> File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122,
>>>>> in
>>>>> __new__
>>>>> >>
>>>>> >> bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
>>>>> >>
>>>>> >> DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to
>>>>> >> connect to socket /var/run/dbus/system_bus_socket: No such file or
>>>>> >> directory
>>>>> >>
>>>>> >> -------------------- >> begin captured logging <<
>>>>> --------------------
>>>>> >>
>>>>> >> 2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset
>>>>> >> --cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None)
>>>>> >> (commands:69)
>>>>> >>
>>>>> >> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED: <err> =
>>>>> >> 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> =
>>>>> 1
>>>>> >> (commands:93)
>>>>> >>
>>>>> >> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset
>>>>> >> --cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None)
>>>>> >> (commands:69)
>>>>> >>
>>>>> >> 2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED: <err> =
>>>>> >> 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> =
>>>>> 1
>>>>> >> (commands:93)
>>>>> >>
>>>>> >> --------------------- >> end captured logging <<
>>>>> ---------------------
>>>>>
>>>>>
>>>>> _______________________________________________
>>>
>>>> Devel mailing list
>>>> Devel@ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>>
>>>
>>>
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.phx.ovirt.org/mailman/listinfo/devel
