Re: Quote of the day

[Ethan Furman]

On 05/18/2016 05:43 PM, Steven D'Aprano wrote:

On Thu, 19 May 2016 09:30 am, Ethan Furman wrote:


On 05/18/2016 03:52 PM, Gregory Ewing wrote:

Ned Batchelder wrote:



I'm not sure how
the test runner could determine that it was empty.  I guess it could
introspect the test function to see if it had any real code in it,


Then people would just get clever at putting dummy code
in the test that fools the test runner but doesn't really
test anything...


Some would have, sure.

Either way, it's a solved issue now because we (finally ;) have the
@skip decorator.


That only solves the problem for responsible, decent developers.


Since I'm not a manager (and when I was, I wasn't the PHB type), 
responsible, decent developers are where I focus my attention.


--
~Ethan~

--
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Steven D'Aprano]

On Thu, 19 May 2016 09:30 am, Ethan Furman wrote:

> On 05/18/2016 03:52 PM, Gregory Ewing wrote:
>> Ned Batchelder wrote:
> 
>>> I'm not sure how
>>> the test runner could determine that it was empty.  I guess it could
>>> introspect the test function to see if it had any real code in it,
>>
>> Then people would just get clever at putting dummy code
>> in the test that fools the test runner but doesn't really
>> test anything...
> 
> Some would have, sure.
> 
> Either way, it's a solved issue now because we (finally ;) have the
> @skip decorator.

That only solves the problem for responsible, decent developers.

But I guarantee you that, right now, as we speak, there is some poor schmuck
out there whose Pointy Haired Boss has given him a Key Performance
Indicator of X tests passing (not failing or skipped) per week, and he's
responding by writing tests which pass by not testing anything.



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Ethan Furman]

On 05/18/2016 03:52 PM, Gregory Ewing wrote:

Ned Batchelder wrote:



I'm not sure how
the test runner could determine that it was empty.  I guess it could
introspect the test function to see if it had any real code in it,


Then people would just get clever at putting dummy code
in the test that fools the test runner but doesn't really
test anything...


Some would have, sure.

Either way, it's a solved issue now because we (finally ;) have the 
@skip decorator.


--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Gregory Ewing]

Ned Batchelder wrote:

I'm not sure how
the test runner could determine that it was empty.  I guess it could
introspect the test function to see if it had any real code in it,


Then people would just get clever at putting dummy code
in the test that fools the test runner but doesn't really
test anything...

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Steven D'Aprano]

On Thu, 19 May 2016 02:05 am, Ethan Furman wrote:

> On 05/18/2016 08:35 AM, Thomas Mlynarczyk wrote:
>> On 18/05/16 17:21, Ned Batchelder wrote:
> 
>>> Ideally, an empty test wouldn't be a success, but I'm not sure how
>>> the test runner could determine that it was empty.  I guess it could
>>> introspect the test function to see if it had any real code in it,
>>> but I don't know of a test runner that does that.
>>
>> Simple: a function which does not produce at least one "failure" or
>> "pass" does not test anything. No need to introspect the code. Just
>> check if the total score of failures and passes has changed after the
>> function was run.

I think you have misunderstood how unittest currently works. A do-nothing
test already counts as a pass. Here's a dumb test which is pointless,
followed by an even dumber one that does literally nothing:


[steve@ando ~]$ cat dumbtest.py
import unittest

class MyTest(unittest.TestCase):
def test_something(self):
self.assertEqual(100, 100.0)
def test_nothing(self):
pass


And now watch as both the pointless and the do-nothing tests count as
passed:


[steve@ando ~]$ python -m unittest --verbose dumbtest
test_nothing (dumbtest.MyTest) ... ok
test_something (dumbtest.MyTest) ... ok

--
Ran 2 tests in 0.001s

OK



So to start with, for your solution to be workable, we'd have to change the
way unittest decides what is a success and what isn't.

 
> Not so simple: I have tests that do nothing besides build objects.  If
> building the objects raises no errors the test passed.

It wouldn't be hard to add a "success" method to unittest, so that after
building the object you just call self.success() to flag it as passing.

But now the obvious way to have fake unittests is:

def test_nothing(self):
self.success()


The problem here is not a technical problem. It is a cultural or human
problem: somebody, due to malice, incompetence, overwork, ignorance or
stupidity, wrote a fake test that didn't actually test anything. Maybe
their intentions were good, and they meant for it to do something and it
just got forgotten... or maybe they deliberately thought that they could
satisfy the letter of the requirement "must have unit tests" without
putting in the hard work to satisfy the spirit of it.

Either way, until the testing framework contains enough artificial
intelligence to actually reason about whether the test is *useful* or not,
there's no technological way to solve this problem. You need a person[1]
intelligent enough to make a judgement "wait a minute, this code doesn't
test anything useful".

And that's a hard problem. Even human beings do poorly at that. The idea
that the test framework could solve it is naive.


> Although, for the benefit of empty tests not passing I could add a
> do-nothing assert:
> 
>self.assertTrue(created_obj)
> 
> (it's a do-nothing because if the object wasn't created the test would
> have already failed).

It wouldn't have failed. It would have raised an exception, which is
different. (Curse you English, we need more words for describing kinds of
failure!!!)

unittest supports four different test results:

- pass
- fail
- error (raise an exception)
- skip

which print as . F E S respectively. The tests you're describing will print
as E rather than F, which is better than a failure. A test failure is code
that silently does the wrong thing:

"I find it amusing when novice programmers believe their
 main job is preventing programs from crashing. ... More
 experienced programmers realize that correct code is
 great, code that crashes could use improvement, but
 incorrect code that doesn’t crash is a horrible nightmare."
 -- Chris Smith

while an E means that your code will cleverly raise an exception instead of
doing the wrong thing :-)






[1] Human or machine person, I don't care.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Ethan Furman]

On 05/18/2016 08:35 AM, Thomas Mlynarczyk wrote:

On 18/05/16 17:21, Ned Batchelder wrote:



Ideally, an empty test wouldn't be a success, but I'm not sure how
the test runner could determine that it was empty.  I guess it could
introspect the test function to see if it had any real code in it,
but I don't know of a test runner that does that.


Simple: a function which does not produce at least one "failure" or
"pass" does not test anything. No need to introspect the code. Just
check if the total score of failures and passes has changed after the
function was run.


Not so simple: I have tests that do nothing besides build objects.  If 
building the objects raises no errors the test passed.


Although, for the benefit of empty tests not passing I could add a 
do-nothing assert:


  self.assertTrue(created_obj)

(it's a do-nothing because if the object wasn't created the test would 
have already failed).


--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Ned Batchelder]

On Wednesday, May 18, 2016 at 11:36:03 AM UTC-4, Thomas Mlynarczyk wrote:
> On 18/05/16 17:21, Ned Batchelder wrote:
> > Ideally, an empty test wouldn't be a success, but I'm not sure how
> > the test runner could determine that it was empty.  I guess it could
> > introspect the test function to see if it had any real code in it,
> > but I don't know of a test runner that does that.
> 
> Simple: a function which does not produce at least one "failure" or
> "pass" does not test anything. No need to introspect the code. Just
> check if the total score of failures and passes has changed after the
> function was run.

For test frameworks that use explicit assertion methods (unittest has
self.assertEqual, for example), that could work.  I'm not sure whether
py.test and the other "bare assert" frameworks have the instrumentation
to make that possible.

--Ned.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Thomas Mlynarczyk]

On 18/05/16 17:21, Ned Batchelder wrote:
> Ideally, an empty test wouldn't be a success, but I'm not sure how
> the test runner could determine that it was empty.  I guess it could
> introspect the test function to see if it had any real code in it,
> but I don't know of a test runner that does that.

Simple: a function which does not produce at least one "failure" or
"pass" does not test anything. No need to introspect the code. Just
check if the total score of failures and passes has changed after the
function was run.

Thomas

-- 
Ce n'est pas parce qu'ils sont nombreux à avoir tort qu'ils ont raison!
(Coluche)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Ned Batchelder]

On Wednesday, May 18, 2016 at 8:06:11 AM UTC-4, Thomas Mlynarczyk wrote:
> On 17/05/16 12:39, Cem Karan wrote:
> > Just downloaded and used a library that came with unit tests, which all 
> > passed.
> > [...]
> > I discovered they had commented out the bodies of some of the unit
> tests...
> 
> Shouldn't the unit test framework have those "empty" tests reported as
> "todo"/"incomplete" or whatever? (I know that PHPUnit reports such tests
> as "passed" which I find utterly wrong.)

The xUnit pattern is that if a test runs without an error or a failed
assertion, then the test passes.  An empty test function that does
nothing will meet this criterion, so it passes.

Ideally, an empty test wouldn't be a success, but I'm not sure how
the test runner could determine that it was empty.  I guess it could
introspect the test function to see if it had any real code in it,
but I don't know of a test runner that does that.

--Ned.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Chris Angelico]

On Wed, May 18, 2016 at 10:05 PM, Thomas Mlynarczyk
 wrote:
> On 17/05/16 12:39, Cem Karan wrote:
>> Just downloaded and used a library that came with unit tests, which all 
>> passed.
>> [...]
>> I discovered they had commented out the bodies of some of the unit
> tests...
>
> Shouldn't the unit test framework have those "empty" tests reported as
> "todo"/"incomplete" or whatever? (I know that PHPUnit reports such tests
> as "passed" which I find utterly wrong.)

In Python, the unittest framework allows you to 'skip' tests for any
reason. That would be the best.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Thomas Mlynarczyk]

On 17/05/16 12:39, Cem Karan wrote:
> Just downloaded and used a library that came with unit tests, which all 
> passed.
> [...]
> I discovered they had commented out the bodies of some of the unit
tests...

Shouldn't the unit test framework have those "empty" tests reported as
"todo"/"incomplete" or whatever? (I know that PHPUnit reports such tests
as "passed" which I find utterly wrong.)

Thomas

-- 
Ce n'est pas parce qu'ils sont nombreux à avoir tort qu'ils ont raison!
(Coluche)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Marko Rauhamaa]

Michael Torrie :

> On 05/17/2016 08:27 AM, Paul Rudin wrote:
>> Marko Rauhamaa  writes:
>>> That's a long time to be without a product to sell.
>> 
>> But you do have the option of building a kernel incorporating your fix
>> and using that.
>
> Sure as an individual end user that may be the best option. But not
> necessarily for a business.

Correct, the answer would be no.

> The cost of doing that could be prohibitive.

Really, the customer would simply refuse to do it. They are not in the
business of building kernels. Also, they would immediately fall out of
any kind of distro support if they improvised with their own kernel.

> When I did IT professionally, our policy with regards to Linux was to
> stick with existing packages from a known set of (mostly) official
> channels and to discourage any installing of libraries and frameworks
> from source. Allowing packages to be installed from source was just a
> maintenance nightmare. RPM (or deb or whatever) brings at least a tiny
> bit of stability and consistency.

Exactly.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Michael Torrie]

On 05/17/2016 08:27 AM, Paul Rudin wrote:
> Marko Rauhamaa  writes:
>> That's a long time to be without a product to sell.
> 
> But you do have the option of building a kernel incorporating your fix
> and using that.

Sure as an individual end user that may be the best option. But not
necessarily for a business.  The cost of doing that could be
prohibitive.  Sometimes we forget just how costly open source software
can be (really *all* software).  They can either deal with lost revenue
waiting, or they can budget a tremendous amount of money, time, and
effort to support their own kernel which would entail doing updates, QA
testing, etc.  Letting the upstream vendor do all that (their core
business after all) is often the least costly option.  Though it sounds
like they've already spent a lot of money doing QA to identify this bug.

When I did IT professionally, our policy with regards to Linux was to
stick with existing packages from a known set of (mostly) official
channels and to discourage any installing of libraries and frameworks
from source.  Allowing packages to be installed from source was just a
maintenance nightmare.  RPM (or deb or whatever) brings at least a tiny
bit of stability and consistency.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Paul Rudin]

Marko Rauhamaa  writes:

> Paul Rudin :
>
>> Marko Rauhamaa  writes:
>>> The feeling of powerlessness can be crushing when you depend on a
>>> third-party component that is broken with no fix in sight.
>>
>> Presumably it depends on whether you have the source for the third
>> party component...
>
> Just having such an experience. The linux kernel has a critical bug in a
> major distribution (who shall be left unnamed here) that has been fixed
> in a later kernel version.
>
> Thanks to linux being free software, I managed to pin down the root
> cause after more than a month of debugging. I sent a bug report to the
> linux vendor and attached a tiny patch. The vendor has graciously agreed
> to consider releasing an update in the summer (we are in the process of
> verifying the fix).
>
> The problem was first detected in December. A semi-reliable reproduction
> was discovered in early February. The root cause and proposed fix was
> identified mid-March. A vendor fix will likely come out by the end of
> June.
>
> That's a long time to be without a product to sell.
>

But you do have the option of building a kernel incorporating your fix
and using that.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Marko Rauhamaa]

Paul Rudin :

> Marko Rauhamaa  writes:
>> The feeling of powerlessness can be crushing when you depend on a
>> third-party component that is broken with no fix in sight.
>
> Presumably it depends on whether you have the source for the third
> party component...

Just having such an experience. The linux kernel has a critical bug in a
major distribution (who shall be left unnamed here) that has been fixed
in a later kernel version.

Thanks to linux being free software, I managed to pin down the root
cause after more than a month of debugging. I sent a bug report to the
linux vendor and attached a tiny patch. The vendor has graciously agreed
to consider releasing an update in the summer (we are in the process of
verifying the fix).

The problem was first detected in December. A semi-reliable reproduction
was discovered in early February. The root cause and proposed fix was
identified mid-March. A vendor fix will likely come out by the end of
June.

That's a long time to be without a product to sell.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Chris Angelico]

On Tue, May 17, 2016 at 7:54 PM, Paul Rudin  wrote:
>> Also:
>>
>>With a third party solution I don't need to fix the bugs.
>>
>>But with an in-house solution I at least *can* fix the bugs.
>>
>> The feeling of powerlessness can be crushing when you depend on a
>> third-party component that is broken with no fix in sight.
>>
>>
>
> Presumably it depends on whether you have the source for the third party
> component...

Yes and no. A student of mine asked me how hard it would be to use in
Python a service that provided Java, .NET, and a couple of other SDKs,
but not Python. Source is available for them, sure, but they're so
massive and complicated that it's utterly impractical. (I had been
hoping the SDKs were basically just offering a friendly API to an
underlying HTTP-based service, but no.) So the options were (1) use
Jython so the Java SDK became usable, or (2) fire off a subprocess
that does the work and pipes it back to your app, or (3) spend about
fifty years porting a gigantic lot of code to a new language. And if
there'd been bugs in any of the code, well, options 1 and 2 mean my
student (who knows Python but none of the SDK languages on offer)
would be completely unable to fix it, source or no source - and with
option 3, it'd make the port virtually impossible.

Having the source available is great. It tells you which projects you
DON'T want to touch.

ChrisA
fully aware that some of his projects will be in other people's "DON'T
want to touch" boxes
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Cem Karan]

On May 17, 2016, at 4:30 AM, Marko Rauhamaa  wrote:

> Radek Holý :
> 
>> 2016-05-17 9:50 GMT+02:00 Steven D'Aprano <
>> steve+comp.lang.pyt...@pearwood.info>:
>> 
>>> Overhead in the office today:
>>> 
>>> "I don't have time to learn an existing library - much faster to make
>>> my own mistakes!"
>> 
>> *THUMBS UP* At least they are aware of that "own mistakes" part... Not
>> like my employer...
> 
> Also:
> 
>   With a third party solution I don't need to fix the bugs.
> 
>   But with an in-house solution I at least *can* fix the bugs.
> 
> The feeling of powerlessness can be crushing when you depend on a
> third-party component that is broken with no fix in sight.

+1000 on this one.  Just downloaded and used a library that came with unit 
tests, which all passed.  When I started using it, I kept getting odd errors.  
Digging into it, I discovered they had commented out the bodies of some of the 
unit tests... glad it was open source, at least I *could* dig into the code and 
figure out what was going on :/

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Paul Rudin]

Marko Rauhamaa  writes:

> Radek Holý :
>
>> 2016-05-17 9:50 GMT+02:00 Steven D'Aprano <
>> steve+comp.lang.pyt...@pearwood.info>:
>>
>>> Overhead in the office today:
>>>
>>> "I don't have time to learn an existing library - much faster to make
>>> my own mistakes!"
>>
>> *THUMBS UP* At least they are aware of that "own mistakes" part... Not
>> like my employer...
>
> Also:
>
>With a third party solution I don't need to fix the bugs.
>
>But with an in-house solution I at least *can* fix the bugs.
>
> The feeling of powerlessness can be crushing when you depend on a
> third-party component that is broken with no fix in sight.
>
>

Presumably it depends on whether you have the source for the third party
component...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Sivan Greenberg]

But isn't that counter wise to batteries included? :)

On Tue, May 17, 2016 at 11:30 AM, Marko Rauhamaa  wrote:

> Radek Holý :
>
> > 2016-05-17 9:50 GMT+02:00 Steven D'Aprano <
> > steve+comp.lang.pyt...@pearwood.info>:
> >
> >> Overhead in the office today:
> >>
> >> "I don't have time to learn an existing library - much faster to make
> >> my own mistakes!"
> >
> > *THUMBS UP* At least they are aware of that "own mistakes" part... Not
> > like my employer...
>
> Also:
>
>With a third party solution I don't need to fix the bugs.
>
>But with an in-house solution I at least *can* fix the bugs.
>
> The feeling of powerlessness can be crushing when you depend on a
> third-party component that is broken with no fix in sight.
>
>
> Marko
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Sivan Greenberg
Co founder & CTO
Vitakka Consulting
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Marko Rauhamaa]

Radek Holý :

> 2016-05-17 9:50 GMT+02:00 Steven D'Aprano <
> steve+comp.lang.pyt...@pearwood.info>:
>
>> Overhead in the office today:
>>
>> "I don't have time to learn an existing library - much faster to make
>> my own mistakes!"
>
> *THUMBS UP* At least they are aware of that "own mistakes" part... Not
> like my employer...

Also:

   With a third party solution I don't need to fix the bugs.

   But with an in-house solution I at least *can* fix the bugs.

The feeling of powerlessness can be crushing when you depend on a
third-party component that is broken with no fix in sight.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quote of the day

[Radek Holý]

2016-05-17 9:50 GMT+02:00 Steven D'Aprano <
steve+comp.lang.pyt...@pearwood.info>:

> Overhead in the office today:
>
>
> "I don't have time to learn an existing library - much faster to make my
> own
> mistakes!"
>
>
>
> --
> Steve
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

*THUMBS UP* At least they are aware of that "own mistakes" part... Not like
my employer...
-- 
https://mail.python.org/mailman/listinfo/python-list

Quote of the day

[Steven D'Aprano]

Overhead in the office today:


"I don't have time to learn an existing library - much faster to make my own 
mistakes!"



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list