Re: [Tutor] Unit testing: Just the API or internal use only methods, too?
On 7/16/19 4:41 PM, boB Stepp wrote: > Peter Otten, while responding to one of my questions in the past, > mentioned something in passing that apparently has been mulling around > in the back of my head. I don't recall his exact words, but he > essentially said that I should be testing the public interface to my > classes, but not the methods only used internally by the class and not > meant to be publicly accessible. Is this generally how I should be > viewing testing? Would someone be willing to expand at some length on > this topic? Test everything (within reason). If you practice TDD, say, where your tests are the contract for how an interface shall behave, that makes a lot more sense for public APIs, which shall not change or be flagged immediately by your unit tests, than for internal functions which should be able to change within reason to suit implementation needs. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] raising exceptions in constructor code?
On 16Jul2019 23:49, Alan Gauld wrote: On 16/07/2019 22:56, Mats Wichmann wrote: thrown. This gets watered down to the mantra "Don't throw exceptions from within constructors." Does this carry over to Python? If you mean __init__, that's not a constructor, so you should set your mind at rest :) It's more properly an "initializer", the instance has already been constructed when it's called. FWIW The true constructor is __new__() and its quite rarely overridden by application programmers. But if you do, it's not common that you'd do anything that would merit an exception. __new__ pretty much just sets up the structure of the object ready for initialisation by __init__. Incidentally, this two stage construction/initialisation is also found in other OOP languages like Smalltalk and Objective C (and Lisp?). And to return to the OP's question: The __init__ method (and arguably __new__ if you touch it - very rare) is like other Python code: resource allocation should normally get unwound as objects become unreferenced. So raising an exception should be a pretty safe thing to do. That is a simplification. Of course if you implement an object with side effects _outside_ the Python object space (maybe it opened a scratch file to support something), it is your responsibility to ensure release in the object's __del__ method. But an object that just allocates a bunch of lists or dicts or the like? Python will clean that up for you. That said, I try to do cheap initialisation before exspensive initialisation. So allocating locks, opening files, starting worker threads: these come at the bottom of the __init__ method. Also, it is _ROUTINE_ to raise exceptions from __init__: like any other method we _expect_ you to raise ValueError if the initialiser parameters are insane (negatively sized arrays, etc etc). So in Python, raising exceptions in __init__ is normal: it shouldn't happen when you programme is running correctly of course, but it is the _preferred_ action when your initialiser cannot complete correctly. Consider: x = Foo() After this assignment we expect "x" to be a usable instance of Foo. We don't put special checks; what would such checks look like? (There are some answers for that, but they're all poor.) So raising an exception is what happens if __init__ fails. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] raising exceptions in constructor code?
On Tue, Jul 16, 2019 at 04:29:15PM -0500, James Hartley wrote: > I ask this having more C++ knowledge than sense. > > There is an adage in the halls of everything Stroustrup that one needs to > think about how resource allocation will be unwound if an exception is > thrown. This gets watered down to the mantra "Don't throw exceptions from > within constructors." Does this carry over to Python? I'm trying to > develop a Pythonistic mindset as opposed to carrying over old baggage... No, it is perfectly safe to raise exceptions from within the Python constructors, whether you are using __new__ (the true constructor) or __init__ (the initialiser). The only tricky part is if you allocate resources external to the object, like this: class Weird(object): openfiles = [] def __new__(cls, fname): f = open(fname) cls.openfiles.append(f) # New instance: instance = super().__new__(cls) if condition: raise ValueError return instance Even if the __new__ constructor fails, I've kept a reference to an open file in the class. (I could have used a global variable instead.) That would be bad. But notice I had to work hard to make this failure mode, and write the code in a weird way. The more natural way to write that^1 would be: class Natural(object): def __init__(self, fname): self.openfile = open(fname) if condition: raise ValueError Now if there is an exception, the garbage collector will collect the instance and close the open file as part of the collection process. That might not be immediately, for example Jython might not close the file until interpreter shutdown. But the earlier example will definitely leak an open file, regardless of which Python interpreter you use, while the second will only leak if the garbage collector fails to close open files. Here's a better example that doesn't depend on the quirks of the garbage collector: class Leaky(object): instances = [] def __init__(self): self.instance.append(self) if random.random() < 0.1: raise ValueError This will hold onto a reference to the instance even if the initialiser (constructor) fails. But you normally wouldn't do that. class NotLeaky(object): def __init__(self): if random.random() < 0.1: raise ValueError try: x = NotLeaky() except ValueError: pass Now either the call to NotLeaky succeeds, and x is bound to the instance, or it fails, and x is *not* bound to the instance. With no references to the newly-created instance, it will be garbage collected. ^1 Actually that's not too natural either. It is not usually a good idea to hold onto an open file when you aren't actively using it, as the number of open files is severely constrained on most systems. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing: Just the API or internal use only methods, too?
On 16/07/2019 23:41, boB Stepp wrote: > essentially said that I should be testing the public interface to my > classes, but not the methods only used internally by the class and not > meant to be publicly accessible. I suspect he meant that you should publish the tests for the API but not necessarily for the internal/private methods. You should definitely test all code you write, but how formally you test the private stuff is up to you. But publishing the public API tests allows clients to run them too. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] raising exceptions in constructor code?
On 16/07/2019 22:56, Mats Wichmann wrote: >> thrown. This gets watered down to the mantra "Don't throw exceptions from >> within constructors." Does this carry over to Python? > > If you mean __init__, that's not a constructor, so you should set your > mind at rest :) It's more properly an "initializer", the instance has > already been constructed when it's called. FWIW The true constructor is __new__() and its quite rarely overridden by application programmers. But if you do, it's not common that you'd do anything that would merit an exception. __new__ pretty much just sets up the structure of the object ready for initialisation by __init__. Incidentally, this two stage construction/initialisation is also found in other OOP languages like Smalltalk and Objective C (and Lisp?). -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Unit testing: Just the API or internal use only methods, too?
Peter Otten, while responding to one of my questions in the past, mentioned something in passing that apparently has been mulling around in the back of my head. I don't recall his exact words, but he essentially said that I should be testing the public interface to my classes, but not the methods only used internally by the class and not meant to be publicly accessible. Is this generally how I should be viewing testing? Would someone be willing to expand at some length on this topic? TIA! -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] How would I replace the data of a Pandas Series with the values of a dictionary?
Hi all,
I have a problem trying to match items in a dict and pandas series in
Python.
I have a dict ( called city_dict )of cities and city_id's ; for each city (
which is a key in the dict ), a unique city_id is a value in that dict.
So for example, city_dict = { New York : 1001, LA : 1002, Chicago : 1003 }.
New York is a key, 1001 is a value.
Now I have a panda Series called dfCities. In this series is a bunch of
cities, including the cities in city_dict.
My goal is to replace the cities in dfCities with the city_id's in a brand
new csv file. So if dfCities has New York in it, I want to replace it with
it's value in the dictionary, so 1001.
Approaches I've tried - checking to see if the keys match the cities in
dfCities in a 'if in' statement ( such as "if city_dict.keys() in
dfSeries"), and then doing a straight replace ( can't do that since series
are ambiguous in truth values). Therefore I tried using .any() for Pandas
series (since .all() would strictly want all values in dfCities to match,
and all values don't match )
Afterwards, tried to directly match the series with keys and the clarified
truth value series, but dict_keys are unhashable, so I had to convert the
keys to str and see if I could compare strings ( with a stringified
dfCities )
Then I realized that even if I can get a if statement to start checking
(if dfCities.str.contains(keyss).any(): ) (keyss being the stringified
version of the keys for city_dict ), I don't know how to build a approach
to cross check the values of city_dict with the cities in dfCities ( I have
a vague notion that I should check if the keys of city_dict match with
dfCities, and then replace the cities in dfCities with the values of
city_dict in a new csv file output. However, I don't know how to replace
data in a Series with vaues of a dict ).
So I would like to ask the community what approach I can take to build to
that piece of the puzzle. I feel I have most of the solution, but I'm
missing something.
Thanks for reading and I appreciate the help.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] raising exceptions in constructor code?
On 7/16/19 3:29 PM, James Hartley wrote: > I ask this having more C++ knowledge than sense. > > There is an adage in the halls of everything Stroustrup that one needs to > think about how resource allocation will be unwound if an exception is > thrown. This gets watered down to the mantra "Don't throw exceptions from > within constructors." Does this carry over to Python? I'm trying to > develop a Pythonistic mindset as opposed to carrying over old baggage... If you mean __init__, that's not a constructor, so you should set your mind at rest :) It's more properly an "initializer", the instance has already been constructed when it's called. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] raising exceptions in constructor code?
I ask this having more C++ knowledge than sense. There is an adage in the halls of everything Stroustrup that one needs to think about how resource allocation will be unwound if an exception is thrown. This gets watered down to the mantra "Don't throw exceptions from within constructors." Does this carry over to Python? I'm trying to develop a Pythonistic mindset as opposed to carrying over old baggage... Thanks! Jim ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading .csv data vs. reading an array
Chip Wachob wrote: > I tried it anyhow, with this being an example of my source data: > > "Record Length",202,"Points",-0.005640001706,1.6363 > "Sample Interval",5e-09,s,-0.005639996706,1.65291 > "Trigger Point",1128000,"Samples",-0.005639991706,1.65291 > "Trigger Time",0.341197,s,-0.005639986706,1.60309 > ,,,-0.005639981706,1.60309 > "Horizontal Offset",-0.00564,s,-0.005639976706,1.6363 > ,,,-0.005639971706,1.65291 > ,,,-0.005639966706,1.65291 > ,,,-0.005639961706,1.6363 > . > . > . > > Note that I want the items in the third and fourth column of the csv file > for my time and voltage. > > When I tried to use the unpack, they all came over as strings. I can't > seem to convert them selectively.. Try wrapping the reader like this: $ cat custom_reader.py import csv import io data = """\ "Record Length",202,"Points",-0.005640001706,1.6363 "Sample Interval",5e-09,s,-0.005639996706,1.65291 "Trigger Point",1128000,"Samples",-0.005639991706,1.65291 "Trigger Time",0.341197,s,-0.005639986706,1.60309 ,,,-0.005639981706,1.60309 "Horizontal Offset",-0.00564,s,-0.005639976706,1.6363 ,,,-0.005639971706,1.65291 ,,,-0.005639966706,1.65291 ,,,-0.005639961706,1.6363 """ def maybe_float(s): try: return float(s) except ValueError: return s def myreader(*args, **kw): reader = csv.reader(*args, **kw) for row in reader: yield [maybe_float(field) for field in row] for row in myreader(io.StringIO(data)): print(row) $ python3 custom_reader.py ['Record Length', 202.0, 'Points', -0.005640001706, 1.6363] ['Sample Interval', 5e-09, 's', -0.005639996706, 1.65291] ['Trigger Point', 1128000.0, 'Samples', -0.005639991706, 1.65291] ['Trigger Time', 0.341197, 's', -0.005639986706, 1.60309] ['', '', '', -0.005639981706, 1.60309] ['Horizontal Offset', -0.00564, 's', -0.005639976706, 1.6363] ['', '', '', -0.005639971706, 1.65291] ['', '', '', -0.005639966706, 1.65291] ['', '', '', -0.005639961706, 1.6363] If you find that performance suffers more than you are willing to accept here's an alternative implementation of maybe_float() that may be faster for some inputs: def maybe_float(s): if s and s[:1] in " 0123456789+-": try: return float(s) except ValueError: return s return s ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Object references in Python
On 7/16/19 2:33 PM, Steven D'Aprano wrote:
> x = Parrot()
>
> Now x is a reference to a Parrot instance. y remains a reference to the
> list.
>
> x.colour is a reference to the string "blue" (by default).
>
> x.speak is a reference to the "speak" method of Parrot objects.
>
>
>
> Does this help?
>
>
>
Let's add one more little cute one for good measure:
>>> def foo():
... print("This function does Foo")
...
>>> foo()
This function does Foo
>>> # we created a function object, and foo is a reference to it
...
>>> x = foo
>>> # x should be a reference to the same object
...
>>> x()
This function does Foo
>>> x is foo
True
>>> def foo():
... print("This function no longer Foos")
...
>>> # we created a new function object, and foo is now a reference to it
...
>>> foo()
This function no longer Foos
>>> x()
This function does Foo
>>> # x is still a reference to the original function object
...
>>> x is foo
False
>>>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Object references in Python
On Tue, Jul 16, 2019 at 12:08:10PM +, AHIA Samuel wrote:
> Please what are references in Python development?
x = 19
y = x
The name "x" is a reference to the int 19; the name y is a reference to
the same int.
x = "Hello world"
Now the name "x" is a reference to the string "Hello world". y remains a
reference to the int 19.
x = None
Now the name "x" is a reference to the None object.
x = [100, 200, 300]
Now the name "x" is a reference to a list with three items:
- The first item of x, x[0], is a reference to the int 100.
- The second item of x, x[1], is a reference to the int 200.
- The third item of x, x[2], is a reference to the int 300.
y = x
Now y is no longer a reference to the int 19 as before, but is a
reference to the same list that x is a reference to. There are now two
references to the list object: x and y.
(If you are a C programmer, you can think of x and y both being pointers
to the same list. This is not completely correct, but it isn't far
wrong.)
Since x and y are references to the same list, we can equally say:
- The first item of y, y[0], is a reference to the int 100.
- The second item of y, y[1], is a reference to the int 200.
- The third item of y, y[2], is a reference to the int 300.
x.append(None)
Now the name x is still a reference to the same list as before, except
that we have added a new item to the end of the list:
- The fourth item of x, x[3], is a reference to the None object.
- The fourth item of y, y[3], is a reference to the None object.
Since both x and y are references to the same list, any change to the x
list is a change to the y list (since they are the same).
class Parrot:
def __init__(self, colour="blue"):
self.colour = colour
def speak(self):
print("Polly wants a cracker!")
Now the name "Parrot" is a reference to the class "Parrot".
x = Parrot()
Now x is a reference to a Parrot instance. y remains a reference to the
list.
x.colour is a reference to the string "blue" (by default).
x.speak is a reference to the "speak" method of Parrot objects.
Does this help?
--
Steven
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Impersonation
On 7/15/19 9:36 PM, Jim wrote: Mats, Hopefully you recognize my email address as someone you have given advice concerning Python. Over the last month or so I have received at least 3 emails supposedly coming from you that I am sure you did not send. The from line is: Mats Wichmann The body is: On Monday, July 15, 2019 10:36 AM, Mats Wichmann wrote: Hope you are well. Just wanted to share something with you http://www.bt6q.lnhaxf.info/ I just wanted you to know that it seems someone is trying to impersonate you. Regards, Jim My apologies. I intended that this message go only to Mats. Regards, Jim ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading .csv data vs. reading an array
> On Jul 16, 2019, at 04:31, Alan Gauld via Tutor wrote: > > On 16/07/2019 09:59, Oscar Benjamin wrote: > >>> All true, but sed - once you get used to it! - is easier IMHO >>> and usually faster than Python - it's written in C... >> >> I always think I'll like sed but whenever I actually try to use it I >> just can't get the syntax together. > > > Id been using awk and sed for 15 years before I discovered > Python (and perl) so they are kind of second nature. > > If you can't settle with sed try awk, it's much simpler to > use and almost as powerful but not as fast. I think awk is > one of the most neglected of the *nix tools now that > scripting languages like perl/python/ruby exist. But for > line by line file processing it is superb. The first O’Reilly book I ever purchased was “Sed & Awk”, and has been one of the most invaluable over the last 20 years. While they are not the simplest tools to master, they are worth the effort; especially when you are trying to do inline “one-liners” to massage data or process large files. That doesn’t mean it’s a requirement to know them, but it does get easier with practice. That said, if making a little python one-off filter to do what you need is faster (to write) and works (well enough), it comes down to what your time is worth. — David Rock da...@graniteweb.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Object references in Python
Please what are references in Python development? ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading .csv data vs. reading an array
On 16/07/2019 09:59, Oscar Benjamin wrote: >> All true, but sed - once you get used to it! - is easier IMHO >> and usually faster than Python - it's written in C... > > I always think I'll like sed but whenever I actually try to use it I > just can't get the syntax together. Id been using awk and sed for 15 years before I discovered Python (and perl) so they are kind of second nature. If you can't settle with sed try awk, it's much simpler to use and almost as powerful but not as fast. I think awk is one of the most neglected of the *nix tools now that scripting languages like perl/python/ruby exist. But for line by line file processing it is superb. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading .csv data vs. reading an array
On Tue, 16 Jul 2019 at 01:44, Alan Gauld via Tutor wrote:
>
> On 15/07/2019 21:28, Mats Wichmann wrote:
>
> > course Python can do that too, by working line-at-a-time, explicitly by
> > calling readlines() or implicitly by looping over the file handle. The
> > latter looks something like this;
> >
> > with open("/path/to/datafile", "r") as f:
> > for line in f:
> > if REDFLAGTEXT in line: # skip these
> > continue
> > do-something-with line
>
> All true, but sed - once you get used to it! - is easier IMHO
> and usually faster than Python - it's written in C...
I always think I'll like sed but whenever I actually try to use it I
just can't get the syntax together. I do use vim and can do
find/replace there. It seems like every different utility grep, egrep,
sed, vim etc has subtly different escaping rules or maybe I just
haven't got my head around it.
When writing this pull request:
https://github.com/sympy/sympy_doc/pull/32
I spent something like 15 minutes trying to get sed to work before
giving up. It took me 2 minutes to write and run the Python script
that I ended up using.
Oscar
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
