Re: [Tutor] try and file existence
In a message of Sat, 15 Aug 2015 15:20:19 -0700, Clayton Kirkwood writes: If you want to locate dangling symlinks, os.path.exists will return False, so the symlink is there, but the file it pointed to is long gone. Can't you do that with os.path.open() and get a value in os.path.status? (I think that is the thing to call) crk Laura There is no os.path.open I assume you are thinking of os.open? or open the builtin? or maybe os.stat? If what you really want to do is open the file, if it exists, then trying to open it and then if that fails handle whatever problem you get is most often the way to go. But often you never wanted to open it in the first place. Maybe it is a directory. Maybe it is a lockfile --- oops, I will come back later. There are a lot of times when the sense you really want is not 'check if this valued file exists' but rather 'check for cruft you don't want to find'. Open is also slow, which again isn't a problem if you need to open the file anyway, but will matter if all you want to do is check your entire large filesystem for files named 'core'. Laura ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On 16/08/15 07:18, boB Stepp wrote: tonight, it appears that I need to create a test SQLite db. I don't see any other way that I can test such code without having a db to test against. Correct. And it is a non trivial task but immensely useful since you can populate it with sample data representing every possible tricky scenario - multiple parents, no parents, duplicate students, etc. etc. You can then test the code that handles those scenarios easily by referencing the appropriate test data.. 1) It would seem that I need to install a stand-alone version of SQLite, so that I can create this test db. You should do that anyway. You will inevitably want to run SQL queries directly. In fact you will probably have batch reports to create that are much easier done using raw sql in a file rather than going through Python. Even inserting your data will quite likely be easier done with raw SQL. separate Python program whose sole purpose would be to create this test db. But if I go with installing a stand-alone SQLite, will I run into version conflicts with whatever version of SQLite is bundled in the standard library of Python 3.4.3? Theoretically yes. But in practice I've never had an issue. The SQL doesn't change much. But if you are really, really worried there is a trick you can pull with the Python interpreter. The cursor object has an executescript() method that takes a SQL file as an argument. 2) If I install the test db I can conceptually see that I should be able to test all of the Python code that interacts with it. However, do I need to figure out some way to test what will eventually become the *real* db that the program will generate and use? How will I know if my test db structure and the resulting actual db structure that the ultimate user(s) will populate are in agreement? Create the structure with one SQL script. Insert the data with another one. Use the same creation script for test and production databases. That ensures both have the same structure but with different content. Or am I over-analyzing here? No, they are valid concerns. Fortunately, more of an issue in theory than in practice. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On Sun, Aug 16, 2015 at 01:18:06AM -0500, boB Stepp wrote: Being committed to TDD for this project, I am not yet ready to write code till I figure out how to write the tests. It occurs to me that writing tests for code that interacts with the SQLite db may be non-trivial (At least for me!). After doing some online research tonight, it appears that I need to create a test SQLite db. I don't see any other way that I can test such code without having a db to test against. So this leads to several questions: 1) It would seem that I need to install a stand-alone version of SQLite, so that I can create this test db. Either that or write a separate Python program whose sole purpose would be to create this test db. But if I go with installing a stand-alone SQLite, will I run into version conflicts with whatever version of SQLite is bundled in the standard library of Python 3.4.3? *scratches head* I'm not sure what you mean by this. You create a sqlite database like this: import sqlite3 conn = sqlite3.connect(spam.db) To create a second database, you do this: another = sqlite3.connect(eggs.db) So I expect that you could do something like this: - In your test directory, write a script which creates, and populates, a small database; you only need to run this script once, although if it is small enough, there's no harm to running it each time the test suite runs. It could even be part of the unit test setup code. - Since you create the database, you know what the content will be. - Write your application so that the database location is configurable when the application starts, not hard-coded. The easiest way to do this, although not the best way, is to make the database connection a global variable, then monkey-patch it from your test suite: import myapplication import sqlite3 db = sqlite3.connect(eggs.db) myapplication.DB.close() myapplication.DB = db # run tests - Your unit tests can set the database to the test DB and you can now check that functions return the results you expect. This, by the way, is a good idea even if you aren't testing the DB layer. You don't want a bug or badly-thought out test in your test suite to mess up the actual database used for live data. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] How to test my code's interactions with SQLite db?
Being committed to TDD for this project, I am not yet ready to write code till I figure out how to write the tests. It occurs to me that writing tests for code that interacts with the SQLite db may be non-trivial (At least for me!). After doing some online research tonight, it appears that I need to create a test SQLite db. I don't see any other way that I can test such code without having a db to test against. So this leads to several questions: 1) It would seem that I need to install a stand-alone version of SQLite, so that I can create this test db. Either that or write a separate Python program whose sole purpose would be to create this test db. But if I go with installing a stand-alone SQLite, will I run into version conflicts with whatever version of SQLite is bundled in the standard library of Python 3.4.3? 2) If I install the test db I can conceptually see that I should be able to test all of the Python code that interacts with it. However, do I need to figure out some way to test what will eventually become the *real* db that the program will generate and use? How will I know if my test db structure and the resulting actual db structure that the ultimate user(s) will populate are in agreement? Or am I over-analyzing here? TIA! -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] variable naming conventions
In a message of Sat, 15 Aug 2015 18:24:53 -0700, D Wyatt writes: It seems every book I read these days uses camel case for variable names in Python. I was once told that using underscores is preferred. Is there a preference in the Python community or does it really matter? I'd like to instill good habits while I'm learning. Thanks in advance, -- Deb Wyatt in WA The Python Community is too large now to have only one preference. People's preferences for CamelCase or this_uses_underscores is usually language independent -- if you prefer it when you are using language X, you probably prefer it when you are using langauge Y, too. The Python Standard Library, for the most part, uses underscores for variable names and CamelCase for class names. See PEP 008 for 'preferred Python Style' -- such as it is, but be aware that if you join a project it is more important to follow the conventions used there than to hop up and down saying 'This is not PEP 008 !' Laura ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] variable naming conventions
Laura Creighton l...@openend.se writes: The Python Standard Library, for the most part, uses underscores for variable names and CamelCase for class names. Note a sharp distinction between camelCaseNames, which the Python community eschews, versus TitleCaseNames, which are embraced for names of classes. See PEP 008 for 'preferred Python Style' -- such as it is, but be aware that if you join a project it is more important to follow the conventions used there than to hop up and down saying 'This is not PEP 008 !' Yes. If you're looking for a style guide, choose PEP 8, because your style will then be maximally familiar to others when you inevitably need to collaborate with some other people. If you're joining a community that doesn't have consistently-enforced, coherent conventinos, make a strong push for PEP 8 for the same reason. But if you're joining an existing community that has chosen a coherent convention which it enforces consistently, go with that. -- \ “We demand rigidly defined areas of doubt and uncertainty!” | `\—Vroomfondel, _The Hitch-Hiker's Guide To The Galaxy_, Douglas | _o__)Adams | Ben Finney ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
On 16/08/15 03:10, boB Stepp wrote: time. I have not ever thought about formal project directory structures before, so I may need a bit of guidance here. As you've discovered this varies wildly between projects. Here is my default structure, but bear in mind most of my projects have been big multi-team, multi-language type structures so you probably don't need all of this. project - doc project documents: contracts reqs, designs, test specs etc - man(*) user docs - bin(*) the master exe or main.py type files - lib(*) the shipping libraries - src the code -- lang folder per language used - sql, python, C, bash, etc --- lib modules/packages - subfolder per package --- test test code - sub-tree under this, depends on test tools. --- toolstools used but not shipped - db load/reset etc --- main folder in some languages, a file in others Of course there may be others needed under those - such as data, config, etc. And you need some kind of script to copy the shipping files into their final resting places(stored under tools of course). The shipping folders are those marked with (*) (although with open source you may well ship all/some of the src tree too). And on a big project their may well be a separate production environment created for each live release that enables testing of fault reports from users etc. But that's almost certainly overkill for your needs. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
boB Stepp robertvst...@gmail.com writes: Being committed to TDD for this project, I am not yet ready to write code till I figure out how to write the tests. It occurs to me that writing tests for code that interacts with the SQLite db may be non-trivial (At least for me!). That's correct. One of the primary benefits of Test-Driven Development is to enforce good design on your code: you'll need to design your system so that it's testable with clear, narrowly-defined interfaces. SQL queries are *not* a clear, narrowly-defined interface. So, as you're discovering, it is very difficult to write unit tests for code that could execute some arbitrary query. So that points to a need for better design: Don't directly issue arbitrary SQL queries in code which implements higher-level features. Instead, define a much more specific interface between your feature code and the lower-level code that interacts with the database. Put SQL queries only in very narrowly-defined database interaction functions, where the input and output can be tested easily with unit tests. The unit tests present mock database API functions, that implement only enough to satisfy the narrow actions each low-level function will perform. If you can't set up a trivially-simple fixture to pretend to be the database API, the function is doing too much. Break it down further, possibly along the way finding duplication and refactoring those into levels of abstraction. Put feature code only in higher-level code, where the input and output doesn't have anything to do with the details of SQL and can therefore be tested easily with unit tests. The unit tests make trivially-simple collections — basic types like mappings or sequences — to fake the results from the lower-level code. If you can't make a trivially-simple sequence or mapping to pretend to be the result from the lower-level data code, the function is doing too much. Break it down further, find duplications, refactor them to abstration layers. A database is a very heavy external dependency. You should not be attempting to “mock the world” in order to make your tests run. Instead, you should be making your code well designed: small and simple and narrowly-defined functions, each one of which is easy to test with simple fixtures. -- \ “I call him Governor Bush because that's the only political | `\ office he's ever held legally.” —George Carlin, 2008 | _o__) | Ben Finney ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] variable existence q
Clayton Kirkwood wrote: Above is the actual code. Clayton, try to understand how scoping works in Python before you go back to your actual code. Can you predict what the following snippet will produce? x = first global def f(): return x def g(): return x x = local x = second global print(f()) print(g()) What will the first print() produce? 'first global', i. e. the value the name x is bound to when f is created, or 'second global', the value the name x is bound to when f() is invoked? What will g() try to return? The local variable x or the global variable x? Does it succeed? If not, why? Once you are clear about both problems fixing your actual code should be easy. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] variable existence q
On Sat, Aug 15, 2015 at 03:38:31PM -0700, Clayton Kirkwood wrote:
top_directory = /users/Clayton/Pictures
target_directory = top_directory #directory we are checking
filetypes = ('jpg', 'png', 'avi', 'mp4', 'mov', 'bmp')
imports...
def override_defaults():
with open( user_preferences ) as f:
for line in f.readline():
llist = line.split()
if llist[0] == '#': #comment line to ignore
continue
elif llist[0] == 'top_directory':
if len(llist) == 1:
pass
else:
top_directory = llist[1]
This line tells the compiler that top_directory must be a local
variable, since you assign to it within a function. As a local variable,
it only gets set on *some* paths through the function, so you get an
error, same as this:
def example(n):
if n % 2 == 0:
td = test
else:
pass
return td # fails half the time
Python uses a simple rule to decide whether or not a variable is a local
or not. If the variable is assigned to *anywhere* in the function, it is
treated as local. Even if the line is never actually executed! (For this
purpose, del is treated as a de facto assignment too.) So you can even
do this:
x = 23
def test():
return x
# code below here is never executed
if False:
# and even if it were, code inside this block is never executed
x = 42 # makes x a local variable
and calling test() will now give an UnboundLocalError.
To tell Python not to treat it as a local, you need to declare it
global. Same with target_directory. So one solution is to put this as
the first line of your function:
global top_directory, target_directory
(Technically, you can put it anywhere inside the function, yes, even
after the return statement, and it will have the same effect. But don't
do that. Always put it at the start.)
Another solution is to write the function like this:
def override_defaults():
top = top_directory
target = target_directory
with open( user_preferences ) as f:
for line in f: # no need for f.readlines
line = line.strip() # ignore leading and trailing whitespace
words = line.split()
if words[0].startswith('#'): #comment line to ignore
continue
elif words[0] == 'top_directory':
top = words[1]
[ ... ]
return (top, file_types, target)
--
Steve
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How best to determine if a db exists before trying to open it? [Was: try and file existence]
On Sat, Aug 15, 2015 at 06:24:12PM -0500, boB Stepp wrote: db = sqlite3.connect(my_db.db) 1) This will open the db if it exists already, which is normally what I will want. But... 2) My understanding is that if for whatever reason the db file is not found, then the connect statement will create a new instance of the db, which is what I normally would not want (Except at the time of initial creation). [...] If I am understanding everything so far, I think that my situation would be appropriate for using os.path.exists(). Is this correct? Sure, why not? If might not be an utterly bullet-proof solution, but it will do the job for now and you can always revisit it later if needed. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On 16/08/15 09:10, Ben Finney wrote: So that points to a need for better design: Don't directly issue arbitrary SQL queries in code which implements higher-level features. Instead, define a much more specific interface between your feature code and the lower-level code that interacts with the database. This is a good point. You may recall my earlier messages talking about MVC and the fact that only the Model should interact with the database via SQL? So the model interface is effectively your app's API into the database. And when unit testing the Model you will probably use mocks for the SQL calls in most cases. So only a very small number of tests need a real test database. Where you do need a test database is in testing the database itself (data structure, constraints, relationships etc). Much of that you can(and should) do from raw SQL, but at some point (integration testing) you need to run the actual SQL in your Model... Another point worth mentioning that does NOT apply with Sqlite is that most server DBs provide a Stored Procedure concept and many database designers prefer to make all updates to data via stored procedures and expose all read access via read-only views. This is especially so when exposing the API over a network (eg as web services). SQlite does not offer stored procedures (but does offer views) and expects to work on a local machine rather than over a network so it doesn't apply there. But if you ever move to Firebird, Postgres or MySQL it would be a thing to consider. tests. The unit tests present mock database API functions, that implement only enough to satisfy the narrow actions each low-level function will perform. This is also good advise but remember that unit tests only form a small part of the overall testing (although they are the bit normally used for TDD) and for system tests you will need a test database. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] try and file existence
On Sat, Aug 15, 2015 at 07:04:47PM -0500, boB Stepp wrote: On Sat, Aug 15, 2015 at 6:41 PM, Steven D'Aprano st...@pearwood.info wrote: On Sat, Aug 15, 2015 at 02:24:21PM -0500, boB Stepp wrote: I understand your points, but wonder then what is the intended use for os.path.exists()? That is, in what types of circumstances would it be both appropriate and safe to use? def print_file_names(possible_names): print(List of file names checked) print(-- for name in possible_names: if os.path.exists(name): print(name) else: print(missing:, name) Chuckle! Your example, giving about the most benign possible uses, is for emphasis? Well, not really. I was trying to think of a case where you want to check whether a file exists but not actually open the file (or at least, not open the file *yet*). -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
On 2015-08-16 01:28, Alan Gauld wrote: Here is my default structure project - doc project documents: contracts reqs, designs, test specs etc - man(*) user docs - bin(*) the master exe or main.py type files - lib(*) the shipping libraries - src the code -- lang folder per language used - sql, python, C, bash, etc --- lib modules/packages - subfolder per package --- test test code - sub-tree under this, depends on test tools. --- toolstools used but not shipped - db load/reset etc --- main folder in some languages, a file in others Alan, Assuming the above structure and further assuming that your python test suite is under test, how do you arrange to import code that is under main? I have only been able to figure out how to import code that is at the same or lower level in the file structure. If one's test code was in --- test/test.py, an import of --- main or main/mycode.py would have to 'find' the latter by traveling up the tree to test and then over to main, or over to main and then down to mycode.py. I've not been able to do that. Is there a way? Alex ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
On 16/08/15 16:41, Alex Kleider wrote: - src the code -- lang folder per language used - sql, python, C, bash, etc --- lib modules/packages - subfolder per package --- test test code - sub-tree under this, depends on test tools. --- toolstools used but not shipped - db load/reset etc --- main folder in some languages, a file in others Alan, Assuming the above structure and further assuming that your python test suite is under test, how do you arrange to import code that is under main? Thee are several options. 1) create links from, main to the test files needed 2) alter sys.path so imports can see the test folder 3) alter the PYTHONPATH environment var But in most of my Python projects main is a file rather than a folder so test is under main anyway. This above is a generic structure because most of my projects use 4 or 5 languages at least. Some IDEs require different structures too so the separate lang sub-structures may vary to suit the toolset usee. I've not been able to do that. Is there a way? I suspect in this case the easiest solution is a link (aka shortcut in windoze) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
Hi Bob, By the way, when you're unit testing with Sqlite, you might find it convenient to use the :memory: option, which keeps the database in RAM rather than on disk. That should make the setup and tear-down of the testing environment easier to maintain. The principle is similar to that of when we're unit testing functions whose inputs or outputs are file-like objects; although we can use real files, we can find in-memory structures like io.StringIO useful, since they leave no residue once the tests are completed. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On Sun, Aug 16, 2015 at 4:03 AM, Steven D'Aprano st...@pearwood.info wrote: On Sun, Aug 16, 2015 at 01:18:06AM -0500, boB Stepp wrote: 1) It would seem that I need to install a stand-alone version of SQLite, so that I can create this test db. Either that or write a separate Python program whose sole purpose would be to create this test db. But if I go with installing a stand-alone SQLite, will I run into version conflicts with whatever version of SQLite is bundled in the standard library of Python 3.4.3? *scratches head* I'm not sure what you mean by this... The thought is to download and install the SQLite command-line shell program available at http://www.sqlite.org/download.html which is currently at version 3.8.11.1. The SQLite in my standard library for Python 3.4.3 is version 3.8.3.1. (Until just now checking what the actual versions are, I did not realize how close they are.) Using this command-line version (which I was calling stand alone) I would independently of my program create a test db for use in my program's tests. So I expect that you could do something like this: - In your test directory, write a script which creates, and populates, a small database; you only need to run this script once, although if it is small enough, there's no harm to running it each time the test suite runs. It could even be part of the unit test setup code. And this was my second thought for creating a test db for use in my unit tests. However, I was thinking that the first approach would be better as then the test db itself would be totally independent of whatever code I write in the actual program. It would also work as a design tool as the command-line SQLite would allow me to easily get a visual representation of the tables, etc., which I think would be helpful as I start to code. Also, per Alan's suggestions it would be more easy to add the tricky cases that I would want to be certain my tests handle. This, by the way, is a good idea even if you aren't testing the DB layer. You don't want a bug or badly-thought out test in your test suite to mess up the actual database used for live data. Yeah, I am really concerned about this possibility. My other concern is that I would unknowingly structure my test db differently than what my program would generate. boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Writing back to same CSV in the next column
Hello,
Thank you for your guidance. Using your pseudocode I have put together the
following:
import socket
import csv
in_file = open('top500ips.csv', 'r')
out_file = open('top500ips_out.csv', 'w')
for line in in_file:
try:
name = socket.gethostbyaddr(line.strip())
out_file.write(line + '\t' + (str(name))
except socket.herror:
out_file.write(line + '\t' + errrMsg)
in_file.close()
out_file.close()
-
I am getting few errors. Such as, I had to add 'str in front of (name) to
address TypeError: Can't convert 'int' object to str implicitly.
Also, not sure why I keep getting SyntaxError: invalid syntax pointing to the
except line.
Please advise.
Thank you.
Thank you.
On Wednesday, August 12, 2015 7:07 AM, Nym City via Tutor
tutor@python.org wrote:
Hello,
Please find the two requested files attached. The 'before' file is what I am
reading into my program. The 'after' file is what I would like to have my
output to look like. Ideally, I want it to be the same file but if its easier
to create a new file for the output - that is ok too.
I do not have the understanding of binary vs text modes. But I have found this
online, can't say I understand it though:
http://fileinfo.com/help/binary_vs_text_files Thank you.
On Tuesday, August 11, 2015 4:10 AM, Alan Gauld alan.ga...@btinternet.com
wrote:
On 11/08/15 01:23, Nym City via Tutor wrote:
import socket
import csv
ListOfIPAddresses = []
with open('top500ips.csv', 'rb') as f:
for line in f:
line = line.strip()
ListOfIPAddresses.append(line)
f.close()
You don;t need the f.close(). The 'with' structiure
does that automatically.
# print(ListOfIPAddresses)
newFile = open('top500ips.csv', 'w')
The original file was opened in binary mode, you
are opening it here in text mode. Are you sure
that's correct? Do you undertand the significance
of binary v text modes?
Also 'w' mode effectively creates a new empty file
so you will need to recreate every line that was
in the input file. Its usually better to rename
the original file to something like top500ips.bak
and then create a new file with the original name.
If all goes well you can delete the .bak version,
if something goes wrong you can rename it back
to the original.
for address in ListOfIPAddresses:
try:
ResolvedAddresses = socket.gethostbyaddr(address)[0]
You save the result into the variable but do nothing with it.
The next time round the loop the result will be overwritten and the
previous one lost. You are not writing anything to the file.
except socket.herror as e:
print(No resolution available for %s: %s % (address, e))
newFile.write.(ResolvedAddresses + \n)Thank you.
You are only writing to the file when you get the error.
But at that point ResolvedAddresses will contain the result from
the previous iteration of the loop so it may well be misleading.
You in effect only write the host to file for the entry
before lines that cause errors. I'm pretty sure thats not what
you want.
The other thing is that you are only writing the name. So your
file will only contain a short column of names. Again I don't
think that's what you wanted.
Can you send us an example of before and after?
ie about 5 lines of content from the file before you start
and what it should look like after you finish?
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On 16/08/15 23:29, boB Stepp wrote: http://www.sqlite.org/download.html You definitely want this. You treat it like the prompt in Pyython. A place to try out SQL queries before you put them into your Python code. Also you can write long sql code in a .sql filer and read them into the interpreter using the .read command sqlite3 .read createdb.sql sqlite3 .read populate_tests.sql sqlite3 .read clear_tables.sql sqlite3 .read populate_base_data.sql etc etc. Also you can ask the interpreter to describe table structures and constraints etc. Think TDD here... Remember to use semi-colons to terminate statements! design tool as the command-line SQLite would allow me to easily get a visual representation of the tables, etc., Depends what you mean by visual! But there are several GUI tools available that will provide an admin GUI for your DB, this is useful for visual representation of tables(like a spreadsheet) and for ad-hoc updates to fields. In practice I only use these on my smart phone (I store the database on Dropbox) but then I'm fairly fluent in SQL at the sqlite3 prompt! is that I would unknowingly structure my test db differently than what my program would generate. As I said, create separate .sql files to create an empty database and to populate the structure with initial data. That way you use the exact same structure file in your code as in your tests (remember the cursor.executescript() method I mentioned earlier! - that works from inside your code as well as the prompt!) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On 17/08/15 00:52, boB Stepp wrote: sqlite3 .read populate_base_data.sql I am assuming that the .read command would be replaced inside the program by the cursor.executescript() method you mentioned? This will be quite handy, I think. No. The executescript() method *replaces* .read .read is only used within the interpreter. There are a whole bunch of these pseudo-commands, including .quit to exit. In executescript() you just pass in the name of the file you want executed. Very easy. But like os.system() not so easy to check it all worked OK... an exercise for the reader, as they say... Also you can ask the interpreter to describe table structures and constraints etc. Think TDD here... Again the . commands. Try sqlite3 .help You'll like it :-) In particular try the .tables, .trace and .schema commands. aid testing? Being an insane accumulator of books, I have acquired, The Definitive Guide to SQLite, 2nd ed., by Grand Allen and Mike Owens, c. 2010. I don;t know it - I own Using SQLite from O'Reilly. But the web docs are good too. and how much flexibility I should allow the user to have to extend the db structure as shipped. For instance I can easily see the user wanting to add new types of information she wants to track for her student. I'd allow a small amount of leeway here. The usual approach is to use string fields for the data and a meta table with field, type, name information. So if you have a student table with three user defined fields UDF1, UDF2,UDF3. You define a meta table called meta_student that has field, name, type columns containing things like: UDF1, BOOL, hasTattoos UDF2, FILENAME, Mugshot UDF3, STRING, Birthmark (You could have a single meta_data table with table as first column but that just makes your SQL even more complex IMHO! Small tables are OK. Also some implementations allow user specific definitions but that breaks badly if records are shared if, for example, user1.UDF1 is hasTattoos and user2.UDF1 is isPregnant) BUT! You need a lot of infra-structure around this since these fields will be global to all users. And you need to manually do any data validation of input to ensure that only valid data gets stored. You will need a bank of supported types/validator functions. And your UI needs to ensure users can only create appropriate options (think drop down lists and dictionaries). Personally I'd only add this flexibility if you really need it. Its a major code overhead and a big performance hit too. allowed such functionality. But in this example, how many new columns of table data do I cut off the tests at, for instance? I've seen commercial systems that allow up to 100 UDFs. In practice I'd go for 3-10. if they need more than 10 then your requirements capture and user testing was very poor! Eventually RAM or some kind of overflow condition will transpire, With a database that tends to be disk space so think Terabytes. Its not normally an issue these days! want to test that a *lot* of table columns can be added without breaking anything will require me to set some reasonable upper limit for what constitutes a *lot*. The biggest issue, as with user defined attributes in objects is that your static code doesn't know what these new fields are. Imagine how they play with a search function? Building a SQL query against a boolean value in the UDF1 column means translating the boolean result into its string equivalent and doing a fill scan text search. Slow You really want database tables to be defined as fully as possible as soon as possible. UDFs are a (sometimes necessary) cost of failure! -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On Sun, Aug 16, 2015 at 7:36 PM, Danny Yoo d...@hashcollision.org wrote: By the way, when you're unit testing with Sqlite, you might find it convenient to use the :memory: option, which keeps the database in RAM rather than on disk. That should make the setup and tear-down of the testing environment easier to maintain. I had noted this option, but I don't think I would have ever come up with your idea on my own. Thanks! The principle is similar to that of when we're unit testing functions whose inputs or outputs are file-like objects; although we can use real files, we can find in-memory structures like io.StringIO useful, since they leave no residue once the tests are completed. Likewise thanks for opening my eyes to this perspective as well! -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On Sun, Aug 16, 2015 at 6:04 PM, Alan Gauld alan.ga...@btinternet.com wrote: On 16/08/15 23:29, boB Stepp wrote: http://www.sqlite.org/download.html You definitely want this. You treat it like the prompt in Pyython. I had just finished installing and testing the installation just before your email arrived. Very easy to do! A place to try out SQL queries before you put them into your Python code. Also you can write long sql code in a .sql filer and read them into the interpreter using the .read command sqlite3 .read createdb.sql sqlite3 .read populate_tests.sql sqlite3 .read clear_tables.sql sqlite3 .read populate_base_data.sql I am assuming that the .read command would be replaced inside the program by the cursor.executescript() method you mentioned? This will be quite handy, I think. Also you can ask the interpreter to describe table structures and constraints etc. Think TDD here... Would you mind giving a specific example illustrating how this would aid testing? Being an insane accumulator of books, I have acquired, The Definitive Guide to SQLite, 2nd ed., by Grand Allen and Mike Owens, c. 2010. I haven't made it yet to the commands you are alluding to. design tool as the command-line SQLite would allow me to easily get a visual representation of the tables, etc., Depends what you mean by visual! I just meant getting a text display of table contents in neat aligned columns. The book I mentioned has already showed me how to do this. This will be more than plenty for me. At this time, while concentrating on learning SQL, I don't want to take any GUI shortcuts. is that I would unknowingly structure my test db differently than what my program would generate. As I said, create separate .sql files to create an empty database and to populate the structure with initial data. That way you use the exact same structure file in your code as in your tests (remember the cursor.executescript() method I mentioned earlier! - that works from inside your code as well as the prompt!) This looks okay from the bare bones standpoint. However, I am wondering how much pre-existing structure I should impose on the user, and how much flexibility I should allow the user to have to extend the db structure as shipped. For instance I can easily see the user wanting to add new types of information she wants to track for her student. I think the program should allow her to do things like this, but while totally hiding the SQL interactions behind the scenes from the user. But allowing this sort of functionality automatically creates differences between what I've been testing in development and what the user creates through normal program use. I suppose, however, that I can design test cases that mimics all allowed such functionality. But in this example, how many new columns of table data do I cut off the tests at, for instance? Eventually RAM or some kind of overflow condition will transpire, which suggests if I want to test that a *lot* of table columns can be added without breaking anything will require me to set some reasonable upper limit for what constitutes a *lot*. boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to test my code's interactions with SQLite db?
On Sun, Aug 16, 2015 at 7:55 PM, Alan Gauld alan.ga...@btinternet.com wrote: On 17/08/15 00:52, boB Stepp wrote: sqlite3 .read populate_base_data.sql I am assuming that the .read command would be replaced inside the program by the cursor.executescript() method you mentioned? This will be quite handy, I think. Oops! I wrote the above paragraph in a different form, thinking that .read was in the Python DB API in some form, too, then went to the docs to verify this, and saw that the method you mentioned, executescript(), was serving the interpreter's .read command's function, and did not rewrite the paragraph appropriately. I knew what you meant, though. and how much flexibility I should allow the user to have to extend the db structure as shipped. For instance I can easily see the user wanting to add new types of information she wants to track for her student. I'd allow a small amount of leeway here. The usual approach is to use string fields for the data and a meta table with field, type, name information. So if you have a student table with three user defined fields UDF1, UDF2,UDF3. You define a meta table called meta_student that has field, name, type columns containing things like: UDF1, BOOL, hasTattoos UDF2, FILENAME, Mugshot UDF3, STRING, Birthmark (You could have a single meta_data table with table as first column but that just makes your SQL even more complex IMHO! Small tables are OK. Also some implementations allow user specific definitions but that breaks badly if records are shared if, for example, user1.UDF1 is hasTattoos and user2.UDF1 is isPregnant) BUT! You need a lot of infra-structure around this since these fields will be global to all users. And you need to manually do any data validation of input to ensure that only valid data gets stored. You will need a bank of supported types/validator functions. And your UI needs to ensure users can only create appropriate options (think drop down lists and dictionaries). Hmm. This is much more complex than I imagined, and I'm cynical by nature. Personally I'd only add this flexibility if you really need it. Its a major code overhead and a big performance hit too. Well, while this is a one-teacher project only, there will be constant back-and-forth between me and said teacher, so I really would not need this for her. However, if this program actually gets created and works well, then I can see other teachers in the school using it, and they may have different data they want to track. So I was thinking of making it very flexible in this regard. Maybe I should hold off on this until I see how what I need to do now plays out. allowed such functionality. But in this example, how many new columns of table data do I cut off the tests at, for instance? I've seen commercial systems that allow up to 100 UDFs. In practice I'd go for 3-10. if they need more than 10 then your requirements capture and user testing was very poor! [...] The biggest issue, as with user defined attributes in objects is that your static code doesn't know what these new fields are. Imagine how they play with a search function? Building a SQL query against a boolean value in the UDF1 column means translating the boolean result into its string equivalent and doing a fill scan text search. Slow You really want database tables to be defined as fully as possible as soon as possible. UDFs are a (sometimes necessary) cost of failure! I'm glad I think to ask these questions! And that you folks are here to answer them!! boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
In a message of Sun, 16 Aug 2015 12:46:59 -0700, Alex Kleider writes: On 2015-08-16 10:45, Alan Gauld wrote: Thee are several options. 1) create links from, main to the test files needed 2) alter sys.path so imports can see the test folder 3) alter the PYTHONPATH environment var I suspect in this case the easiest solution is a link Thanks Alan. Creating a link is familiar to me but your answer brings up another question: Where/how is the best place/way to set PYTHONPATH? I've never been clear to me if it's part of the shell (bash) environment or part of the python interpreter's environment. Alex bash. If you share your computer with somebody they each want their own. Laura ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Complicating a simple expression (Python 3)
Complicating a simple expression Coding Exercise: Complication Assume that the grader defines two variables A and B for you. Write a program which prints out the value min(A, B) However, there is a catch: your program is not allowed to use the min function. Instead, use max in a clever way to simulate min. Hint, Method 1 What is max(-A, -B)? Hint, Method 2 What is min(A, B)+max(A, B)? My last code that worked somewhat --- Original = max(A, B) Opposite = max(-A, -B) Grader =max(-A,- B)+max(A, B) print (Grader) Did not pass tests. Please check details below and try again. Results for test case 1 out of 5 Before running your code: We defined A equal to 62 and B equal to 36. Program executed without crashing. Program output: 26 Expected this correct output: 36 Result of grading: Your output is not correct. Spreadsheet examples: ABMax(A, B)Min(A, B)Min(A, B)+Max(A, B)Max(A, B)+Max(A, B)Min(A, B)+Min(A, B) 105105152010 510105152010 912129212418 129129212418 22373722597444 37223722597444 4568684511313690 6845684511313690 Max(-A,- B)Max(-A,- B)+Max(A, B)Max(-A,- B)-Max(A, B) -55-15 -55-15 -93-21 -93-21 -2215-59 -2215-59 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Writing back to same CSV in the next column
On 16/08/15 22:42, Nym City wrote:
import socket
import csv
You don't need csv, you aren't using it.
in_file = open('top500ips.csv', 'r')
out_file = open('top500ips_out.csv', 'w')
for line in in_file:
try:
name = socket.gethostbyaddr(line.strip())
out_file.write(line + '\t' + (str(name))
count the parens in the line above...
except socket.herror:
out_file.write(line + '\t' + errrMsg)
in_file.close()
out_file.close()
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
In a message of Sun, 16 Aug 2015 18:45:31 +0100, Alan Gauld writes: On 16/08/15 16:41, Alex Kleider wrote: - src the code -- lang folder per language used - sql, python, C, bash, etc --- lib modules/packages - subfolder per package --- test test code - sub-tree under this, depends on test tools. --- toolstools used but not shipped - db load/reset etc --- main folder in some languages, a file in others Alan, Assuming the above structure and further assuming that your python test suite is under test, how do you arrange to import code that is under main? Thee are several options. 1) create links from, main to the test files needed 2) alter sys.path so imports can see the test folder 3) alter the PYTHONPATH environment var But in most of my Python projects main is a file rather than a folder so test is under main anyway. This above is a generic structure because most of my projects use 4 or 5 languages at least. Some IDEs require different structures too so the separate lang sub-structures may vary to suit the toolset usee. I've not been able to do that. Is there a way? I suspect in this case the easiest solution is a link (aka shortcut in windoze) We have a new mechanism for test discovery in 2.7 and 3.x https://docs.python.org/3/library/unittest.html see 26.3.3 It's been backported. see: https://pypi.python.org/pypi/unittest2 Also, if you are on Python2 and you put your tests in a test subdirectory you need to remember to make an __init__.py file there. Laura ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
On 16/08/15 20:46, Alex Kleider wrote: Where/how is the best place/way to set PYTHONPATH? I've never been clear to me if it's part of the shell (bash) environment or part of the python interpreter's environment. The environment is user specific, so whichever shell the user has (zsh, tcsh, bash etc) sets the environment. When you launch a command that environment is inherited by the command, so the Python interpreter inherits the user environment from which it is launched. This can be a cause of obscure bugs when one user has a different environment to another (eg the developer!) Usually there is a global environment set up in the /etc versions of the shell but a user can overwrite those settings. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
On 2015-08-16 12:45, Laura Creighton wrote: We have a new mechanism for test discovery in 2.7 and 3.x https://docs.python.org/3/library/unittest.html see 26.3.3 It's been backported. see: https://pypi.python.org/pypi/unittest2 Also, if you are on Python2 and you put your tests in a test subdirectory you need to remember to make an __init__.py file there. Laura I've got it working. Thanks, Laura. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Where should unit test files go in a project directory structure and some related questions?
On 2015-08-16 10:45, Alan Gauld wrote: Thee are several options. 1) create links from, main to the test files needed 2) alter sys.path so imports can see the test folder 3) alter the PYTHONPATH environment var I suspect in this case the easiest solution is a link Thanks Alan. Creating a link is familiar to me but your answer brings up another question: Where/how is the best place/way to set PYTHONPATH? I've never been clear to me if it's part of the shell (bash) environment or part of the python interpreter's environment. Alex ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Complicating a simple expression (Python 3)
On 16/08/15 21:28, Joseph Gulizia wrote: Assume that the grader defines two variables A and B for you. Write a program which prints out the value min(A, B) So far a trivial exercise. However, there is a catch: your program is not allowed to use the min function. Instead, use max in a clever way to simulate min. Now its just a stupid exercise. I rally hate when teachers do this. It shows great lack of imagination and actively teaches bad habits... sigh! Hint, Method 1 What is max(-A, -B)? Hint, Method 2 What is min(A, B)+max(A, B)? Note that these are two separate methods. You don't need both of them. In fact you don't really need either of them! Writing the min functionality directly is not exactly difficult! My last code that worked somewhat --- Original = max(A, B) Opposite = max(-A, -B) Grader =max(-A,- B)+max(A, B) print (Grader) OK, Can you explain how you thought that might work? Before running your code: We defined A equal to 62 and B equal to 36. 26 Which is what doing the math suggests: max (-62, -36) is -36 and max(62,36) is 62 so 62 + (-36) is 26 Expected this correct output: 36 Which is the correct min of 36 and 62 Result of grading: Your output is not correct. Spreadsheet examples: ABMax(A, B)Min(A, B)Min(A, B)+Max(A, B)Max(A, B)+Max(A, B)Min(A, B)+Min(A, B) 105105152010 510105152010 912129212418 129129212418 22373722597444 37223722597444 4568684511313690 6845684511313690 Max(-A,- B)Max(-A,- B)+Max(A, B)Max(-A,- B)-Max(A, B) -55-15 -55-15 -93-21 -93-21 -2215-59 -2215-59 This probably lost a little in (email) translation, but looks like overkill to me. Choose one of the two methods above, since that's what they ask you to do. But it's just as easy to implement min() directly: if A B: min = A else: min = B HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
