Re: [Tutor] Stupid bug

[Wayne Werner]

On Wed, Nov 10, 2010 at 5:16 PM, Terry Carroll  wrote:

> This isn't a question, I'm just offering it as a cautionary tale and an
> opportunity to laugh at my own stupidity.
> 
> Turns out my function was working correctly all along; but with my typo, I
> was printing out the value from the first checksum each time.  Doh!
>
> Well, no harm done, other than wasted time, and I did turn up a silly but
> harmless off-by-one error in the process.\


I wish I could say that has never bitten me... but it has, several times.
Including the sister bug - continually importing something and you still get
the old function because you forgot to delete the .pyc file. Whoops!

-Wayne
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Data Directory under site-packages

[Terry Carroll]

On Wed, 10 Nov 2010, Greg Lindstrom wrote:


I'm writing my first module that I intend to put under our company's
"site-packages" directory for everyone to use in their programs.  The
problem I'm having is that I want to place files in a data directory under
the module directory (under site-packages) and I don't know how to set the
path so I pick up the files.  If I use open('./data/myfile') I get the path
of the file importing the module (which could be just about anywhere).  I've
tried various combinations using os.path.abspath() and os.path.dirname() but
have the same problem.  Is there a way I can use files in the subdirectory
(I really do not want dozens more files in the main directory)?


I'm not sure I follow.

You want to put data, i.e., non-python code, in the import path?  That 
sounds unusual to me.


You can find the filename from which a module is imported with the 
module's __file__ attribute; and then os.path.dirname() can get you the 
directory.  So if you wanted to address a subdirectory named "data" in the 
same directory from which you imported a given module, or a file 
"myfile.txt" in that subdirectory, that's possible.


Using the sqlite module as an example on my system:


import sqlite3
sqlite3.__file__

'C:\\Python26\\lib\\sqlite3\\__init__.pyc'

import os
os.path.dirname(sqlite3.__file__)

'C:\\Python26\\lib\\sqlite3'

os.path.join(os.path.dirname(sqlite3.__file__), "data")

'C:\\Python26\\lib\\sqlite3\\data'

os.path.join(os.path.dirname(sqlite3.__file__), "data", "myfile.txt")

'C:\\Python26\\lib\\sqlite3\\data\\myfile.txt'


Is this the kind of thing you're thinking of?

Again, it's highly unusual to put non-code data in the import path; I've 
never heard of this being done before, and it makes me shiver in 
revulsion.  I'm not sure I can articulate exactly what bad effects it will 
have, apart from the inherent messiness of it, but I don't like it.


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Data Directory under site-packages

[Greg Lindstrom]

Hello,

I'm writing my first module that I intend to put under our company's 
"site-packages" directory for everyone to use in their programs.  The problem 
I'm having is that I want to place files in a data directory under the module 
directory (under site-packages) and I don't know how to set the path so I pick 
up the files.  If I use open('./data/myfile') I get the path of the file 
importing the module (which could be just about anywhere).  I've tried various 
combinations using os.path.abspath() and os.path.dirname() but have the same 
problem.  Is there a way I can use files in the subdirectory (I really do not 
want dozens more files in the main directory)?

Thanks for your help,
--greg



CONFIDENTIALITY NOTICE:  This communication contains information 
intended for the use of the individuals to whom it is addressed 
and may contain information that is privileged, confidential or 
exempt from other disclosure under applicable law.  If you are 
not the intended recipient, you are notified that any disclosure, 
printing, copying, distribution or use of the contents is prohibited.  
If you have received this in error, please notify the sender 
immediately by telephone or by returning it by return mail and then 
permanently delete the communication from your system.  Thank you.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] put query result set into dictionary

[Alan Gauld]

"Shawn Matlock"  wrote


I am trying to put the results of a MySQL query into a dictionary.
It fails because it includes the Unicode delimiter - "u'somevalue'".


I don't think so...

envsql = "select envVariable, envValue from ODS_ENV_DICT where 
envName = 'ST2'"

csdbCursor.execute(envsql)



envDict = {}
for envVariable, envValue in csdbCursor.fetchall():
envDict[envVariable].append(envValue)


Append is a list operation but you don;t have a list (yet).
You are just adding a single value to a dictionary.
You need something like:
 envDict[envVariable] = envValue

Or if you really want a list you need to check if the value is
there first and then append. You could do that with setdefault() like 
so:


 envDict.setdefault(envVariable, []).append(envValue)

Which returns the list if one exists or an empty list if it doesn't
and appends the value to it, it then assigns the resulting list
back to the dictionary at the key position


Traceback (most recent call last):
  File "H:\workspace\test\src\root\nested\example.py", line 33, in 


   envDict[envVariable].append(envValue)
KeyError: u'envDB'


Despite the KeyError message its really because you are trying
to use append on a non existent list:


d = {}
d[5].append(7)

Traceback (most recent call last):
 File "", line 1, in 
KeyError: 5

Same error, no unicode in sight.

HTH,

--
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] unicode nightmare

[Alan Gauld]

"danielle davout"  wrote


I simplify it to
v = u'\u0eb4'
X = (1,)
gen = ((v ,v) for x in X for y in X)

What can be so wrong in this line, around it to give the 1lined file
ໄ:ໄ
where ໄ "is" not u'\u0eb4' but  u'\u0ec4' though a direct printing 
looks OK


The code will produce a one line file with v repeated twice.
Now why do you think the character is different?
What have you done to check it?

What do you mean by a direct printing?

print v

maybe?

To write the file corresponding to my nth generator of my list h I 
use

   def ecrire(n):
   f= codecs.open("G"+str(n),"w","utf8")
   for x, tx in h[n]:
   f.write((x + U":"+ tx))
   f.write('\n')


Personally I'd use

f.write(U"%s:%s\n" % (x,tx))

but thats largely a matter of style preference I guess.
But why do you have double parens in the first print?


But In its non simplified form
   h.append( (x + v + y ,tr[x]+ tr[v]+ tr[y]) for x in CC for y in 
OFC) )

before I  have a chance to write anything in the file G5
I have got the KeyError: u'\u0ec4'
yes tr is a dictionary that doesn't have u'\u0ec4' as a key
but tr[v] is well definied ...


OK, but the error is valid in that case.
Which implies that you have bad data in CC.

What exactly are you asking?

--
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] put query result set into dictionary

[Joel Goldstick]

On Wed, Nov 10, 2010 at 5:18 PM, Shawn Matlock wrote:

>  Let’s see if I can ask the right questions the right way. I am trying to
> put the results of a MySQL query into a dictionary. It fails because it
> includes the Unicode delimiter - “u’somevalue’”. Can someone tell me the
> right way to do this, please?
>
>
>
> Thank you,
>
> Shawn
>
> Notice that you are returning strings.  "u'somevalue'" . Your problem is
not with the unicode, its with the quotes that wrap the unicode delimiter.
I haven't used zxJDBC object, but I am guessing if you look into its
documentation you can return what you really want, which is a unicode
string, not a string containing the representation of a unicode string

>
>
> csdbConn = zxJDBC.connect("jdbc:mysql://myhost:3306/mydb", "User",
> "Password", "com.mysql.jdbc.Driver")
>
>
>
> csdbCursor = csdbConn.cursor(1)
>
>
>
> envsql = "select envVariable, envValue from ODS_ENV_DICT where envName =
> 'ST2'"
>
> print 'envsql: ' + envsql
>
>
>
> csdbCursor.execute(envsql)
>
>
>
> envDict = {}
>
> for envVariable, envValue in csdbCursor.fetchall():
>
> print envVariable, envValue
>
> envDict[envVariable].append(envValue)
>
>
>
> csdbCursor.close()
>
> csdbConn.close()
>
>
>
> --
>
> envsql: select envVariable, envValue from ODS_ENV_DICT where envName =
> 'ST2'
>
> envDB systest2
>
> Traceback (most recent call last):
>
>   *File "H:\workspace\test\src\root\nested\example.py", line 33, in
> *
>
> envDict[envVariable].append(envValue)
>
> KeyError: u'envDB'
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
>


-- 
Joel Goldstick
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Stupid bug

[Terry Carroll]

This isn't a question, I'm just offering it as a cautionary tale and an 
opportunity to laugh at my own stupidity.


I have a small function to calculate the MD5 checksum for a file.   It's 
nothing fancy:


###
import hashlib
def md5(filename, bufsize=65536):
"""
Compute md5 hash of the named file
bufsize is 64K by default
"""
m = hashlib.md5()
with open(filename,"rb") as fd:
content = fd.read(bufsize)
while content != "":
m.update(content)
content = fd.read(bufsize)
return m.hexdigest()
###

I've discovered a need to calculate the checksum on the first 10K or so 
bytes of the file (faster when processing a whole CDROM or DVDROM full of 
large files; and also allows me to find when one file is a truncated copy 
of another).


This seemed like an easy enough variation, and I came up with something 
like this:


###
def md5_partial(filename, bufsize=65536, numbytes=10240):
"""
Compute md5 hash of the first numbytes (10K by default) of named file
bufsize is 64K by default
"""
m = hashlib.md5()
with open(filename,"rb") as fd:
bytes_left = numbytes
bytes_to_read = min(bytes_left, bufsize)
content = fd.read(bytes_to_read)
bytes_left = bytes_left - bytes_to_read
while content != "" and bytes_left >0:
m.update(content)
bytes_to_read=min(bytes_left, bufsize)
content = fd.read(bytes_to_read)
bytes_left = bytes_left - bytes_to_read
return m.hexdigest()
###

Okay, not elegant, and violates DRY a little bit, but what the heck.

I set up a small file (a few hundred bytes) and confirmed that md5 and 
md5_partial both returned the same value (where the number of bytes I was 
sampling exceeded the size of the file).  Great, working as desired.


But then when I tried a larger file, I was still getting the same checksum 
for both.  It was clearly processing the entire file.


I started messing with it; putting in counters and print statements, 
using the Gettysburg Address as sample daya and iterating over 
20 bytes at a time, printing out each one, making sure it stopped 
appropriately.  Still no luck.


I spent 90 minutes over two sessions when I finally found my error.

My invocation of the first checksum was:

###
checksumvalue = my.hashing.md5("filename.txt")
# (Not an error: I keep my own modules in Lib/site-packages/my/ )
print checksumvalue
#
# [several lines of code that among other things, define my new
# function being tested]
#
checksumvalue2 = md5_partial("filename.txt", numbytes=200
print checksumvalue

Turns out my function was working correctly all along; but with my typo, I 
was printing out the value from the first checksum each time.  Doh!


Well, no harm done, other than wasted time, and I did turn up a silly but 
harmless off-by-one error in the process.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] variables question.

[Luke Paireepinart]

It's pretty typical for scoping rules. If you define variables outside of your 
funcs you can access them inside the func. If you want to change the value in 
the function you need to declare the scope as global. If you make another 
variable with the same name inside of a function it will shadow the outer 
variable. You could define the variables in a parent script if you want. This 
is actually how imports work basically.

In config.py
foo = "bar"

In your py
from config import foo
def bar():
print foo

-
Sent from a mobile device with a bad e-mail client.
-

On Nov 10, 2010, at 12:21 PM, Jeff Honey  wrote:

> I have a question about where variables are exposed in python.
> 
> I have a monolothic script with a number of functions defined, can those 
> functions share variables? can I instantiate them outside the function of 
> where they are needed? do they need to be wrapped in quotes, ever? For 
> example:
> 
> blah = 123
> foo = 'somestring'
> 
> def function(foo):
>code that needs foo
>anotherfunction(blah) 
> 
> def anotherfunction(blah):
>code that needs blah
>code that uses foo
> 
> how about this:
> 
> def function(blah, foo):
>anotherfunc(blah)
>anotherfunc(foo)
> 
> ...what about my python being called from some parent script (something OTHER 
> than python) that instantiates blah and foo FOR me? Can I just plug those 
> into my code like I would normally? I guess this is more about HOW and WHERE 
> I can make variables available for use.
> 
> --
> ¤¤
> ¤ kyoboku kazeoshi ¤
> ¤¤
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] put query result set into dictionary

[Shawn Matlock]

Let's see if I can ask the right questions the right way. I am trying to put 
the results of a MySQL query into a dictionary. It fails because it includes 
the Unicode delimiter - "u'somevalue'". Can someone tell me the right way to do 
this, please?



Thank you,

Shawn



csdbConn = zxJDBC.connect("jdbc:mysql://myhost:3306/mydb", "User", "Password", 
"com.mysql.jdbc.Driver")



csdbCursor = csdbConn.cursor(1)



envsql = "select envVariable, envValue from ODS_ENV_DICT where envName = 'ST2'"

print 'envsql: ' + envsql



csdbCursor.execute(envsql)



envDict = {}

for envVariable, envValue in csdbCursor.fetchall():

print envVariable, envValue

envDict[envVariable].append(envValue)



csdbCursor.close()

csdbConn.close()



--
envsql: select envVariable, envValue from ODS_ENV_DICT where envName = 'ST2'
envDB systest2
Traceback (most recent call last):
  File "H:\workspace\test\src\root\nested\example.py", line 33, in 
envDict[envVariable].append(envValue)

KeyError: u'envDB'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] variables question.

[Emile van Sebille]

On 11/10/2010 10:21 AM Jeff Honey said...

I have a question about where variables are exposed in python.


You're looking for scoping rules -- see for example 
http://stackoverflow.com/questions/291978/short-description-of-python-scoping-rules 
where they get into some detail, but the short and older version is 1) 
search in local-global-builtin, and 2) if you assign to it in a scope, 
it's local to that scope.


So, blah and foo below are visible in all the functions except any that 
specifically assign to blah or foo.


HTH,

Emile




I have a monolothic script with a number of functions defined, can those 
functions share variables? can I instantiate them outside the function of where 
they are needed? do they need to be wrapped in quotes, ever? For example:

blah = 123
foo = 'somestring'

def function(foo):
 code that needs foo
 anotherfunction(blah)

def anotherfunction(blah):
 code that needs blah
 code that uses foo

how about this:

def function(blah, foo):
 anotherfunc(blah)
 anotherfunc(foo)

...what about my python being called from some parent script (something OTHER 
than python) that instantiates blah and foo FOR me? Can I just plug those into 
my code like I would normally? I guess this is more about HOW and WHERE I can 
make variables available for use.

--
  ¤¤
¤ kyoboku kazeoshi ¤
  ¤¤
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor




___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Tutor Digest, Vol 81, Issue 39

[Alan Gauld]

"marupalli charan"  wrote


Thanks, but first one not working!!!


First one of what?

Please edit the message to pprovide some context
and pay attention to the instruxctions that you so helpfully
posted:


When replying, please edit your Subject line so it is more specific
than "Re: Contents of Tutor digest..."


Today's Topics:

   1. Re: List comprehension question (Richard D. Moores)
   2. Re: query result set (bob gailer)
   3. Re: Columnar Transposition Cipher question (Stefan Behnel)
   4. Re: Columnar Transposition Cipher question (Eike Welk)
   5. Re: List comprehension question (Steven D'Aprano)
   6. Re: Columnar Transposition Cipher question (Steven D'Aprano)
   7. Re: List comprehension question (Albert-Jan Roskam)


--

Message: 1
Date: Tue, 9 Nov 2010 08:17:55 -0800
From: "Richard D. Moores" 
To: "Martin A. Brown" 
Cc: tutor@python.org
Subject: Re: [Tutor] List comprehension question
Message-ID:

Content-Type: text/plain; charset=UTF-8

On Tue, Nov 9, 2010 at 05:51, Martin A. Brown  
wrote:


Hello,

?: def proper_divisors_sum(n):
?: ? ? pd_list = []
?: ? ? for x in range(1, int(n**.5)+1):
?: ? ? ? ? if n % x == 0:
?: ? ? ? ? ? ? factor = int(x)
?: ? ? ? ? ? ? pd_list.append(factor)
?: ? ? ? ? ? ? factor2 = int(n/factor)
?: ? ? ? ? ? ? pd_list.append(factor2)
?: ? ? pd_list = list(set(pd_list))
?: ? ? pd_list.sort()
?: ? ? pd_list = pd_list[:-1]
?: ? ? return sum(pd_list)

A few questions--noting, of course, that I'm not reading this with
an eye toward performance, which it seems you are, but these occur
to me:

?* Why use a list (pd_list = []), when you want unique divisors?
? ?Why not, pd = set() ? ?Then, instead of pd_list.append(factor),
? ?pd.add(factor) or something like pd.update((factor,factor2))?
? ?(See also my suggestion below.)

?* Also, since you are throwing away the divisor n, anyway,
? ?why not skip it from the beginning? ?Like this:

? ? ?pd_list = [1,]
? ? ?for x in range(2, int(n**.5)+1):

?* So, I'm not terribly math aware, but I don't think I have
? ?substantially changed the meaning of your program. ?I find
? ?breaking out the functions makes things a bit clearer to
? ?somebody who might be reading my code later.

Combining these suggestions and splitting into separate functions
(you don't really need to sort before summing), I end up with the
below. ?Note, that I stuff the proper divisor 1 in the set at
creation time. ?This is probably not worth it from an optimization
perspective, but as I have learned, the only way to know is to 
time

it (and I didn't time it).

? ? ?def proper_divisors_sum(n):
? ? ? ? ?return sum(proper_divisors(n))

? ? ?def proper_divisors_sorted(n):
? ? ? ? ?return sorted(proper_divisors(n))

? ? ?def proper_divisors(n):
? ? ? ? ?pd = set((1,))
? ? ? ? ?for x in range(2, int(n**.5)+1):
? ? ? ? ? ? ?factor, mod = divmod(n,x)
? ? ? ? ? ? ?if mod == 0:
? ? ? ? ? ? ? ? ?pd.update((x,factor))
? ? ? ? ?return list(pd)


Thanks for your ideas, Martin. I adopted your idea of starting at 
2,

with an initial list of [1].  It resulted in a small increase in
speed. And you're right, I don't need to sort.

Speed is important, because the function is the heart of my amiable
numbers script. Did you happen to see that post of mine?

#Here's my fastest version before seeing Martin's post:
def proper_divisors_sum1(n):
pd_list = []
for x in range(1, int(n**.5)+1):
if n % x == 0:
pd_list.append(x)
pd_list.append(n//x)
pd_list = list(set(pd_list))
pd_list.sort()
pd_list = pd_list[:-1]
return sum(pd_list)

#This uses several of Martin's ideas
# It's now my fastest version
def proper_divisors_sum2(n):
pd = set((1,))
for x in range(2, int(n**.5)+1):
if n % x == 0:
pd.update((x, n//x))
return sum(list(pd))

Here's what I put together from all your points, except for the
splitting off of several small functions:

# This incorporates all of Martin's ideas
# Seems that divmod() is a speed killer
def proper_divisors_sum3(n):
pd = set((1,))
for x in range(2, int(n**.5)+1):
factor, mod = divmod(n,x)
if mod == 0:
pd.update((x,factor))
return sum(list(pd))

It may be that I've haven't done your suggestions justice, but that
function is 76% slower than proper_divisors_sum2().

See  for a speed test with n 
=

100,000 and 100,000 loops

Thanks again, Martin.

Dick


--

Message: 2
Date: Tue, 09 Nov 2010 12:00:56 -0500
From: bob gailer 
To: tutor@python.org
Subject: Re: [Tutor] query result set
Message-ID: <4cd97e48.9080...@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

When starting a new subject, please start with a new email. Do not
"hijack" an existing one, as that causes your query to appear as 
part of

the hijacked thread.


--
Bob Gailer
919-636-4239
Chapel Hill NC

Re: [Tutor] variables question.

[Alan Gauld]

"Jeff Honey"  wrote


I have a question about where variables are exposed in python.


Take a look at the namespaces topic in my tutor for more details 
but...



I have a monolothic script with a number of functions defined,
can those functions share variables?


Yes but its usually bad practice.
Python considers module level variables to be "global".
global vars can be read inside a functon within the same module.
global vars can be changed inside a function in the same module 
provided they are declared as global within the function:


foo = 42

def f():
 global foo   # use the global var
 foo = 66
 return foo

print foo
print f()
print foo

can I instantiate them outside the function of where they are 
needed?


Yes, see foo above


do they need to be wrapped in quotes, ever? For example:

blah = 123


blah refers to an integer


foo = 'somestring'


foo refers to a string. literal strings need quotes

bar = raw_uinput('?')# bar refers to a strintg input by the user


def function(foo):
code that needs foo
anotherfunction(blah)


No problem


def anotherfunction(blah):
code that needs blah
code that uses foo


This needs a global foo line added at the top.


def function(blah, foo):
anotherfunc(blah)
anotherfunc(foo)


No problem. blah and foo are local variables inside function().
They are passed as arguments to anotherfunc() whioch weill
assign the objects to their parameters and treat as local
variables within anotherfunc()


..what about my python being called from some parent script
(something OTHER than python)


Thats possible but trickier


that instantiates blah and foo FOR me?


As is this.

But bopth are easier if its another Python script.


Can I just plug those into my code like I would normally?


No.


I guess this is more about HOW and WHERE I can
make variables available for use.


See my tutorial topic...


--
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] List comprehension question

[Steven D'Aprano]

Richard D. Moores wrote:

On Tue, Nov 9, 2010 at 12:54, Steven D'Aprano  wrote:

Richard D. Moores wrote:


See  for a speed test with n =
100,000 and 100,000 loops

As a general rule, you shouldn't try to roll your own speed tests. There are
various subtleties that can throw your results right out. Timing small code
snippets on modern multi-tasking computers is fraught with difficulties.


Yes, but I thought that if I ran the tests the way I did, several
times and then reversing the order of the functions, and seeing that
the results were very similar, I could be pretty sure of my findings.
Could you point out where taking even that amount of care could lead
me astray?


Well, the biggest problem is that you're re-inventing the wheel. But in 
truth, what you did was not terrible, particularly if you're using 
Python 3 where range() is quite lightweight rather than a heavy 
operation that creates a big list.


There are various traps when timing code:

* There are commonly two functions you use for getting the time: 
time.time() and time.clock(). On Windows, the best one to use is 
clock(), but on other operating systems, you should probably use time(). 
 The timeit module automatically chooses the right one for you.


* Comparing apples and oranges. There's not a lot that timeit can do 
there, since it requires you *understand* what the code is that you are 
timing, but it helps by allowing you to split initialisation away from 
the code you are timing.


* Timing too much. One trap is to do this:

t = time.time()
for _ in range(100):
do_something()
t = time.time() - t

This is especially bad in Python 2.x!

The problem is that you are including the time it takes to create a list 
of one million integers in your timing code. That's like starting the 
stop watch at a race while the athletes are still in the changing rooms, 
getting dressed, and yet you'd be amazed how often people do it! If the 
time taken by each call to do_something() is very large, then the extra 
time will be negligible, but if do_something is a fast snippet, it might 
take you nearly as long to build the list as it does to run the code 
you're trying to time!


timeit ensures that as little as possible overhead is included in the 
execution being timed.


* Forgetting that computers are multi-tasking machines these days. At 
every instant, the operating system is giving tiny time-slices to your 
program and potentially dozens of others. If you're busy loading a dozen 
web pages, playing a DVD, compiling a large program and trying to time a 
code snippet, it will be obvious that everything is going to be running 
slow. But if you time something *once*, what's not obvious is that just 
by chance you might get a number that is much slower than normal because 
the OS *just happened* to call your anti-virus during the test. timeit 
fights against that by running your code in a loop, and then working out 
the average time per loop, *and* then repeating the loop multiple times 
(three by default) and picking the lowest figure. Higher figures are 
slower due to external factors.


* Forgetting your CPU cache effects. Modern CPUs do all sorts of amazing 
wizardry to make code run fast. Sometimes slow, but hopefully more often 
fast. The first time you run a piece of code, chances are it will be 
slower than normal, because the hardware caches will have other stuff in 
them. But the second time, the code and data will already be lined up in 
the cache and pipelines ready to go. This is why if you time something a 
dozen times, the *first* time could be two or three times slower than 
the rest.



I've probably forgotten some... but anyway, the important thing is that 
there's lots of subtleties to timing code, and Tim Peters has already 
thought of them so you don't have to :)




--
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] unicode nightmare

[danielle davout]

Hi,
I really badly need any start of explanation ..
Please help me !
I have got  a list  of 64 generators from which I generate files.
58 give what I expected
1 is getting me mad, not the first, not the last the fifth...
I simplify it to
v = u'\u0eb4'
X = (1,)
gen = ((v ,v) for x in X for y in X)

What can be so wrong in this line, around it to give the 1lined file
ໄ:ໄ
where ໄ "is" not u'\u0eb4' but  u'\u0ec4' though a direct printing looks OK
To write the file corresponding to my nth generator of my list h I use
def ecrire(n):
f= codecs.open("G"+str(n),"w","utf8")
for x, tx in h[n]:
f.write((x + U":"+ tx))
f.write('\n')
f.close()
But In its non simplified form
h.append( (x + v + y ,tr[x]+ tr[v]+ tr[y]) for x in CC for y in OFC) )
 before I  have a chance to write anything in the file G5
I have got the KeyError: u'\u0ec4'
yes tr is a dictionary that doesn't have u'\u0ec4' as a key
but tr[v] is well definied ...

(# /usr/lib/python2.5/encodings/utf_8.pyc matches
/usr/lib/python2.5/encodings/utf_8.py
import encodings.utf_8 # precompiled from /usr/lib/python2.5/encodings/utf_8.pyc
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14)
[GCC 4.3.2] on linux2)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] List comprehension question

[Steven D'Aprano]

Richard D. Moores wrote:


def proper_divisors_sum(n):
return sum(list(divisors(n))) - n


There's no need to call list first. sum() will happily operate on any 
sort of iterable -- lists, sums, iterators, generators, range objects. 
Anything except strings, which would be pointless even if you could do 
it, which you can't, but even so you can fool sum() to work with strings 
with a bit of effort.




Using Steven's suggested speed test
this gets 6.2404818210135886

My up-to-now fastest version,

[...]

gets 6.1753780857622473

So they're about even.


I'd say that the difference is probably statistically insignificant. 
Even if it's consistent on your PC, on another PC with a slightly 
different processor, slightly different memory, or a different operating 
system, it could very well go the other way.


In any case, since those times are ~6 seconds per 100,000 loops, the 
real difference is only 60 microseconds -- completely trivial unless 
you're doing some *serious* number crunching.



P.S. don't take that as a put down -- you should be pleased that your 
code is around as fast as Tim Peter's code :)



--
Steven

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Need help with script for finding pairs of amicable numbers

[col speed]

>
> --
>
> Message: 2
> Date: Tue, 09 Nov 2010 12:35:26 +0100
> From: Stefan Behnel 
> To: tutor@python.org
> Subject: Re: [Tutor] List comprehension question
> Message-ID: 
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Richard D. Moores, 09.11.2010 12:07:
>  That sqrt(n) works for speeding up the finding of primes, but here we
>  want to use int(n/2) (and why didn't I think of that?), which results
>  in about a 2x speedup. See.
> >>>
> >>> NO! Use  int(n/2)+1 .  I'll correct that in
> >>>   and report back.
> >>
> >> See
> >>
> >> But now I'll have to redo again using Stefan's ingenious suggestion.
> >
> > Here's what I wrote using Stefan's suggestion:
> >
> > def proper_divisors_sum(n):
> >  pd_list = []
> >  for x in range(1, int(n**.5)+1):
> >  if n % x == 0:
> >  factor = int(x)
> >  factor2 = int(n/factor)
> >  pd_list.extend([factor, factor2])
> >  pd_list = list(set(pd_list))
> >  pd_list.sort()
> >  pd_list = pd_list[:-1]
> >  return sum(pd_list)
>
>
>
>
>
> -
> Just a thought, but maybe getting the sums using prime factors would be a
> bit faster?

Get the prime factors with their powers. Add 1 to the power and subtract 1
from that. Then divide that by the factor minus 1. multiply that by the next
factor. An example is easier:

Take 1800, it's prime factors are - 2**3, 3**2, 5**2. So we do:
((2**4-1)/1) * ((3**3-1)/2) * ((5**3-1)/4) = (15*26*124)/8 = 1800

I have a couple of programmes ( stolen from the internet) that work out the
prime factors quite quickly, if you are interested.

Regards
Colin
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] variables question.

[Jeff Honey]

I have a question about where variables are exposed in python.

I have a monolothic script with a number of functions defined, can those 
functions share variables? can I instantiate them outside the function of where 
they are needed? do they need to be wrapped in quotes, ever? For example:

blah = 123
foo = 'somestring'

def function(foo):
code that needs foo
anotherfunction(blah) 

def anotherfunction(blah):
code that needs blah
code that uses foo

how about this:

def function(blah, foo):
anotherfunc(blah)
anotherfunc(foo)

...what about my python being called from some parent script (something OTHER 
than python) that instantiates blah and foo FOR me? Can I just plug those into 
my code like I would normally? I guess this is more about HOW and WHERE I can 
make variables available for use.

--
 ¤¤
¤ kyoboku kazeoshi ¤
 ¤¤
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Tutor Digest, Vol 81, Issue 39

[marupalli charan]

Thanks, but first one not working!!!

On 11/10/10, tutor-requ...@python.org  wrote:
> Send Tutor mailing list submissions to
>   tutor@python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>   http://mail.python.org/mailman/listinfo/tutor
> or, via email, send a message with subject or body 'help' to
>   tutor-requ...@python.org
>
> You can reach the person managing the list at
>   tutor-ow...@python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Tutor digest..."
>
>
> Today's Topics:
>
>1. Re: List comprehension question (Richard D. Moores)
>2. Re: query result set (bob gailer)
>3. Re: Columnar Transposition Cipher question (Stefan Behnel)
>4. Re: Columnar Transposition Cipher question (Eike Welk)
>5. Re: List comprehension question (Steven D'Aprano)
>6. Re: Columnar Transposition Cipher question (Steven D'Aprano)
>7. Re: List comprehension question (Albert-Jan Roskam)
>
>
> --
>
> Message: 1
> Date: Tue, 9 Nov 2010 08:17:55 -0800
> From: "Richard D. Moores" 
> To: "Martin A. Brown" 
> Cc: tutor@python.org
> Subject: Re: [Tutor] List comprehension question
> Message-ID:
>   
> Content-Type: text/plain; charset=UTF-8
>
> On Tue, Nov 9, 2010 at 05:51, Martin A. Brown  wrote:
>>
>> Hello,
>>
>> ?: def proper_divisors_sum(n):
>> ?: ? ? pd_list = []
>> ?: ? ? for x in range(1, int(n**.5)+1):
>> ?: ? ? ? ? if n % x == 0:
>> ?: ? ? ? ? ? ? factor = int(x)
>> ?: ? ? ? ? ? ? pd_list.append(factor)
>> ?: ? ? ? ? ? ? factor2 = int(n/factor)
>> ?: ? ? ? ? ? ? pd_list.append(factor2)
>> ?: ? ? pd_list = list(set(pd_list))
>> ?: ? ? pd_list.sort()
>> ?: ? ? pd_list = pd_list[:-1]
>> ?: ? ? return sum(pd_list)
>>
>> A few questions--noting, of course, that I'm not reading this with
>> an eye toward performance, which it seems you are, but these occur
>> to me:
>>
>> ?* Why use a list (pd_list = []), when you want unique divisors?
>> ? ?Why not, pd = set() ? ?Then, instead of pd_list.append(factor),
>> ? ?pd.add(factor) or something like pd.update((factor,factor2))?
>> ? ?(See also my suggestion below.)
>>
>> ?* Also, since you are throwing away the divisor n, anyway,
>> ? ?why not skip it from the beginning? ?Like this:
>>
>> ? ? ?pd_list = [1,]
>> ? ? ?for x in range(2, int(n**.5)+1):
>>
>> ?* So, I'm not terribly math aware, but I don't think I have
>> ? ?substantially changed the meaning of your program. ?I find
>> ? ?breaking out the functions makes things a bit clearer to
>> ? ?somebody who might be reading my code later.
>>
>> Combining these suggestions and splitting into separate functions
>> (you don't really need to sort before summing), I end up with the
>> below. ?Note, that I stuff the proper divisor 1 in the set at
>> creation time. ?This is probably not worth it from an optimization
>> perspective, but as I have learned, the only way to know is to time
>> it (and I didn't time it).
>>
>> ? ? ?def proper_divisors_sum(n):
>> ? ? ? ? ?return sum(proper_divisors(n))
>>
>> ? ? ?def proper_divisors_sorted(n):
>> ? ? ? ? ?return sorted(proper_divisors(n))
>>
>> ? ? ?def proper_divisors(n):
>> ? ? ? ? ?pd = set((1,))
>> ? ? ? ? ?for x in range(2, int(n**.5)+1):
>> ? ? ? ? ? ? ?factor, mod = divmod(n,x)
>> ? ? ? ? ? ? ?if mod == 0:
>> ? ? ? ? ? ? ? ? ?pd.update((x,factor))
>> ? ? ? ? ?return list(pd)
>
> Thanks for your ideas, Martin. I adopted your idea of starting at 2,
> with an initial list of [1].  It resulted in a small increase in
> speed. And you're right, I don't need to sort.
>
> Speed is important, because the function is the heart of my amiable
> numbers script. Did you happen to see that post of mine?
>
> #Here's my fastest version before seeing Martin's post:
> def proper_divisors_sum1(n):
> pd_list = []
> for x in range(1, int(n**.5)+1):
> if n % x == 0:
> pd_list.append(x)
> pd_list.append(n//x)
> pd_list = list(set(pd_list))
> pd_list.sort()
> pd_list = pd_list[:-1]
> return sum(pd_list)
>
> #This uses several of Martin's ideas
> # It's now my fastest version
> def proper_divisors_sum2(n):
> pd = set((1,))
> for x in range(2, int(n**.5)+1):
> if n % x == 0:
> pd.update((x, n//x))
> return sum(list(pd))
>
> Here's what I put together from all your points, except for the
> splitting off of several small functions:
>
> # This incorporates all of Martin's ideas
> # Seems that divmod() is a speed killer
> def proper_divisors_sum3(n):
> pd = set((1,))
> for x in range(2, int(n**.5)+1):
> factor, mod = divmod(n,x)
> if mod == 0:
> pd.update((x,factor))
> return sum(list(pd))
>
> It may be that I've haven't done your suggestions justice, but that
> function is 76% slower than proper_divisors_sum2().
>
> See  for a speed test with n =
> 100,000 and 100,000 l