Re: [ANN] IPython 0.13 is officially out!
On 01-Jul-2012 13:56, Leo wrote: On 2012-07-01 01:55 +0800, Fernando Perez wrote: - ~6 months of work. - 373 pull requests merged. - 742 issues closed (non-pull requests). - contributions from 62 authors. - 1760 commits. - a diff of 114226 lines. Thank you for the hard work. Leo I have tried to update 0.12 in Ubuntu 12.04 but as of now it can not find 0.13. Any suggestions on how to get it into Ubuntu 12.04 would be appreciated. -- http://mail.python.org/mailman/listinfo/python-list
Re: Encapsulation, inheritance and polymorphism
On 20-Jul-2012 10:27, Steven D'Aprano wrote: On Fri, 20 Jul 2012 08:20:57 +1000, Chris Angelico wrote: Since the current evidence indicates the universe will just keep expanding, it's more of a "deep freeze death..." Heat death means *lack* of heat. The second law of thermodynamics states that energy tends to go from higher states to lower, with heat being the very lowest. It's possible to do work using (say) kinetic energy, and in the process, some of that energy becomes heat. It's also possible to do work with any difference in temperature (eg Stirling engines), so the state of the universe in which it's no longer possible to do any work will be one in which all energy is heat and everything's at the same temperature. That doesn't mean a lack of heat; in fact, it implies that there'll be rather more heat than there now is, because we currently have a whole lot of chemical energy available to be used. Yes, but the point is, that heat will be *incredibly* diffuse, essentially spread over the entire universe, which will be MUCH bigger than it is now, and hence the temperature will be low even though the total amount of heat will be high. The average temperature of the universe now is about 2.7 degrees above absolute zero (i.e. 2.7 K, -270.45 C or -454.81 F), with individual hotspots reaching into millions of degrees or higher. By the time the last of the stars burn out, the average temperature will be a minuscule fraction of a degree above absolute zero, and the only hotspots will be the slowly cooling neutron stars. But in any case, that's a long way off... I once went to an astronomy lecture where the lecturer was talking about the eventual death of the sun. He said, "In about 10 billion years, the sun will consume almost all of its fuel. It will cool and expand into a red giant, and the earth will be engulfed by the expanded sun and destroyed." This fellow sitting next to me got all agitated, stood up and cried out, "Does the government know about this? We have to do something!" The lecturer said "Don't worry sir, there's no need to panic, this won't happen for billions of years." The fellow looked relived and said "Oh thank god, I thought you said *million*!" How does this relate to the python list? "This mailing list is a general discussion list for the Python programming language." --- from http://mail.python.org/mailman/listinfo/python-list/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Strange behavior
On 2012-08-14 17:38, light1qu...@gmail.com wrote: Hi, I am migrating from PHP to Python and I am slightly confused. I am making a function that takes a startingList, finds all the strings in the list that begin with 'x', removes those strings and puts them into a xOnlyList. However if you run the code you will notice only one of the strings beginning with 'x' is removed from the startingList. If I comment out 'startingList.remove(str);' the code runs with both strings beginning with 'x' being put in the xOnlyList. Using the print statement I noticed that the second string that begins with 'x' isn't even identified by the function. Why does this happen? def testFunc(startingList): xOnlyList = []; for str in startingList: if (str[0] == 'x'): print str; xOnlyList.append(str) startingList.remove(str) #this seems to be the problem print xOnlyList; print startingList testFunc(['xasd', 'xjkl', 'sefwr', 'dfsews']) #Thanks for your help! You might find the following useful: def testFunc(startingList): xOnlyList = []; j = -1 for xl in startingList: if (xl[0] == 'x'): xOnlyList.append(xl) else: j += 1 startingList[j] = xl if j == -1: startingList = [] else: del startingList[j:-1] return(xOnlyList) testList1 = ['xasd', 'xjkl', 'sefwr', 'dfsews'] testList2 = ['xasd', 'xjkl', 'xsefwr', 'xdfsews'] testList3 = ['xasd', 'jkl', 'sefwr', 'dfsews'] testList4 = ['asd', 'jkl', 'sefwr', 'dfsews'] xOnlyList = testFunc(testList1) print 'xOnlyList = ',xOnlyList print 'testList = ',testList1 xOnlyList = testFunc(testList2) print 'xOnlyList = ',xOnlyList print 'testList = ',testList2 xOnlyList = testFunc(testList3) print 'xOnlyList = ',xOnlyList print 'testList = ',testList3 xOnlyList = testFunc(testList4) print 'xOnlyList = ',xOnlyList print 'testList = ',testList4 And here is another version using list comprehension that I prefer testList1 = ['xasd', 'xjkl', 'sefwr', 'dfsews'] testList2 = ['xasd', 'xjkl', 'xsefwr', 'xdfsews'] testList3 = ['xasd', 'jkl', 'sefwr', 'dfsews'] testList4 = ['asd', 'jkl', 'sefwr', 'dfsews'] def testFunc2(startingList): return([x for x in startingList if x[0] == 'x'], [x for x in startingList if x[0] != 'x']) xOnlyList,testList = testFunc2(testList1) print xOnlyList print testList xOnlyList,testList = testFunc2(testList2) print xOnlyList print testList xOnlyList,testList = testFunc2(testList3) print xOnlyList print testList xOnlyList,testList = testFunc2(testList4) print xOnlyList print testList -- http://mail.python.org/mailman/listinfo/python-list
Fwd: Re: Strange behavior
Original Message Subject:Re: Strange behavior Date: Tue, 14 Aug 2012 21:32:16 +0200 From: Virgil Stokes To: light1qu...@gmail.com On 2012-08-14 17:38, light1qu...@gmail.com wrote: Hi, I am migrating from PHP to Python and I am slightly confused. I am making a function that takes a startingList, finds all the strings in the list that begin with 'x', removes those strings and puts them into a xOnlyList. However if you run the code you will notice only one of the strings beginning with 'x' is removed from the startingList. If I comment out 'startingList.remove(str);' the code runs with both strings beginning with 'x' being put in the xOnlyList. Using the print statement I noticed that the second string that begins with 'x' isn't even identified by the function. Why does this happen? def testFunc(startingList): xOnlyList = []; for str in startingList: if (str[0] == 'x'): print str; xOnlyList.append(str) startingList.remove(str) #this seems to be the problem print xOnlyList; print startingList testFunc(['xasd', 'xjkl', 'sefwr', 'dfsews']) #Thanks for your help! You might find the following useful: def testFunc(startingList): xOnlyList = []; j = -1 for xl in startingList: if (xl[0] == 'x'): xOnlyList.append(xl) else: j += 1 startingList[j] = xl if j == -1: startingList = [] else: del startingList[j:-1] return(xOnlyList) testList1 = ['xasd', 'xjkl', 'sefwr', 'dfsews'] testList2 = ['xasd', 'xjkl', 'xsefwr', 'xdfsews'] testList3 = ['xasd', 'jkl', 'sefwr', 'dfsews'] testList4 = ['asd', 'jkl', 'sefwr', 'dfsews'] xOnlyList = testFunc(testList1) print 'xOnlyList = ',xOnlyList print 'testList = ',testList1 xOnlyList = testFunc(testList2) print 'xOnlyList = ',xOnlyList print 'testList = ',testList2 xOnlyList = testFunc(testList3) print 'xOnlyList = ',xOnlyList print 'testList = ',testList3 xOnlyList = testFunc(testList4) print 'xOnlyList = ',xOnlyList print 'testList = ',testList4 And here is another version using list comprehension that I prefer testList1 = ['xasd', 'xjkl', 'sefwr', 'dfsews'] testList2 = ['xasd', 'xjkl', 'xsefwr', 'xdfsews'] testList3 = ['xasd', 'jkl', 'sefwr', 'dfsews'] testList4 = ['asd', 'jkl', 'sefwr', 'dfsews'] def testFunc2(startingList): return([x for x in startingList if x[0] == 'x'], [x for x in startingList if x[0] != 'x']) xOnlyList,testList = testFunc2(testList1) print xOnlyList print testList xOnlyList,testList = testFunc2(testList2) print xOnlyList print testList xOnlyList,testList = testFunc2(testList3) print xOnlyList print testList xOnlyList,testList = testFunc2(testList4) print xOnlyList print testList -- http://mail.python.org/mailman/listinfo/python-list
Re: Strange behavior
On 15-Aug-2012 02:19, Steven D'Aprano wrote:
On Tue, 14 Aug 2012 21:40:10 +0200, Virgil Stokes wrote:
You might find the following useful:
def testFunc(startingList):
xOnlyList = []; j = -1
for xl in startingList:
if (xl[0] == 'x'):
That's going to fail in the starting list contains an empty string. Use
xl.startswith('x') instead.
Yes, but this was by design (tacitly assumed that startingList was both a list
and non-empty).
xOnlyList.append(xl)
else:
j += 1
startingList[j] = xl
Very cunning, but I have to say that your algorithm fails the "is this
obviously correct without needing to study it?" test. Sometimes that is
unavoidable, but for something like this, there are simpler ways to solve
the same problem.
Sorry, but I do not sure what you mean here.
if j == -1:
startingList = []
else:
del startingList[j:-1]
return(xOnlyList)
And here is another version using list comprehension that I prefer
def testFunc2(startingList):
return([x for x in startingList if x[0] == 'x'], [x for x in
startingList if x[0] != 'x'])
This walks over the starting list twice, doing essentially the same thing
both times. It also fails to meet the stated requirement that
startingList is modified in place, by returning a new list instead.
This can meet the requirement that startingList is modified in place via the
call to this function (see the attached code).
Here's an example of what I mean:
py> mylist = mylist2 = ['a', 'x', 'b', 'xx', 'cx'] # two names for one
list
py> result, mylist = testFunc2(mylist)
py> mylist
['a', 'b', 'cx']
py> mylist2 # should be same as mylist
['a', 'x', 'b', 'xx', 'cx']
Yes, I had a typo in my original posting --- sorry about that!
Here is the obvious algorithm for extracting and removing words starting
with 'x'. It walks the starting list only once, and modifies it in place.
The only trick needed is list slice assignment at the end.
def extract_x_words(words):
words_with_x = []
words_without_x = []
for word in words:
if word.startswith('x'):
words_with_x.append(word)
else:
words_without_x.append(word)
words[:] = words_without_x # slice assignment
return words_with_x
Suppose words was not a list --- you have tacitly assumed that words is a list.
The only downside of this is that if the list of words is so enormous
that you can fit it in memory *once* but not *twice*, this may fail. But
the same applies to the list comprehension solution.
But, this is not the only downside if speed is important --- it is slower than
the list comprehension method (see results that follows).
Here is a summary of three algorithms (algorithm-1, algorithm-2, algorithm-2A)
that I tested (see attached code). Note, algorithm-2A was obtained by removing
the slice assignment in the above code and modifying the return as follows
def extract_x_words(words):
words_with_x = []
words_without_x = []
for word in words:
if word.startswith('x'):
words_with_x.append(word)
else:
words_without_x.append(word)
#words[:] = words_without_x # slice assignment
return words_with_x, words_without_x
Of course, one needs to modify the call for "in-place" update of startingList as
follows:
xOnlyList,startingList = extract_x_words(startingList)
Here is a summary of my timing results obtained for 3 different algorithms for
lists with 100,000 strings of length 4 in each list:
Method
average (sd) time in seconds
algorithm-1 (list comprehension)
0.11630 (0.0014)
algorithm-2 (S. D'Aprano)
0.17594 (0.0014)
algorithm-2A (modified S. D'Aprano)
0.18217 (0.0023)
These values were obtained from 100 independent runs (MC simulations) on lists
that contain 100,000 strings. Approximately 50% of these strings contained a
leading 'x'. Note, that the results show that algorithm-2 (suggested by S.
D'Aprano) is approximately 51% slower than algorithm-1 (list comprehensions) and
algorithm-2A (simple modification of algorithm-2) is approximately 57% slower
than algorithm-1. Why is algorithm-2A slower than algorithm-2?
I would be interested in seeing code that is faster than algorithm-1 --- any
suggestions are welcomed. And of course, if there are any errors in my attached
code please inform me of them and I will try to correct them as soon as
possible. Note, some of the code is actually irrelevant for the original
"Strange behavior" post.
Have a good day!
'''
Purpose: Time three different algorithms for the same task
Author: V. Stoke
Re: Strange behavior
On 16-Aug-2012 15:02, Peter Otten wrote:
Virgil Stokes wrote:
def testFunc(startingList):
xOnlyList = []; j = -1
for xl in startingList:
if (xl[0] == 'x'):
That's going to fail in the starting list contains an empty string. Use
xl.startswith('x') instead.
Yes, but this was by design (tacitly assumed that startingList was both a
list and non-empty).
You missunderstood it will fail if the list contains an empty string, not if
the list itself is empty:
words = ["alpha", "", "xgamma"]
[word for word in words if word[0] == "x"]
Traceback (most recent call last):
File "", line 1, in
IndexError: string index out of range
The startswith() version:
[word for word in words if word.startswith("x")]
['xgamma']
Also possible:
[word for word in words if word[:1] == "x"]
['xgamma']
def testFunc1(startingList):
'''
Algorithm-1
Note:
One should check for an empty startingList before
calling testFunc1 -- If this possibility exists!
'''
return([x for x in startingList if x[0] == 'x'],
[x for x in startingList if x[0] != 'x'])
I would be interested in seeing code that is faster than algorithm-1
In pure Python? Perhaps the messy variant:
def test_func(words):
nox = []
append = nox.append
withx = [x for x in words if x[0] == 'x' or append(x)]
return withx, nox
Very nice Peter,
Here are the new results for timing with your method added (algorithm-3).
Method
average (sd) time in seconds
algorithm-1 (list comprehension)
0.11774 (0.002968)
algorithm-2 (S. D'Aprano)
0.17573 (0.003385)
algorithm-2A (modified S. D'Aprano)
0.18116 (0.003081)
algorithm-3 (improved list comprehension)
0.06639 (0.001728)
Algorithm-3 is 43% faster than algorithm-1. Again, the code used to obtain
these results is attached.
Thanks Peter for your contribution
'''
Purpose: Time four different algorithms for the same task
Author: V. Stokes (v...@it.uu.se, 2012-08-16 (15:46), 2012-08-16)
Refs:
python-list@python.org list
* Strange behavior, 14-Aug-2012 17:38, light1qu...@gmail.com
* Re: Strange behavior, 14-Aug-2012 21:40, Stokes, Virgil
* Re: Strange behavior, 15-Aug-2012 02:19, Steven D'Aprano
* Re: Strange behavior, 16-Aug-2012 15:02, Peter Otten
Notes:
1. The mean and standard deviation over the runs (MC simulations)
are estimated using recursive equations.
2. A seed (syd) is used with the RNG for repeatability. Each run is
started with a new seed to force the generation of independent
random sequences.
3. Warning! No checks are made on the parameters passed to
the functions (this was by design).
4. No effort has been made to make this code elegant. My focus was
to make the code clear and easy to understand.
5. This was executed on a Windows Vista 32-bit platform with Python 2.6.6
Processor: Intel(R) core(TM)2 Duo CPU E8500@3.16GHz 3.17GHz
6. The estimated time to completion is displayed after each run.
'''
import random as random
import math as math
from time import clock # clock gives good resolution on MS Windows
def testFunc1(startingList):
'''
Algorithm-1
Note:
One should check for an empty startingList before
calling testFunc1 -- If this possibility exists!
'''
return([x for x in startingList if x[0] == 'x'],
[x for x in startingList if x[0] != 'x'])
def testFunc2(words):
'''
Algorithm-2
'''
words_with_x = []
words_without_x = []
for word in words:
if word.startswith('x'):
words_with_x.append(word)
else:
words_without_x.append(word)
words[:] = words_without_x # slice assignment
return words_with_x
def testFunc2A(words):
'''
Algorithm-2A
'''
words_with_x = []
words_without_x = []
for word in words:
if word.startswith('x'):
words_with_x.append(word)
else:
words_without_x.append(word)
#words[:] = words_without_x # slice assignment
return words_with_x, words_without_x
def testFunc3(words):
'''
Algorithm-3 (from: Peter Otten)
'''
nox = []
append = nox.append
withx = [x for x in words if x[0] == 'x' or append(x)]
return withx, nox
def genStrList(NChar,NStrng,Alph,leadChr):
'''
Purpose: Generate a list of NStrng elements with each element a string
of length NChar and constrained such that approx. 50% of the
strings will b
Re: Strange behavior
On 16-Aug-2012 19:40, Steven D'Aprano wrote:
On Thu, 16 Aug 2012 13:18:59 +0200, Virgil Stokes wrote:
On 15-Aug-2012 02:19, Steven D'Aprano wrote:
On Tue, 14 Aug 2012 21:40:10 +0200, Virgil Stokes wrote:
You might find the following useful:
def testFunc(startingList):
xOnlyList = []; j = -1
for xl in startingList:
if (xl[0] == 'x'):
That's going to fail in the starting list contains an empty string. Use
xl.startswith('x') instead.
Yes, but this was by design (tacitly assumed that startingList was both
a list and non-empty).
As Peter already pointed out, I said it would fail if the list contains
an empty string, not if the list was empty.
xOnlyList.append(xl)
else:
j += 1
startingList[j] = xl
Very cunning, but I have to say that your algorithm fails the "is this
obviously correct without needing to study it?" test. Sometimes that is
unavoidable, but for something like this, there are simpler ways to
solve the same problem.
Sorry, but I do not sure what you mean here.
In a perfect world, you should be able to look at a piece of code, read
it once, and see whether or not it is correct. That is what I mean by
"obviously correct". For example, if I have a function that takes an
argument, doubles it, and prints the result:
def f1(x):
print(2*x)
that is obviously correct. Whereas this is not:
def f2(x):
y = (x + 5)**2 - (x + 4)**2
sys.stdout.write(str(y - 9) + '\n')
because you have to study it to see whether or not it works correctly.
Not all programs are simple enough to be obviously correct. Sometimes you
have no choice but to write something which requires cleverness to get
the right result. But this is not one of those cases. You should almost
always prefer simple code over clever code, because the greatest expense
in programming (time, effort and money) is to make code correct.
Most code does not need to be fast. But all code needs to be correct.
[...]
This can meet the requirement that startingList is modified in place via
the call to this function (see the attached code).
Good grief! See, that's exactly the sort of thing I'm talking about.
Without *detailed* study of your attached code, how can I possibly know
what it does or whether it does it correctly?
Very strange question? Perhaps, you should work on understanding code that you
have not written, or maybe you should learn more about Python, or I really
don't know how to help you with this question.
Your timing code calculates the mean using a recursive algorithm. Why
don't you calculate the mean the standard way: add the numbers and divide
by the total? What benefit do you gain from a more complicated algorithm
when a simple one will do the job just as well?
A lot of questions that suggest you have not made much of an effort to answer
them yourself. Try a little numerical analysis/research before asking such
questions (This is how you often respond to others on this list who would like
help --- try apply your advice to other to yourself. I will give you a start:
* Knuth, D. E. (1998) /The Art of Computer Programming vol. 2: Seminumerical
Algorithms/ /(3rd edition)/. Addison-Wesley, Boston.
[hint: study p. 232]
* Welford, B. P. (1962) Note on a method for calculating sums of squares and
products. T/echnometrics/ *4*(3).
[hint: pp. 419-420]
You have spent a lot of effort creating a complicated, non-obvious piece
of timing code, with different random seeds for each run, and complicated
ways of calculating timing statistics... but unfortunately the most
important part of any timing test, the actually *timing*, is not done
correctly. Consequently, your code is not correct.
How do you know how much effort I used? Code "non-obvious" and "complicated" for
you does not mean that this is also true for others. Could you please be more
specific --- saying code is not correct without providing details is not very
useful. I did say in an earlier email in reference to my code "if there are any
errors in my attached code please inform me of them and I will try to correct
them as soon as possible".
With an average time of a fraction of a second, none of those timing
results are trustworthy, because they are vulnerable to interference from
other processes, the operating system, and other random noise.
Please explain what you mean by the timing results not being trustworthy and how
this vulnerability works --- in detail please.
You spend
a lot of time processing the timing results, but it is Garbage In,
Garbage Out -- the results are not trustworthy, and if they are correct,
it is only by accident.
Fantastic --- a lot of criticism but little that can be helpful. What
specifically is the "Garbage In"?
Later in your post, you run some tests, and are surprised by the result:
Why is
Re: Books?
On 22-Aug-2012 16:04, Steven D'Aprano wrote: On Tue, 21 Aug 2012 18:36:50 -0700, Anonymous Group wrote: What books do you recomend for learning python? Preferably free and/or online. Completely by coincidence, I have just discovered, and I mean *literally* just a few minutes ago, this book: http://www.springer.com/mathematics/computational+science+%26+engineering/book/978-3-642-30292-3 http://codingcat.com/knjige/python/A%20Primer%20on%20Scientific%20Programming%20with%20Python.pdf I wish it had existed when I was a beginner! I haven't read the whole thing, but dammit it looks like exactly the sort of book I would have adored as a newbie. (Your mileage may vary.) I second this --- this is a very good book IMHO. I have the first edition (2009) and have found it very useful. Good tip! -- http://mail.python.org/mailman/listinfo/python-list
Installation of yappi (timing module)
I have been doing some experiments with different modules for the timing of functions and code segments. One module I would like to test is yappi (thread aware timer) which is listed at PyPI. However, I have been unable to install it on Windows Vista and Windows 7 (Python 2.7 on both). I have tried both easy_install and pip (as suggested at http://code.google.com/p/yappi/). Here is what happens with easy_install C:\Users\Virgil>easy_install yappi Searching for yappi Reading http://pypi.python.org/simple/yappi/ Reading http://yappi.googlecode.com/ Best match: yappi 0.62 Downloading http://yappi.googlecode.com//files/yappi-0.62.tar.gz Processing yappi-0.62.tar.gz Writing c:\users\virgil\appdata\local\temp\easy_install-tzt5gl\yappi-0.62\setup.cfg Running yappi-0.62\setup.py -q bdist_egg --dist-dir c:\users\virgil\appdata\local\temp\easy_install-tzt5gl\yappi-0.62\egg-dist-tmp-t3qodo In file included from D:\python27\include\Python.h:8, from config.h:4, from _yappi.c:10: D:\python27\include\pyconfig.h:68: io.h: No such file or directory D:\python27\include\pyconfig.h:296: stdio.h: No such file or directory In file included from config.h:4, from _yappi.c:10: D:\python27\include\Python.h:19: limits.h: No such file or directory D:\python27\include\Python.h:22: #error "Something's broken. UCHAR_MAX should be defined in limits.h." D:\python27\include\Python.h:26: #error "Python's source code assumes C's unsigned char is an 8-bit type." D:\python27\include\Python.h:33: stdio.h: No such file or directory D:\python27\include\Python.h:35: #error "Python.h requires that stdio.h define NULL." D:\python27\include\Python.h:38: string.h: No such file or directory D:\python27\include\Python.h:40: errno.h: No such file or directory D:\python27\include\Python.h:42: stdlib.h: No such file or directory D:\python27\include\Python.h:49: stddef.h: No such file or directory D:\python27\include\Python.h:56: assert.h: No such file or directory In file included from D:\python27\include\Python.h:58, from config.h:4, from _yappi.c:10: D:\python27\include\pyport.h:306: stdlib.h: No such file or directory D:\python27\include\pyport.h:312: math.h: No such file or directory D:\python27\include\pyport.h:325: time.h: No such file or directory D:\python27\include\pyport.h:377: sys\stat.h: No such file or directory In file included from D:\python27\include\Python.h:85, from config.h:4, from _yappi.c:10: D:\python27\include\unicodeobject.h:4: stdarg.h: No such file or directory D:\python27\include\unicodeobject.h:57: ctype.h: No such file or directory D:\python27\include\unicodeobject.h:120: wchar.h: No such file or directory In file included from D:\python27\include\Python.h:94, from config.h:4, from _yappi.c:10: D:\python27\include\stringobject.h:10: stdarg.h: No such file or directory In file included from D:\python27\include\Python.h:98, from config.h:4, from _yappi.c:10: D:\python27\include\bytearrayobject.h:9: stdarg.h: No such file or directory In file included from D:\python27\include\Python.h:121, from config.h:4, from _yappi.c:10: D:\python27\include\pyerrors.h:319: stdarg.h: No such file or directory In file included from D:\python27\include\Python.h:126, from config.h:4, from _yappi.c:10: D:\python27\include\modsupport.h:10: stdarg.h: No such file or directory In file included from _yappi.c:10: config.h:15: stdint.h: No such file or directory In file included from _yappi.c:23: timing.h:8: windows.h: No such file or directory error: Setup script exited with error: command 'gcc' failed with exit status 1 And pip fails with similar problems (same pyconfig errors where C++ header files are not found). In both cases yappi-0.62.tar.gz was downloaded.Note: 1) I also tried to install from the source which also failed with similar problems, 2) I have both cygwin and MinGW gcc compilers on my systems and they do contain in their include folder these "missing" header files. Any suggestions on how yappi can be installed would be appreciated. --V :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Installation of yappi (timing module)
On 24-Aug-2012 12:28, Virgil Stokes wrote: I have been doing some experiments with different modules for the timing of functions and code segments. One module I would like to test is yappi (thread aware timer) which is listed at PyPI. However, I have been unable to install it on Windows Vista and Windows 7 (Python 2.7 on both). I have tried both easy_install and pip (as suggested at http://code.google.com/p/yappi/). Here is what happens with easy_install C:\Users\Virgil>easy_install yappi Searching for yappi Reading http://pypi.python.org/simple/yappi/ Reading http://yappi.googlecode.com/ Best match: yappi 0.62 Downloading http://yappi.googlecode.com//files/yappi-0.62.tar.gz Processing yappi-0.62.tar.gz Writing c:\users\virgil\appdata\local\temp\easy_install-tzt5gl\yappi-0.62\setup.cfg Running yappi-0.62\setup.py -q bdist_egg --dist-dir c:\users\virgil\appdata\local\temp\easy_install-tzt5gl\yappi-0.62\egg-dist-tmp-t3qodo In file included from D:\python27\include\Python.h:8, from config.h:4, from _yappi.c:10: D:\python27\include\pyconfig.h:68: io.h: No such file or directory D:\python27\include\pyconfig.h:296: stdio.h: No such file or directory In file included from config.h:4, from _yappi.c:10: D:\python27\include\Python.h:19: limits.h: No such file or directory D:\python27\include\Python.h:22: #error "Something's broken. UCHAR_MAX should be defined in limits.h." D:\python27\include\Python.h:26: #error "Python's source code assumes C's unsigned char is an 8-bit type." D:\python27\include\Python.h:33: stdio.h: No such file or directory D:\python27\include\Python.h:35: #error "Python.h requires that stdio.h define NULL." D:\python27\include\Python.h:38: string.h: No such file or directory D:\python27\include\Python.h:40: errno.h: No such file or directory D:\python27\include\Python.h:42: stdlib.h: No such file or directory D:\python27\include\Python.h:49: stddef.h: No such file or directory D:\python27\include\Python.h:56: assert.h: No such file or directory In file included from D:\python27\include\Python.h:58, from config.h:4, from _yappi.c:10: D:\python27\include\pyport.h:306: stdlib.h: No such file or directory D:\python27\include\pyport.h:312: math.h: No such file or directory D:\python27\include\pyport.h:325: time.h: No such file or directory D:\python27\include\pyport.h:377: sys\stat.h: No such file or directory In file included from D:\python27\include\Python.h:85, from config.h:4, from _yappi.c:10: D:\python27\include\unicodeobject.h:4: stdarg.h: No such file or directory D:\python27\include\unicodeobject.h:57: ctype.h: No such file or directory D:\python27\include\unicodeobject.h:120: wchar.h: No such file or directory In file included from D:\python27\include\Python.h:94, from config.h:4, from _yappi.c:10: D:\python27\include\stringobject.h:10: stdarg.h: No such file or directory In file included from D:\python27\include\Python.h:98, from config.h:4, from _yappi.c:10: D:\python27\include\bytearrayobject.h:9: stdarg.h: No such file or directory In file included from D:\python27\include\Python.h:121, from config.h:4, from _yappi.c:10: D:\python27\include\pyerrors.h:319: stdarg.h: No such file or directory In file included from D:\python27\include\Python.h:126, from config.h:4, from _yappi.c:10: D:\python27\include\modsupport.h:10: stdarg.h: No such file or directory In file included from _yappi.c:10: config.h:15: stdint.h: No such file or directory In file included from _yappi.c:23: timing.h:8: windows.h: No such file or directory error: Setup script exited with error: command 'gcc' failed with exit status 1 And pip fails with similar problems (same pyconfig errors where C++ header files are not found). In both cases yappi-0.62.tar.gz was downloaded.Note: 1) I also tried to install from the source which also failed with similar problems, 2) I have both cygwin and MinGW gcc compilers on my systems and they do contain in their include folder these "missing" header files. Any suggestions on how yappi can be installed would be appreciated. --V :-) Problem solved! The ordering of the gcc compilers in my PATH statement caused this failure. I have ordered these compilers such that the first one contains the required header files and the installation of yappi is now successful. --V -- http://mail.python.org/mailman/listinfo/python-list
Re: OT Questions
On 17-Oct-2012 11:31, Chris Angelico wrote: On Wed, Oct 17, 2012 at 5:27 PM, Dwight Hutto wrote: On Wed, Oct 17, 2012 at 2:06 AM, Demian Brecht wrote: I can't ascertain what your strengths are as I don't work with you on a daily basis (one of the many benefits of working with people smarter than you ;)). Doubt that, unless they have 160+ I.Q.'s(been seeing psychiatrists since I was 13). I'm very secure in my childlike intellectualism. A high IQ just proves ability to score well on IQ tests. On the whole, your statement strikes me as reminiscent of Sheldon Cooper's insistence that "I'm not crazy, my mother had me tested!". Personally, I've never taken an IQ test, so I don't know how well I'd score. But I'm a school dropout, never went to college/uni/TAFE/etc/etc, don't have any certifications of any sort. I'm a pretty uneducated fella, according to my résum&htmlentitiesdontworkhere; (that's "resume" when folded into ASCII). So according to how most people think about intelligence, I probably have a sub-par IQ. On the flip side, I'm a professional programmer, I run a server where people play Dungeons and Dragons, and I'm a well-respected wordsmith as Dungeon Master. Plus, I work in theatre (in fact, at the moment I'm posting from the bio box, sitting next to the follow spot that I'll be operating for the next two weeks). So I think I have enough muscle upstairs to get through life... But Dwight (and I'll continue to address you as such until you change your mail headers), a LOT of what you're saying is coming across as over-inflated ego. Maybe you are a majorly interdisciplinary learner; but boasting that you're "the most interdisciplinary learner [we] might have ever encountered" just comes across poorly. One thing I've learned from various groups is that, no matter how X you are, there's someone else who's even more X - for any X. Maybe it isn't true somewhere, maybe you really are the peak - but more than likely you aren't, and it's much more pleasant to be proved better than your claim than to be proved worse. (There are exceptions, of course. I have absolutely no doubt that I am the person most familiar with the RosMud++ code and thus the person best positioned to maintain that project. This is because I wrote it. But I am not claiming to be the best C++ programmer in the world, because there are a lot of other C++ programmers among the seven billion here.) ChrisA An excellent response Chris :-) -- http://mail.python.org/mailman/listinfo/python-list
Fast forward-backward (write-read)
I am working with some rather large data files (>100GB) that contain time series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform various types of processing on these data (e.g. moving median, moving average, and Kalman-filter, Kalman-smoother) in a sequential manner and only a small number of these data need be stored in RAM when being processed. When performing Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these variables saved to an external file from the forward pass, in reverse order --- from last written to first written. Finally, to my question --- What is a fast way to write these variables to an external file and then read them in backwards? -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 23-Oct-2012 18:09, Tim Chase wrote: On 10/23/12 09:31, Virgil Stokes wrote: I am working with some rather large data files (>100GB) that contain time series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform various types of processing on these data (e.g. moving median, moving average, and Kalman-filter, Kalman-smoother) in a sequential manner and only a small number of these data need be stored in RAM when being processed. When performing Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these variables saved to an external file from the forward pass, in reverse order --- from last written to first written. Finally, to my question --- What is a fast way to write these variables to an external file and then read them in backwards? Am I missing something, or would the fairly-standard "tac" utility do the reversal you want? It should[*] be optimized to handle on-disk files in a smart manner. Not sure about "tac" --- could you provide more details on this and/or a simple example of how it could be used for fast reversed "reading" of a data file? Otherwise, if you can pad the record-lengths so they're all the same, and you know the total number of records, you can seek to Total-(RecSize*OneBasedOffset) and write the record,optionally padding if you need/can. At least on *nix-like OSes, you can seek into a sparse-file with no problems (untested on Win32). The records lengths will all be the same and yes seek could be used; but, I was hoping for a faster method. Thanks Tim! :-) -tkc [*] Just guessing here. Would be disappointed if it *wasn't*. -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 23-Oct-2012 18:17, Paul Rubin wrote: Virgil Stokes writes: Finally, to my question --- What is a fast way to write these variables to an external file and then read them in backwards? Seeking backwards in files works, but the performance hit is significant. There is also a performance hit to scanning pointers backwards in memory, due to cache misprediction. If it's something you're just running a few times, seeking backwards the simplest approach. If you're really trying to optimize the thing, you might buffer up large chunks (like 1 MB) before writing. If you're writing once and reading multiple times, you might reverse the order of records within the chunks during the writing phase. I am writing (forward) once and reading (backward) once. You're of course taking a performance bath from writing the program in Python to begin with (unless using scipy/numpy or the like), enough that it might dominate any effects of how the files are written. I am currently using SciPy/NumPy Of course (it should go without saying) that you want to dump in a binary format rather than converting to decimal. Yes, I am doing this (but thanks for "underlining" it!) Thanks Paul :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 23-Oct-2012 18:35, Dennis Lee Bieber wrote: On Tue, 23 Oct 2012 16:31:17 +0200, Virgil Stokes declaimed the following in gmane.comp.python.general: Finally, to my question --- What is a fast way to write these variables to an external file and then read them in backwards? Stuff them into an SQLite3 database and retrieve using a descending sort? Have never worked with a database; but, could be worth a try (at least to compare I/O times). Thanks Dennis :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 23-Oct-2012 19:56, Tim Chase wrote: On 10/23/12 12:17, Virgil Stokes wrote: On 23-Oct-2012 18:09, Tim Chase wrote: Finally, to my question --- What is a fast way to write these variables to an external file and then read them in backwards? Am I missing something, or would the fairly-standard "tac" utility do the reversal you want? It should[*] be optimized to handle on-disk files in a smart manner. Not sure about "tac" --- could you provide more details on this and/or a simple example of how it could be used for fast reversed "reading" of a data file? Well, if you're reading input.txt (and assuming it's one record per line, separated by newlines), you can just use tac < input.txt > backwards.txt which will create a secondary file that is the first file in reverse order. Your program can then process this secondary file in-order (which would be backwards from your source). I might have misunderstood your difficulty, but it _sounded_ like you just want to inverse the order of a file. Yes, I do wish to inverse the order, but the "forward in time" file will be in binary. --V -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 24-Oct-2012 00:36, David Hutto wrote:
Don't forget to use timeit for an average OS utilization.
I'd suggest two list comprehensions for now, until I've reviewed it some more:
forward = ["%i = %s" % (i,chr(i)) for i in range(33,126)]
backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]
for var in forward:
print var
for var in backward:
print var
You could also use a dict, and iterate through a straight loop that
assigned a front and back to a dict_one = {0 : [0.100], 1 : [1.99]}
and the iterate through the loop, and call the first or second in the
dict's var list for frontwards , or backwards calls.
But there might be faster implementations, depending on other
function's usage of certain lower level functions.
Missed the part about it being a file. Use:
forward = ["%i = %s" % (i,chr(i)) for i in range(33,126)]
backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]
print forward,backward
Interesting approach for small data sets (or blocks from a much larger data
set).
Thanks David :-)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 24-Oct-2012 02:06, Oscar Benjamin wrote:
On 23 October 2012 15:31, Virgil Stokes wrote:
I am working with some rather large data files (>100GB) that contain time
series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
format. I perform various types of processing on these data (e.g. moving
median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
manner and only a small number of these data need be stored in RAM when
being processed. When performing Kalman-filtering (forward in time pass, k =
0,1,...,N) I need to save to an external file several variables (e.g. 11*32
bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
(backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
variables saved to an external file from the forward pass, in reverse order
--- from last written to first written.
Finally, to my question --- What is a fast way to write these variables to
an external file and then read them in backwards?
You mentioned elsewhere that you are using numpy. I'll assume that the
data you want to read/write are numpy arrays.
Numpy arrays can be written very efficiently in binary form using
tofile/fromfile:
import numpy
a = numpy.array([1, 2, 5], numpy.int64)
a
array([1, 2, 5])
with open('data.bin', 'wb') as f:
... a.tofile(f)
...
You can then reload the array with:
with open('data.bin', 'rb') as f:
... a2 = numpy.fromfile(f, numpy.int64)
...
a2
array([1, 2, 5])
Numpy arrays can be reversed before writing or after reading using;
a2
array([1, 2, 5])
a2[::-1]
array([5, 2, 1])
Assuming you wrote the file forwards you can make an iterator to yield
the file in chunks backwards like so (untested):
def read_backwards(f, dtype, chunksize=1024 ** 2):
dtype = numpy.dtype(dtype)
nbytes = chunksize * dtype.itemsize
f.seek(0, 2)
fpos = f.tell()
while fpos > nbytes:
f.seek(fpos, 0)
yield numpy.fromfile(f, dtype, chunksize)[::-1]
fpos -= nbytes
yield numpy.fromfile(f, dtype)[::-1]
Oscar
Ok Oscar,
Thanks for the tip and I will look into this more.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 24-Oct-2012 00:57, Demian Brecht wrote: This is a classic example of why the old external processing algorithms of the 1960s and 70s will never be obsolete. No matter how much memory you have, there will always be times when you want to process more data than you can fit into memory. But surely nobody will *ever* need more than 640k… Right? Demian Brecht @demianbrecht http://demianbrecht.github.com Yes, I can still remember such quotes --- thanks for jogging my memory, Demian :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 24-Oct-2012 00:53, Steven D'Aprano wrote: On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote: On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes wrote: I am working with some rather large data files (>100GB) [...] Finally, to my question --- What is a fast way to write these variables to an external file and then read them in backwards? Don't forget to use timeit for an average OS utilization. Given that the data files are larger than 100 gigabytes, the time required to process each file is likely to be in hours, not microseconds. That being the case, timeit is the wrong tool for the job, it is optimized for timings tiny code snippets. You could use it, of course, but the added inconvenience doesn't gain you any added accuracy. Here's a neat context manager that makes timing long-running code simple: http://code.activestate.com/recipes/577896 Thanks for this link I'd suggest two list comprehensions for now, until I've reviewed it some more: I would be very surprised if the poster will be able to fit 100 gigabytes of data into even a single list comprehension, let alone two. You are correct and I have been looking at working with blocks that are sized to the RAM available for processing. This is a classic example of why the old external processing algorithms of the 1960s and 70s will never be obsolete. No matter how much memory you have, there will always be times when you want to process more data than you can fit into memory. Thanks for your insights :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 23-Oct-2012 22:03, Cousin Stanley wrote: Virgil Stokes wrote: Not sure about "tac" --- could you provide more details on this and/or a simple example of how it could be used for fast reversed "reading" of a data file ? tac is available as a command under linux $ whatis tac tac (1) - concatenate and print files in reverse $ whereis tac tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz $ man tac SYNOPSIS tac [OPTION]... [FILE]... DESCRIPTION Write each FILE to standard output, last line first. With no FILE, or when FILE is -, read standard input. I only know that the tac command exists but have never used it myself Unfortunately, I may be forced to process the data on a Windows platform; but, thanks Cousin for the Linux tip. -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 24-Oct-2012 17:11, rusi wrote: On Oct 23, 7:52 pm, Virgil Stokes wrote: I am working with some rather large data files (>100GB) that contain time series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform various types of processing on these data (e.g. moving median, moving average, and Kalman-filter, Kalman-smoother) in a sequential manner and only a small number of these data need be stored in RAM when being processed. When performing Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these variables saved to an external file from the forward pass, in reverse order --- from last written to first written. Finally, to my question --- What is a fast way to write these variables to an external file and then read them in backwards? Have you tried gdbm/bsddbm? They are meant for such (I believe). Probably needs to be installed for windows; works for linux. If I were you I'd try out with the giant data on linux and see if the problem is solved, then see how to install for windows Thanks Rusi :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 24-Oct-2012 01:46, Paul Rubin wrote: Virgil Stokes writes: Yes, I do wish to inverse the order, but the "forward in time" file will be in binary. I really think it will be simplest to just write the file in forward order, then use mmap to read it one record at a time. It might be possible to squeeze out a little more performance with reordering tricks but that's the first thing to try. Thanks Paul, I am working on this approach now... -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 28-Oct-2012 12:18, Dave Angel wrote: On 10/24/2012 03:14 AM, Virgil Stokes wrote: On 24-Oct-2012 01:46, Paul Rubin wrote: Virgil Stokes writes: Yes, I do wish to inverse the order, but the "forward in time" file will be in binary. I really think it will be simplest to just write the file in forward order, then use mmap to read it one record at a time. It might be possible to squeeze out a little more performance with reordering tricks but that's the first thing to try. Thanks Paul, I am working on this approach now... If you're using mmap to map the whole file, you'll need 64bit Windows to start with. I'd be interested to know if Windows will allow you to mmap 100gb at one stroke. Have you tried it, or are you starting by figuring how to access the data from the mmap? Thanks very much for pursuing my query, Dave. I have not tried it yet --- temporarily side-tracked; but, I will post my findings on this issue. -- http://mail.python.org/mailman/listinfo/python-list
Re: Fast forward-backward (write-read)
On 2012-10-28 19:21, Oscar Benjamin wrote:
On 28 October 2012 14:20, Virgil Stokes wrote:
On 28-Oct-2012 12:18, Dave Angel wrote:
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes writes:
Yes, I do wish to inverse the order, but the "forward in time" file
will be in binary.
I really think it will be simplest to just write the file in forward
order, then use mmap to read it one record at a time. It might be
possible to squeeze out a little more performance with reordering tricks
but that's the first thing to try.
Thanks Paul,
I am working on this approach now...
If you're using mmap to map the whole file, you'll need 64bit Windows to
start with. I'd be interested to know if Windows will allow you to mmap
100gb at one stroke. Have you tried it, or are you starting by figuring
how to access the data from the mmap?
Thanks very much for pursuing my query, Dave.
I have not tried it yet --- temporarily side-tracked; but, I will post my
findings on this issue.
If you are going to use mmap then look at the numpy.memmap function.
This wraps pythons mmap so that you can access the contents of the
mapped binary file as if it was a numpy array. This means that you
don't need to handle the bytes -> float conversions yourself.
import numpy
a = numpy.array([4,5,6], numpy.float64)
a
array([ 4., 5., 6.])
with open('tmp.bin', 'wb') as f: # write forwards
... a.tofile(f)
... a.tofile(f)
...
a2 = numpy.memmap('tmp.bin', numpy.float64)
a2
memmap([ 4., 5., 6., 4., 5., 6.])
a2[3]
4.0
a2[5:2:-1] # read backwards
memmap([ 6., 5., 4.])
Oscar
Thanks Oscar!
--
http://mail.python.org/mailman/listinfo/python-list
Re: Obnoxious postings from Google Groups
On 04-Nov-2012 12:13, Jamie Paul Griffin wrote: / ru...@yahoo.com wrote on Fri 2.Nov'12 at 11:39:10 -0700 / (I also hope I haven't just been suckered by a troll attempt, windows/unix is better then unix/windows being an age-old means of trolling.) No, i'm not a "troll". I was just adding my opinion to the thread, I assumed that was allowed. I didn't say UNIX is better than Windows, did I; I just feel that Windows is not -- for me anyway -- the most suitable plaform for learning about the science of computing and coding, etc... being a computer science student that's the view i have and share with those I learn with and from. Why must people be accused of trolling everytime they make a statement that conveys a preference over one platform or language, for example, than the other. Provoking someone by labeling them a troll or implying they might be is a bit childish really. Well stated Jamie --- I agree. I don't believe that all members of this list label you as a troll. --V -- http://mail.python.org/mailman/listinfo/python-list
Re: Beginner Tutorials
On 18-Jan-2013 15:47, Rik wrote: Hi, I've developed a website for beginners to Python. I'd appreciate any comments or criticism. It's still under development, and should be finished in the next few months. Oh, and it's free to use. www.usingpython.com You have done well Rik. I like your approach to passwords for solutions and your selection of topics is quite good for a "jump start" with Python. However, I suggest that in your menu you change several of your items to lower case (for consistency with the Python language): For -> for, While -> while, if-Else -> if-else, Elif -> elif. I am curious --- what software did you use to create your nice web pages for this tutorial? In summary --- good work Rik :-) --V -- http://mail.python.org/mailman/listinfo/python-list
Re: Formatting a column's value output
On 27-Jan-2013 17:12, ru...@yahoo.com wrote: On 01/27/2013 02:04 AM, Ferrous Cranus wrote: [...] data = cur.fetchall() for row in data: print ( "" ) for item in row: print( ''' %s ''' % (item, item) ) [...] Okey, so far BUT i want the url linking to happen only for the URL column's value, and not for the hits column too. How do i apply the url link to the URL column's value only? Ferrous, 'row' has two items (the url and the hit count) in it, right? So print each separately rather than in a loop: data = cur.fetchall() for row in data: url = row[0] hits = row[1] print ( "" ) print( " %s : % (url, hits) ) It is nice to see some constructive feedback to Ferrous from the python-list. This will hopefully help to get Ferrous on the right track. -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading data from 2 different files and writing to a single file
On 28-Jan-2013 15:49, Chris Angelico wrote: On Tue, Jan 29, 2013 at 1:37 AM, Dave Angel wrote: What you want is the zip() function for l,s in zip(f1, f2): #you now have one line from each file, # which you can then validate and process Note, this assumes that when a line is "bad" from either file, you're going to also ignore the corresponding line from the other. If you have to accommodate variable misses in the lining up, then your work is *much* harder. Much harder? Not really - see my solution above with a simple 'break'. Much less clear what's going on, though, it is. Iterating together over both files with zip is much cleaner. ChrisA Nice example of the power of zip, Chris :-) -- http://mail.python.org/mailman/listinfo/python-list
Fast file data retrieval?
I have a rather large ASCII file that is structured as follows header line 9 nonblank lines with alphanumeric data header line 9 nonblank lines with alphanumeric data ... ... ... header line 9 nonblank lines with alphanumeric data EOF where, a data set contains 10 lines (header + 9 nonblank) and there can be several thousand data sets in a single file. In addition,*each header has a* *unique ID code*. Is there a fast method for the retrieval of a data set from this large file given its ID code? -- http://mail.python.org/mailman/listinfo/python-list
PyOpenCV -- help?
I have tried to install PyOpenCV without success (error occurs during the installation procedure). I reported the problem to the opencv user group (http://groups.google.com/group/ctypes-opencv) but this group has not been active since June of last year. Anyone know of how to get help with PyOpenCV? -- http://mail.python.org/mailman/listinfo/python-list
How to schedule execution of code?
Suppose that I have some Python code (vers. 2.6) that has been converted into an *.exe file and can be executed on a Windows (Vista or 7) platform. What can one do to have this *.exe executed at a set of specific times each day? In addition, if a day is missed (e.g. computer on which it resides goes down), then it will be executed the next time the computer is successfully started up. It might be useful to know that the purpose of this code is to collect data from a set of RSS feeds. Any suggestions would be appreciated. -- http://mail.python.org/mailman/listinfo/python-list
Processing a key pressed in Python 3.6
I would appreciate help on finding a solution to following problem. This
is the general structure of the "problem" code:
while True
# Get some data from the web and process it
...
...
# Write these data to a file
...
...
# Check if a_key has been pressed in the command window
...
...
if a_key_pressed:
# Perform some pre-termination tasks
...
...
# Close the output data file
...
...
raise SystemExit('Exit')
I am running the code with Python 3.6 on a windows 10 platform. I have
tried many approaches (mainly those posted on stackoverflow) but I have
yet to find an approach that works for this structure.
Note:
1) The code is executed in the windows 10 command window
2) I am not using wxPython, IDLE, or pyGame in this application.
3) The time to get the data, process it and write it to a file can
take from 0.5 sec to 1.5 sec
4) The key hit need not be echoed to the command window
--
https://mail.python.org/mailman/listinfo/python-list
Re: Processing a key pressed in Python 3.6 🦉
Thanks very much Chris,
This code worked perfectly for "Enter". Your knowledge of Python and
more specifically this elegant solution are greatly appreciated. I now
know that I need to learn more about threads. :-)
On 2018-01-23 20:15, Chris Angelico wrote:
On Wed, Jan 24, 2018 at 5:50 AM, Virgil Stokes wrote:
I would appreciate help on finding a solution to following problem. This is
the general structure of the "problem" code:
while True
# Get some data from the web and process it
...
...
# Write these data to a file
...
...
# Check if a_key has been pressed in the command window
...
...
if a_key_pressed:
# Perform some pre-termination tasks
...
...
# Close the output data file
...
...
raise SystemExit('Exit')
I am running the code with Python 3.6 on a windows 10 platform. I have tried
many approaches (mainly those posted on stackoverflow) but I have yet to
find an approach that works for this structure.
Note:
1) The code is executed in the windows 10 command window
2) I am not using wxPython, IDLE, or pyGame in this application.
3) The time to get the data, process it and write it to a file can
take from 0.5 sec to 1.5 sec
4) The key hit need not be echoed to the command window
Are you okay with demanding a specific key, rather than simply "press
any key"? Even better, key combination? Handle Ctrl-C by catching
KeyboardInterrupt and you can take advantage of Python's existing
cross-platform handling of the standard interrupt signal.
If Ctrl-C won't work for you, how about stipulating that it be Enter?
"Press Enter to quit" isn't too much worse than "Press any key to
quit" (plus you have less chance of accidentally terminating the
program when you don't want to). Spin off a thread to wait for enter.
I've tested this only on Linux, but it ought to work:
import threading
import time
shutdown = False
def wait_for_enter():
print("Hit Enter to quit.")
input()
global shutdown; shutdown = True
threading.Thread(target=wait_for_enter).start()
while "more work to do":
print("Getting data...")
time.sleep(1)
print("Saving data to file...")
time.sleep(1)
if shutdown:
print("Pre-termination...")
time.sleep(1)
raise SystemExit("exit")
If it doesn't, try switching around which is the secondary thread and
which is the primary - spin off a thread to do the work, then call
input() in the main thread.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Processing a key pressed in Python 3.6
Another follow-up question:
How would this code be modified to handle using the "Esc" key instead of
the "Enter" key?
On 2018-01-23 20:15, Chris Angelico wrote:
On Wed, Jan 24, 2018 at 5:50 AM, Virgil Stokes wrote:
I would appreciate help on finding a solution to following problem. This is
the general structure of the "problem" code:
while True
# Get some data from the web and process it
...
...
# Write these data to a file
...
...
# Check if a_key has been pressed in the command window
...
...
if a_key_pressed:
# Perform some pre-termination tasks
...
...
# Close the output data file
...
...
raise SystemExit('Exit')
I am running the code with Python 3.6 on a windows 10 platform. I have tried
many approaches (mainly those posted on stackoverflow) but I have yet to
find an approach that works for this structure.
Note:
1) The code is executed in the windows 10 command window
2) I am not using wxPython, IDLE, or pyGame in this application.
3) The time to get the data, process it and write it to a file can
take from 0.5 sec to 1.5 sec
4) The key hit need not be echoed to the command window
Are you okay with demanding a specific key, rather than simply "press
any key"? Even better, key combination? Handle Ctrl-C by catching
KeyboardInterrupt and you can take advantage of Python's existing
cross-platform handling of the standard interrupt signal.
If Ctrl-C won't work for you, how about stipulating that it be Enter?
"Press Enter to quit" isn't too much worse than "Press any key to
quit" (plus you have less chance of accidentally terminating the
program when you don't want to). Spin off a thread to wait for enter.
I've tested this only on Linux, but it ought to work:
import threading
import time
shutdown = False
def wait_for_enter():
print("Hit Enter to quit.")
input()
global shutdown; shutdown = True
threading.Thread(target=wait_for_enter).start()
while "more work to do":
print("Getting data...")
time.sleep(1)
print("Saving data to file...")
time.sleep(1)
if shutdown:
print("Pre-termination...")
time.sleep(1)
raise SystemExit("exit")
If it doesn't, try switching around which is the secondary thread and
which is the primary - spin off a thread to do the work, then call
input() in the main thread.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Processing a key pressed in Python 3.6
Ok Dennis,
You were correct. The following also works in the Windows 10 command window.
import time
import msvcrt
while "more work to do":
print("Getting data...")
time.sleep(1)
print("Saving data to file...")
time.sleep(1)
key = msvcrt.getwch()
#print('key: %s'%key) # just to show the result
if key == chr(27):
print("Pre-termination...")
time.sleep(1)
raise SystemExit("exit")
Note, I am using the "Esc" key to exit, which is one answer to my last
posting on this topic.
On 2018-01-23 20:37, Dennis Lee Bieber wrote:
On Tue, 23 Jan 2018 19:50:57 +0100, Virgil Stokes declaimed
the following:
I am running the code with Python 3.6 on a windows 10 platform. I have
tried many approaches (mainly those posted on stackoverflow) but I have
yet to find an approach that works for this structure.
Note:
1) The code is executed in the windows 10 command window
2) I am not using wxPython, IDLE, or pyGame in this application.
3) The time to get the data, process it and write it to a file can
take from 0.5 sec to 1.5 sec
4) The key hit need not be echoed to the command window
And none of your searching found
https://docs.python.org/3/library/msvcrt.html
which is part of the standard library (and documented in the library
manual). The module IS Windows specific, you'd have to rewrite the code to
run on Linux.
Now, if your requirement was to detect a keypress WHILE the data
fetch/processing was happening, the solution will be different.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Processing a key pressed in Python 3.6
Yes, I am aware of this Dennis. However, but, on my system I actually
had it running without the msvcrt.kbhit()
This occurred during testing while trying different options. And I was
able to reproduce this several times. Why? I do not have an answer. This
is one reason why I posted the code. It would be interesting to know if
anyone else has obtained the same results.
Note: 1) 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:57:36) [MSC v.1900 64
bit (AMD64)]
2) msvcrt is built-in (at least on my system)
On 2018-01-24 18:28, Dennis Lee Bieber wrote:
On Wed, 24 Jan 2018 02:13:35 +0100, Virgil Stokes *declaimed* ?
the following:
key = msvcrt.getwch()
NOTE: per documentation, that is a blocking read... It won't return
unless a key (any key) has been pressed. That means your "more work to do"
loop requires a key press for each loop.
Recommendation would be to use kbhit() first.
if msvcrt.kbhit():
key = msvcrt.getwch()
...
Consider
-=-=-=-=-
import msvcrt as ms
import time
ESCAPE = chr(27)
workPending = True
cycle = 0
while workPending:
print("Getting imaginary data %s..." % cycle)
time.sleep(1.5)
print("\tSaving imaginary data %s..." % cycle)
time.sleep(1.5)
if ms.kbhit():
key = ms.getwch()
if key == ESCAPE:
print("Pre-termination on cycle %s..." % cycle)
break
else:
print("Random keypress %r found on cycle %s..." % (key, cycle))
cycle += 1
time.sleep(1.5)
print("Termination")
-=-=-=-=-
C:\Users\Wulfraed\Documents\Python Progs>kbhit
Getting imaginary data 0...
Saving imaginary data 0...
Getting imaginary data 1...
Saving imaginary data 1...
Random keypress u'd' found on cycle 1...
Getting imaginary data 2...
Saving imaginary data 2...
Getting imaginary data 3...
Saving imaginary data 3...
Random keypress u'a' found on cycle 3...
Getting imaginary data 4...
Saving imaginary data 4...
Pre-termination on cycle 4...
Termination
C:\Users\Wulfraed\Documents\Python Progs>
--
https://mail.python.org/mailman/listinfo/python-list
Installation of tensorflow via pip -- messages?
First I upgraded my pip *C:\Python36>python -m pip install --upgrade pip* Collecting pip Downloading https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB) 100% || 1.3MB 685kB/s Installing collected packages: pip Found existing installation: pip 9.0.3 Uninstalling pip-9.0.3: Successfully uninstalled pip-9.0.3 Successfully installed pip-10.0.1 Then I upgraded tensorflow *C:\Python36>python -m pip install --upgrade tensorflow* ... ... ... The script tensorboard.exe is installed in 'C:\Python36\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. ... ... The scripts freeze_graph.exe, saved_model_cli.exe, tensorboard.exe, toco.exe and toco_from_protos.exe are installed in 'C:\Python36\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. However, this directory is in my PATH *C:\Python36>path* PATH=C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Python36\Scripts\;C:\Python36\;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Common Files\Intel\Shared Files\cpp\bin\Intel64;C:\Program Files\Intel\iCLS Client\;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\MiKTeX 2.9\miktex\bin\x64\; ... ... Why am I getting this message, that I need to consider adding this directory to PATH when it is already in PATH? Note, all of these *.exe files are in C:\Python36\Scripts. -- https://mail.python.org/mailman/listinfo/python-list
Re: Installation of tensorflow via pip -- messages?
Thanks Paul for the prompt reply, However, each entry in this Windows 10 path has a trailing backslash. If my memory is correct, this is the default for path directories. IMHO it would have been useful to have "warning" somewhere in these messages. On 2018-04-26 20:52, Paul Moore wrote: On 26 April 2018 at 19:33, Virgil Stokes wrote: Why am I getting this message, that I need to consider adding this directory to PATH when it is already in PATH? Note, all of these *.exe files are in C:\Python36\Scripts. The PATH entry ends with a backslash, which is confusing the check done by pip. It's a known issue and has been fixed in the development version of pip, so it'll be resolved in the next release. In the meantime, you can either remove the redundant trailing backslash from your PATH, or just ignore the warning. Paul -- https://mail.python.org/mailman/listinfo/python-list
Leading 0's syntax error in datetime.date module (Python 3.6)
Module info: Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] C:\Python36>pip show datetime Name: DateTime Version: 4.2 Summary: This package provides a DateTime data type, as known from Zope 2. Unless you need to communicate with Zope 2 APIs, you're probably better off using Python's built-in datetime module. Home-page: http://pypi.python.org/pypi/DateTime Author: Zope Foundation and Contributors Author-email: zope-...@zope.org License: ZPL 2.1 Location: c:\python36\lib\site-packages Requires: zope.interface, pytz I tried first to use Python's built-in datetime module as follows: from datetime import date, timedelta d0 = date(2018,02,01) This gave the following error: Syntax Error: invalid token: C:\Users\Virgil Stokes\Desktop\Important Notes_Files\CheckProcessingDate_02.py, line 7, pos 17 d0 = date(2018,02,01) Then I used pip to install the datetime module and the same error occurred! However, when I removed the leading 0's no syntax error was reported and the correct result was returned. d0 = date(2018,2,1) Why does the datetime.date module (both built-in and site-package) not accept leading 0's? -- https://mail.python.org/mailman/listinfo/python-list
Re: Everything good about Python except GUI IDE?
On 2016-Feb-27 19:13, wrong.addres...@gmail.com wrote: On Saturday, 27 February 2016 18:08:36 UTC+2, Dietmar Schwertberger wrote: On 27.02.2016 12:18, wrong.addres...@gmail.com wrote: Isn't there any good GUI IDE like Visual Basic? I hope there are some less well known GUI IDEs which I did not come across. Thanks. As of today, there's no Python GUI builder comparable to VB 6. Thanks for stating this clearly. Everyone here has been trying to show me various ways to do the kind of things I will want to, but nobody clearly admits the limitations I will have to accept if I start with Python. I am starting to wonder if VB.net would be a better solution for the time being. I have learnt enough VB.net to manage my work but it is bloated and Microsoft dependent. There are some like QtDesigner or wxGlade, but they either don't generate Python code directly or they can only be used if you know the underlying toolkit good enough to create the GUI yourself. You may try out some, but I can almost guarantee you that you will come to the same result. If you want a GUI, create it yourself using either wxPython or PyQt. I will check it. I got the impression that you can create a GUI but that has to be converted to Python, and then you need a wrapper to put these forms in, and then they can be compiled or converted to *.exe with py2exe. Not a good way for development/debugging. For engineering applications that's probably the weakest point that Python has. It's holding back a lot of people... Well, for most measurement or control software a GUI is not really needed, but still people want it. In the 1980s everyone was happy with inputs from the command line on a line editor, but today people expect GUIs with graphics and often even animations. It is surprising that a language which seems very popular does not have GUI development infrastructure in place these many years after it got into common use. Regards, Dietmar I agree (at least largely) with the author of this email, in response to Dietmar. I have been working with Python for several years and often a GUI is needed, not by me; but, for users of my software where my target is usually numerical and image processing with a "don't make me think too much" GUI. I have mainly used wxPython (which is rather good, with good support); but, I find it rather awkward in practice and making an *.exe for users that includes wxPython is often a tedious process (at least from my own experiences). Perhaps my skills with wxPython and its API are lacking :-( . After re-reading some of the postings that are connected to GUI problems in the python-list, wxPython-users, and PyQT, it seems to me that an "improved IDLE" for Python might have helped to solve some of their problems. I am quite sure such a development would be well received by Python beginners and those migrating to Python from other languages (e.g. VB). While working on my first wxPython GUI project, I actually switched to VB to test my GUI design and to create an *.exe for the project --- this went rather fast, considering that I had no previous experience with VB. Only afterwards, did I return to wxPython, for implementation in Python 2.7, which took much longer with extensive refactoring. And Dietmar, please don't take this the wrong way, I also agree with some of the points that you have made. And I do like wxPython :-) -- https://mail.python.org/mailman/listinfo/python-list
Saving a file "in the background" -- How?
While running a python program I need to save some of the data that is being created. I would like to save the data to a file on a disk according to a periodical schedule (e.g. every 10 minutes). Initially, the amount of data is small (< 1 MB) but after sometime the amount of data can be >10MB. If a problem occurs during data creation, then the user should be able to start over from the last successfully saved data. For my particular application, no other file is being saved and the data should always replace (not be appended to) the previous data saved. It is important that the data be saved without any obvious distraction to the user who is busy creating more data. That is, I would like to save the data "in the background". What is a good method to perform this task using Python 2.7.8 on a Win32 platform? -- https://mail.python.org/mailman/listinfo/python-list
Re: Loading a module from a subdirectory
On 10-Mar-14 21:31, Virgil Stokes wrote:
I have the following folder-file structure:
C:/PythonCode/VideoPlayerSimulator/
+-- __init__.py (empty file)
+-- GlbVars.py (contains the single
class Glb)
C:/PythonCode/VideoPlayerSimulator/RubberBanding/
+-- __init__.py
+-- ImportDemo.py
where, ImportDemo.py contains the following code (a single line):
from VideoPlayerSimulator.GlbVars import Glb as G
gives the following error when I execute it:
*ImportError: No module named VideoPlayerSimulator.GlbVars*
Note:
1. My sys.path contains:
['C:\\PythonCode\\VideoPlayerSimulator', ...]
2. Python 2.7.5 on a Window 7 platform
3. The same ImportDemo.py file when executed on my Window Vista platform
with the same folder-file structure, *DOES NOT* give an error and
works
as expected.
Why does the error occur, and why does it not occur on the Windows
Vista platform?
After searching for more on this topic I found that the following did
work on Windows 7:
import imp
G = imp.load_source('foo',
'C:/PythonCode/VideoPlayerSimulator/GlbVars.py').Glb
But, I am still unable to determine why my first method does not work on
Window 7.
--
https://mail.python.org/mailman/listinfo/python-list
Loading a module from a subdirectory
I have the following folder-file structure: C:/PythonCode/VideoPlayerSimulator/ +-- __init__.py (empty file) +-- GlbVars.py (contains the single class Glb) C:/PythonCode/VideoPlayerSimulator/RubberBanding/ +-- __init__.py +-- ImportDemo.py where, ImportDemo.py contains the following code (a single line): from VideoPlayerSimulator.GlbVars import Glb as G gives the following error when I execute it: *ImportError: No module named VideoPlayerSimulator.GlbVars* Note: 1. My sys.path contains: ['C:\\PythonCode\\VideoPlayerSimulator', ...] 2. Python 2.7.5 on a Window 7 platform 3. The same ImportDemo.py file when executed on my Window Vista platform with the same folder-file structure, *DOES NOT* give an error and works as expected. Why does the error occur, and why does it not occur on the Windows Vista platform? -- https://mail.python.org/mailman/listinfo/python-list
Re: Sudoku solver
On 27-Mar-2015 15:09, Dave Angel wrote: On 03/27/2015 09:56 AM, Marko Rauhamaa wrote: "Frank Millman" : So what I am talking about is called a "satisfactory" puzzle, which is a subset of a "proper" puzzle. That is impossible to define, though, because some people are mental acrobats and can do a lot of deep analysis in their heads. What's satisfactory to you may not be satisfactory to me. Besides, looking for "satisfactory" patterns can involve a truckload of trial and error. I know, let's use "regular expressions" If you are interested, I have written a python (wxPython GUI) for solving Sudoku problems. It even has a "hint" mode that can be used to lead you to a solution. I have tested it on the world's hardest Sudoku (published by a Finish mathematician) and it solves it very fast. I have also written another version that finds ALL the solutions to any Sudoku problem (that has a solution) using an LP approach --- this is non-trivial. However, I have not been able to give a strict mathematical proof that it does indeed find all solutions. If you wish to really understand the mathematics behind Sudoku then I suggest the book "Taking Sudoku Seriously" by Jason Rosenhouse and Laura Taalman (Oxford University Press, 2011). Unfortunately, they do not consider the LP approach in this book (an oversight IMHO). --V. Stokes -- https://mail.python.org/mailman/listinfo/python-list
An improved version of the main snippet
I have attached what I believe to be an improved version of my main snippet for
testing.
--V :-)
'''
Purpose: get current bid, ask and rate for currency exchanges (FOREX trading)
Note:
1. yahoo seems to give the best estimates for the currency exchange rates
2. Not sure where the "bid" and "ask" values come from.
3. Not sure how often updates to currencies occur during weekdays; but very fast (real-time?) during weekdays
Author: v...@it.uu.se
Vesion: 2015.05.05.2
'''
import pytz
from yahoo_finance import Currency
from datetime import datetime
import time
#from time import clock # clock gives good resolution in MS Windows
NUM_TICS = 500# number of values to be downloaded
CURRENCY_PAIR = 'EUR/SEK'
MY_TIME = "Europe/Stockholm"
OTHER_TIME= "America/New_York"
FILE_OT = 'CurrencyInfo.txt'
PRINT_TIMEZONES = True
PRINT_TIMEZONES = False
if PRINT_TIMEZONES:
for tz in pytz.all_timezones:
time.sleep(0.5)
print (tz)
def updateMeanVar(x,k,mu,vr):
'''
Purpose: Update the estimates for the mean and variance (recursive mean,variance)
Inputs:
x -- new value (x_k)
k -- counter (index) for new value (1,2,...)
mu -- previously estimated mean (x_k not included)
vr -- previously estimated variance (x_k not included)
Otputs:
mu -- updated mean (with x_k included)
vr -- updated variance (with x_k included)
'''
delta = x - mu
mu += delta/k
vr += delta*(x - mu)
return mu,vr
def get_Times(myTimeZone=MY_TIME,otherTimeZone=OTHER_TIME):
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
Mine = pytz.timezone(myTimeZone)
Other= pytz.timezone(otherTimeZone)
nowMine = datetime.now(Mine)
nowOther = datetime.now(Other)
return nowMine.strftime(fmt) + ' ('+nowOther.strftime(fmt)+')'
DateTimeStr = get_Times()
def InitFile(file=FILE_OT,currencyPair=CURRENCY_PAIR,dateTimeStr=DateTimeStr):
f = open(file,'a')
f.write('Currency Pair: %s, TimeStamp; %s\n'%(currencyPair,dateTimeStr))
f.write(' bid ask ratedatetime\n')
return f
def SaveToFile(f,Bid,Ask,Rate,dateTime):
f.write('%s %s %s %s\n'%(Bid,Ask,Rate,dateTime))
pass
currency_Ex = Currency(CURRENCY_PAIR.replace('/',''))
fOt = InitFile()
print ('Currency pair: %s'%CURRENCY_PAIR)
print (' bid ask ratedatetime')
prev_dateTime = currency_Ex.get_trade_datetime()
NMax = NUM_TICS
mu = 0.0; va = 0.0
for k in range(1,NUM_TICS+1):
t0 = time.clock()
currency_Ex.refresh()
Bid = currency_Ex.get_bid()
if Bid == None:
continue
Ask = currency_Ex.get_ask()
if Ask == None:
continue
Rate = currency_Ex.get_rate()
if Rate == None:
continue
dateTime = currency_Ex.get_trade_datetime()
if dateTime == None:
continue
print ('%s %s %s %s'%(Bid,Ask,Rate,dateTime))
if dateTime[:16] != prev_dateTime[:16]:
# Save currency exchange info when minute changes
SaveToFile(fOt,Bid,Ask,Rate,dateTime)
prev_dateTime = dateTime
# Estimate Time To Completion (ETTC) with recursive mean of 'loop' time
dt= time.clock() - t0
mu,va = updateMeanVar(dt,k,mu,va)
ETTC = mu*(NMax-k)
m,s = divmod(ETTC,60)
h,m = divmod(m,60)
print ('ETTC = %d:%02d:%03d'%(h,m,s))
fOt.close()
--
https://mail.python.org/mailman/listinfo/python-list
Questions on Pickle and Shelve
Here is snippet of Python (vers. 2.7.10) code that bothers me.
import cPickle as pickle
print "Pickle lists:"
dogs = ['Rover','King','Spot','Rufus']
cats = ['Mimi','Misty','Sasha']
with open('pickle.dat', 'wb') as pfile:
pickle.dump(dogs, pfile)
pickle.dump(cats,pfile)
del(dogs); del(cats)
with open('pickle.dat', 'rb') as pfile:
dogs = pickle.load(pfile)
cats = pickle.load(pfile)
print dogs, '\n', cats, '\n'
import shelve
# Note! __exit__ attribute undefined for shelve
sfile = shelve.open('shelve.dat')
sfile['dogs'] = dogs
sfile['cats'] = cats
sfile.close()
print "Shelve entries:"
del(cats); del(dogs)
sfile = shelve.open('shelve.dat')
#print sfile
for key in sfile.keys():
print key,' - ',sfile[key]
sfile.close()
1) Which (the pickle or shelve code) takes less total RAM, if dogs and cats
were very large?
2) When the last shelve.open is given, is the entire contents of shelve.data
transferred to RAM? Note, if the print sfile is uncommented then the entire
contents of shelve.data is printed out.
I was under the impression that the entire contents of a shelved file was not
transferred to RAM when it was opened.
--
https://mail.python.org/mailman/listinfo/python-list
Test for an empty directory that could be very large if it is not empty?
Suppose I have a directory C:/Test that is either empty or contains more
than 200 files, all with the same extension (e.g. *.txt). How can I
determine if the directory is empty WITHOUT the generation of a list of
the file names in it (e.g. using os.listdir('C:/Test')) when it is not
empty?
--
https://mail.python.org/mailman/listinfo/python-list
Butterflow installation on windows
The butterflow package (https://pypi.python.org/pypi/butterflow/0.1.4a1) has recently been released. I would like to know if anyone has been able to install it on a windows platform. -- https://mail.python.org/mailman/listinfo/python-list
Execute code after Shut Down command given --- How?
I would like to execute some Python code (popup message to be displayed) when Windows Vista/7 is shut down. That is, this code should execute after "Shut Down" is given from the "Shut Down Windows" popup, but before the actual shut down sequence starts. How to write Python code to accomplish this task? -- http://mail.python.org/mailman/listinfo/python-list
Agent-based modeling
Python seems like a good language to use for agent-based modeling. However, before starting to work on a Python package for this, I would be very interested in knowing about any existing Python code for agent-based modeling. --V -- http://mail.python.org/mailman/listinfo/python-list
Re: Agent-based modeling
On 10-Nov-2011 16:16, Jerry Zhang wrote: 2011/11/10 Virgil Stokes mailto:v...@it.uu.se>> Python seems like a good language to use for agent-based modeling. However, before starting to work on a Python package for this, I would be very interested in knowing about any existing Python code for agent-based modeling. I am assuming you are talking about the pattern design -- agent pattern, right? --V -- http://mail.python.org/mailman/listinfo/python-list Sorry Jerry if my email was unclear. I am referring to agent-based modeling as defined at http://en.wikipedia.org/wiki/Agent-based_model -- http://mail.python.org/mailman/listinfo/python-list
Module msvcrt for Python
I am running Python 2.6.6 on a Windows Vista platform and for some reason the module msvcrt is not present. How can I install the msvcrt module in my Python 2.6.6? God Jul :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Module msvcrt for Python
On 18-Dec-2011 11:31, Virgil Stokes wrote: I am running Python 2.6.6 on a Windows Vista platform and for some reason the module msvcrt is not present. How can I install the msvcrt module in my Python 2.6.6? God Jul :-) I found the problem! My code was using Python 2.5 (inside cygwin) by default and there it was unable to import msvcrt. But, when I ran the same code in my Python 2.6 installation it worked as it should (msvcrt module was imported). Sorry for the "noise". Best :-) -- http://mail.python.org/mailman/listinfo/python-list
CAB files
I would appreciate python code for creating *.cab files. --V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
Re: Checking a file's time stamp -- followup
Christian Heimes wrote:
William Purcell wrote:
Hi all,
I am wanting to check to see the last time a file was edited. For
example, I
have a directory containing two text files, file1.txt and
file2.txt. I
want to be able to process these files but only if they have been
edited
since the last time they were processed. I think that I want to be
able to
check the time stamp of each file. Can anyone tell me how to do that or
point me in a better direction of checking the last time a file was
edited?
>>> import os
>>> stat = os.stat("/etc/passwd")
>>> print stat
(33188, 362259, 2053L, 1, 0, 0, 1690, 1218550501, 1218118498,
1218118498)
>>> dir(stat)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__',
'__eq__', '__ge__', '__getattribute__', '__getitem__',
'__getslice__', '__gt__', '__hash__', '__init__', '__le__',
'__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__str__',
'n_fields', 'n_sequence_fields', 'n_unnamed_fields', 'st_atime',
'st_blksize', 'st_blocks', 'st_ctime', 'st_dev', 'st_gid', 'st_ino',
'st_mode', 'st_mtime', 'st_nlink', 'st_rdev', 'st_size', 'st_uid']
>>> stat.st_mtime
1218118498.0
use these shortcuts, IMHO they are easier than os.stat.
os.path.getmtime() - get modified time
os.path.atime()- get last accessed time (careful some admins turn
this
off on their servers for performance reasons)
os.path.ctime()- get creation time
-Larry
Is it possible to change the time stamp of a file (Win2K platform)? If
yes, how?
--
http://mail.python.org/mailman/listinfo/python-list
Java-to-Python?
I have a rather large Java package for the analysis of networks that I would like to convert to Python. Many of the classes in the Java package are "Serializable". Any recommendations on Java-to-Python (2.6) would be appreciated. --V -- http://mail.python.org/mailman/listinfo/python-list
Re: Sikuli: the coolest Python project I have yet seen...
On 25-Jan-2010 04:18, Ron wrote: Sikuli is the coolest Python project I have ever seen in my ten year hobbyist career. An MIT oepn source project, Sikuli uses Python to automate GUI tasks (in any GUI or GUI baed app that runs the JVM) by simply drag and dropping GUI elements into Python scripts as function arguments. Download at http://sikuli.csail.mit.edu/ I also did this This link is broken! --V -- http://mail.python.org/mailman/listinfo/python-list
Web servers
Any suggestions on using Python to connect to Web servers (e.g. to access financial time series data)? --V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
Accessing a Web server --- how?
If one goes to the following URL: http://www.nordea.se/Privat/Spara%2boch%2bplacera/Strukturerade%2bprodukter/Aktieobligation%2bNr%2b99%2bEuropa%2bAlfa/973822.html it contains a link (click on "Current courses NBD AT99 3113A") to: http://service.nordea.com/nordea-openpages/six.action?target=/nordea.public/bond/nordeabond.page&magic=%28cc+%28detail+%28tsid+310746%29%29%29&; and if you now click on the tab labeled "history and compare" this will take you to: http://service.nordea.com/nordea-openpages/six.action?target=/nordea.public/bond/nordeabond.page&magic=%28cc+%28detail+%28tsid+310746%29+%28view+hist%29%29%29&; Finally...This is where I would like to "connect to" the data on a daily basis or to gather data over different time intervals. I believe that if I can get some help on this, then I will be able to customize the code as needed for my own purposes. It should be clear that this is financial data on a fond managed by Nordea Bank AB. Nordea is one of the largest banks in Scandinavia. Note, that I do have some experience with Python (2.6 mainly), and find it a very useful and powerful language. However, I have no experience with it in the area of Web services. Any suggestions/comments on how to set up this financial data service project would be greatly appreciated, and I would be glad to share this project with any interested parties. Note, I posted a similar message to the list pywebsvcs; but, received no responses. -- V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
A web site using Python
I would like to design a web site that can be used to help people to find a cat that they can adopt. Note, this is a non-profit project, but one that I believe to be quite important. Here are some of my initial thoughts on this project. /Site purpose:/ *To provide a web site for anyone to look at information on cats at this home, and how they can adopt one or more of these homeless cats.* /Some features of the site:/ 1. A cat database that I as the web site designer would create. This database would contain an entry for each cat available for adoption. It would include such things as the name, sex, age, paths to image(s) and/or video(s) of the cat, health status, etc (see below). 2. Anyone accessing this site should be able to easily navigate around it and to examine entries in this database. The client (designated person at the home where the cats are kept for adoption) would be given privileges to modify the database (add, delete, and modify entries). The user interface for the client to update the database should be very easy to use. This GUI provided to the client for modification of the database would be written in Python. 3. There would be no inputs to this web site. There would be an embedded link for a potential customer to send an email to the responsible person (bringing up their email client). 4. Track number of visitors to the site. /Preliminary notes on the database/ Fields: - ID code (key) - Name - Sex (M / F) - Neutered / Not neutered - Age (estimated) - Type (breed) - Tagged (chip or ear marking)/ Not tagged - Checked In date (yy/mm/dd) - Checked Out date (yy/mm/dd) - Status (needs home / has home) - Social state (1,2,3,4,5) - Health state (1,2,3,4,5) - Companion state (1,2,3,4,5) - Image (file name) % multiple files allowed - Video (file name) % multiple files allowed - Medical/vet data (text on vaccinations, etc.) - General information (text on cat that includes comments, observations, etc.) --- Notes on database: * state = 1, Best 5, Worst Examples: Social state = 5, very unfriendly, afraid, etc. 3, can touch if careful 1, very friendly, unafraid Health state = 5, not in good health (e.g. infection) 3, only minor health problems 1, in very good health Companion state = 5, must have another cat or cats as company 3, could be with other cat(s) company 1, does not need the company of another cat Now, with this initial information (granted this is very rough), my question: *How, armed with Python 2.6 (or 2.7) and all of the Python packages available, should I attack the problem of getting this web site up and running on a Windows platform?* Please keep in mind that do have some experience with Python and HTML; but, this would be my first web site project using Python. Any suggestions, study plan, references, etc. would be welcomed. --V -- http://mail.python.org/mailman/listinfo/python-list
Universal Feed Browser problem in feedparser.py
I am running Python 2.6 on a Windows Vista (32-bit) platform. I recently
installed the Universal Feed Parser package (feedparser-5-0). When I try to
execute the following commands:
>>> import feedparser
>>> d = feedparser.parse("http://feedparser.org/docs/examples/atom10.xml";)
which is given at http://www.feedparser.org/
I get an Assertion error in the following function in feedparser.py
def __getattr__(self, key):
try:
return self.__dict__[key]
except KeyError:
pass
try:
assert not key.startswith('_') <--- Error occurs when this
statement is executed.
return self.__getitem__(key)
except:
raise AttributeError, "object has no attribute '%s'" % key
Why does the error occur and how should he be corrected?
--
http://mail.python.org/mailman/listinfo/python-list
Re: random number including 1 - i.e. [0,1]
John Yeung wrote: On Jun 10, 1:52 am, Steven D'Aprano wrote: On Tue, 09 Jun 2009 22:21:26 -0700, John Yeung wrote: Therefore, to me the most up-to-date docs (which say that uniform(a, b) returns a float in the closed interval [a, b]) is closer to correct than before, but still fails to point out the full subtlety of the behavior. Which is? That uniform(a, b) will return a random float in the semi-open interval [a, b) for certain values of a and b; and in the closed interval [a, b] for other values of a and b. (Swap a and b if a > b.) To me, the fact that you sometimes get a semi-open interval and sometimes a closed interval is worth noting in the docs. John I took the following direct from "The Python Library Reference (Release 2.6.2)" , Guido van Rossum, Fred L. Drake, Jr. editor, June 10, 2009. On p. 216, Almost all module functions depend on the basic function random(), which generates a random float uniformly in the semi-open range [0.0, 1.0). Python uses the Mersenne Twister as the core generator. It produces 53-bit precision floats and has a period of 2**19937-1. The underlying implementation in C is both fast and threadsafe. The Mersenne Twister is one of the most extensively tested random number generators in existence. However, being completely deterministic, it is not suitable for all purposes, and is completely unsuitable for cryptographic purposes. The notation above means that 0 is included but 1 is not (as pointed out by Esmail). I agree with Esmail, that it is important to know if this is correct, since the "drawing" of pseudo RVs from other distributions can depend on this function. The following is taken from MatLab (R2007b), The rand function now supports a method of random number generation called the Mersenne Twister. The algorithm used by this method, developed by Nishimura and Matsumoto, generates double precision values in the closed interval [2^(-53), 1-2^(-53)], with a period of (2^19937-1)/2. Note, that it will not generate a 0 or 1; i.e., the interval for the pseudo RV can be written as (0,1) or [2^(-53), 1-2^(-53)], where the latter is more informative. For a full description of the Mersenne twister algorithm, see http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html. If indeed Python 2.6.2 is using the Mersenne twister algorithm as defined by the creators of this algorithm (go to the link given above), then IMHO the documentation should be corrected. I hope that this helps. --V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
matplotlib installation
Any suggestions on installing matplotlib for Python 2.6.2 on a Windows Vista platform? --V -- http://mail.python.org/mailman/listinfo/python-list
On the property function
Does anyone have a good example (or examples) of when "property(...)" can be useful? Thank you, --V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
PyODE
Does anyone have PyODE running on Python 2.6.2? --V -- http://mail.python.org/mailman/listinfo/python-list
On out-of-date Python Applications
I am not a heavy user of Python; but, I do work with it and some of its application packages (e.g. PyODE), in an academic setting. Many of these applications packages have a Windows installer which usually works fine. However, I also try to keep up with the latest release of Python, and this is where I often have problems. That is, the latest Windows installer provided for some of these applications will not install on the latest version of Python. I do understand that there can be a time lag between the release of Python applications and the latest Python. I also appreciate the work of the people that are responsible for these applications. My question is --- Is there anything I, as a user of an application package that is out-of-date with respect to the latest Python, can do to help in this process of bringing an application up-to-date? --V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
Re: On out-of-date Python Applications
David Robinow wrote: On Mon, Jul 20, 2009 at 7:24 AM, John Machin wrote: ... The next step would be to try to compile ODE 0.7 or 0.8 with VS9 -- however this would require "project files" for ODE for VS9, and there aren't any on the ODE website; it has only those for VS3 and VS5. The ODE site is a mess. Go to http://www.ode.org/svn.html and click on: Instructions for accessing the repository Scroll down to the section "Building with Premake" (Note that there is no directory "ode/build" -- you want the "build" directory, which contains premake4.exe) I used "premake4 --with-demos --with-tests vs2008" I have successfully compiled ode-0.11.1 using these instructions. I have not yet run the tests or demos or tried to compile PyODE. Thanks for this information David. Now that you have successfully compiled ode-0.11.1, how can one finish this process --- compile PyODE? --V -- http://mail.python.org/mailman/listinfo/python-list
"Deprecated sets module" with Python 2.6
I would appreciate help on correcting a problem when trying to create an *.exe file using py2exe via GUI2exe with Python 2.6.2. When using GUI2exe to create an *.exe I always get the following warning during the compile process: C:\Python26\lib\site-packages\py2exe\build_exe.py:16: DeprecationWarning: the sets module is deprecated import sets and this results in the creation of an *.exe file that can not be executed. On the other hand, if I use the same procedure (on the same Python code) with Python 2.5, there are no warnings and the *.exe works fine. The procedure used is that given in the example "Less simpler one" at http://code.google.com/p/gui2exe/wiki/GUI2ExeExamplesin Any suggestions, help, ... would be greatly appreciated. Thanks, --V. -- http://mail.python.org/mailman/listinfo/python-list
Re: "Deprecated sets module" with Python 2.6
Diez B. Roggisch wrote: Virgil Stokes schrieb: I would appreciate help on correcting a problem when trying to create an *.exe file using py2exe via GUI2exe with Python 2.6.2. When using GUI2exe to create an *.exe I always get the following warning during the compile process: C:\Python26\lib\site-packages\py2exe\build_exe.py:16: DeprecationWarning: the sets module is deprecated import sets and this results in the creation of an *.exe file that can not be executed. On the other hand, if I use the same procedure (on the same Python code) with Python 2.5, there are no warnings and the *.exe works fine. The procedure used is that given in the example "Less simpler one" at http://code.google.com/p/gui2exe/wiki/GUI2ExeExamplesin Any suggestions, help, ... would be greatly appreciated. If you don't need your app running on python2.3 and earlier, just remove the sets-module and replace it with the builtin "set". Of course Diez, this is a good suggestion. However, in this case the Python code that I am trying to convert into an *.exe file does not refer to sets directly; i.e, the use of this module is buried within a package that is imported (wxPython) which is not under my control. --V Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: "Deprecated sets module" with Python 2.6
Virgil Stokes wrote: Diez B. Roggisch wrote: Virgil Stokes schrieb: I would appreciate help on correcting a problem when trying to create an *.exe file using py2exe via GUI2exe with Python 2.6.2. When using GUI2exe to create an *.exe I always get the following warning during the compile process: C:\Python26\lib\site-packages\py2exe\build_exe.py:16: DeprecationWarning: the sets module is deprecated import sets and this results in the creation of an *.exe file that can not be executed. On the other hand, if I use the same procedure (on the same Python code) with Python 2.5, there are no warnings and the *.exe works fine. The procedure used is that given in the example "Less simpler one" at http://code.google.com/p/gui2exe/wiki/GUI2ExeExamplesin Any suggestions, help, ... would be greatly appreciated. If you don't need your app running on python2.3 and earlier, just remove the sets-module and replace it with the builtin "set". Of course Diez, this is a good suggestion. However, in this case the Python code that I am trying to convert into an *.exe file does not refer to sets directly; i.e, the use of this module is buried within a package that is imported (wxPython) which is not under my control. --V Diez Whoops, the reference to the module sets is in py2exe not wxPython. --V -- http://mail.python.org/mailman/listinfo/python-list
Start-up program
How difficult is to create a program that will be executed when Windows Vista is started? As Windows Calendar does, for example. I am actually more interested in the Python tools that might be used for this task. I hope that this question is not inappropriate for the list. :-\ --V -- http://mail.python.org/mailman/listinfo/python-list
Financial time series data
Has anyone written code or worked with Python software for downloading financial time series data (e.g. from Yahoo financial)? If yes, would you please contact me. --Thanks, V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
Re: Financial time series data
On 09/02/2010 08:15 PM, Hidura wrote:
But what kind of data you want to download?, because the financial
time it's basicly html code and you can work very well with a parser
2010/9/2, Virgil Stokes:
Has anyone written code or worked with Python software for downloading
financial time series data (e.g. from Yahoo financial)? If yes, would you
please contact me.
--Thanks,
V. Stokes
--
http://mail.python.org/mailman/listinfo/python-list
Here is a snippet of python code that I am trying to use for downloading
financial data; but, I do not understand why it returns information from
the second HTML page.
import urllib2
'''
I am trying to read each row of the table at:
http://finance.yahoo.com/q/cp?s=^GSPC
'''
ticker = []
url =
urllib2.urlopen("http://download.finance.yahoo.com/d/quotes.csv...@%5egspc&f=sl1d1t1c1ohgv&e=.csv&h=PAGE".replace('PAGE',
str(0)))
data = url.read()
Note, it does get all 50 rows of the first page; but, why does it also
get the first row of the "next" HTML page?
--V
--
http://mail.python.org/mailman/listinfo/python-list
Re: Financial time series data
A more direct question on accessing stock information from Yahoo. First, use your browser to go to: http://finance.yahoo.com/q/cp?s=%5EGSPC+Components Now, you see the first 50 rows of a 500 row table of information on S&P 500 index. You can LM click on 1 -50 of 500 |First|Previous|Next|Last below the table to position to any of the 10 pages. I would like to use Python to do the following. *Loop on each of the 10 pages and for each page extract information for each row --- How can this be accomplished automatically in Python?* Let's take the first page (as shown by default). It is easy to see the link to the data for "A" is http://finance.yahoo.com/q?s=A. That is, I can just move my cursor over the "A" and I see this URL in the message at the bottom of my browser (Explorer 8). If I LM click on "A" then I will go to this link --- Do this! You should now see a table which shows information on this stock and *this is the information that I would like to extract*. I would like to do this for all 500 stocks without the need to enter the symbols for them (e.g. "A", "AA", etc.). It seems clear that this should be possible since all the symbols are in the first column of each of the 50 tables --- but it is not at all clear how to extract these automatically in Python. Hopefully, you understand my problem. Again, I would like Python to cycle through these 10 pages and extract this information for each symbol in this table. --V -- http://mail.python.org/mailman/listinfo/python-list
