python-blosc 1.0.3 released
Announcing python-blosc 1.0.3
A Python wrapper for the Blosc compression library
What is it?
===
Blosc (http://blosc.pytables.org) is a high performance compressor
optimized for binary data. It has been designed to transmit data to
the processor cache faster than the traditional, non-compressed,
direct memory fetch approach via a memcpy() OS call.
Blosc works well for compressing numerical arrays that contains data
with relatively low entropy, like sparse data, time series, grids with
regular-spaced values, etc.
python-blosc is a Python package that wraps it.
What is new?
Blosc has been updated to 1.1.3, allowing much improved compression
ratio under some circumstances. Also, the number of cores on Windows
platform is detected correctly now (thanks to Han Genuit).
Last, but not least, Windows binaries for Python 2.6 and 2.7 are
provided (both in 32-bit and 64-bit flavors).
For more info, you can see the release notes in:
https://github.com/FrancescAlted/python-blosc/wiki/Release-notes
Basic Usage
===
# Create a binary string made of int (32-bit) elements
import array
a = array.array('i', range(10*1000*1000))
bytes_array = a.tostring()
# Compress it
import blosc
bpacked = blosc.compress(bytes_array, typesize=a.itemsize)
len(bytes_array) / len(bpacked)
110 # 110x compression ratio. Not bad!
# Compression speed?
from timeit import timeit
timeit(blosc.compress(bytes_array, a.itemsize),
import blosc, array; \
a = array.array('i', range(10*1000*1000)); \
bytes_array = a.tostring(), \
number=10)
0.040534019470214844
len(bytes_array)*10 / 0.0405 / (1024*1024*1024)
9.1982476505232444 # wow, compressing at ~ 9 GB/s. That's fast!
# This is actually much faster than a `memcpy` system call
timeit(ctypes.memmove(b.buffer_info()[0], a.buffer_info()[0], \
len(a)*a.itemsize),
import array, ctypes; \
a = array.array('i', range(10*1000*1000)); \
b = a[::-1], number=10)
0.10316681861877441
len(bytes_array)*10 / 0.1031 / (1024*1024*1024)
3.6132786600018565 # ~ 3.6 GB/s is memcpy speed
# Decompress it
bytes_array2 = blosc.decompress(bpacked)
# Check whether our data have had a good trip
bytes_array == bytes_array2
True# yup, it seems so
# Decompression speed?
timeit(s2 = blosc.decompress(bpacked),
import blosc, array; \
a = array.array('i', range(10*1000*1000)); \
bytes_array = a.tostring(); \
bpacked = blosc.compress(bytes_array, a.itemsize), \
number=10)
0.083872079849243164
len(bytes_array)*10 / 0.0838 / (1024*1024*1024)
4.4454538167803275 # decompressing at ~ 4.4 GB/s is pretty good too!
[Using a machine with 8 physical cores with hyper-threading]
The above examples use maximum compression level 9 (default), and
although lower compression levels produce smaller compression ratios,
they are also faster (reaching speeds exceeding 11 GB/s).
More examples showing other features (and using NumPy arrays) are
available on the python-blosc wiki page:
http://github.com/FrancescAlted/python-blosc/wiki
Documentation
=
Please refer to docstrings. Start by the main package:
import blosc
help(blosc)
and ask for more docstrings in the referenced functions.
Download sources
Go to:
http://github.com/FrancescAlted/python-blosc
and download the most recent release from here.
Blosc is distributed using the MIT license, see LICENSES/BLOSC.txt for
details.
Mailing list
There is an official mailing list for Blosc at:
bl...@googlegroups.com
http://groups.google.es/group/blosc
**Enjoy data!**
--
Francesc Alted
--
http://mail.python.org/mailman/listinfo/python-announce-list
Support the Python Software Foundation:
http://www.python.org/psf/donations/
[ANN]: 'tren' Cross-Platform Batch Renaming Tool, Version 1.239 Released
'tren' Version 1.239 is now released and available for download at: http://www.tundraware.com/Software/tren The last public release was 1.217. - What's New In This Release? --- This release fixes several critical bugs and adds a number of new features. Existing users are urged to upgrade at their earliest opportunity: FIXED: Backups weren't being improperly named. FIXED: Changed TREN envvar processing to properly parse quoting. FIXED: Each renaming target must either be something that exists or a wildcard that expands to something that exists. Formerly, nonexistent renaming targets were silently ignored. NEW: Debug now shows the incremental change to each file name as each renaming request is applied left-to-right. Use -dq to see this more clearly. NEW: The environment variable TRENINCL is now supported. This allows you to specify a path to search when looking for include files. NEW: Implemented /NAMESOFAR/ renaming token. NEW: Added -e type case conversion option - new kind of renaming request. c - Capitalize l - Lower s - Swap case t - Title - Char's followiing non-alpha are capitalized u - Upper NEW: Added -T option to allow user to target a substring of the full filename for renaming OTHER: Documentation additions and updates including clarification of old features, description of new features, and improved table-of-contents for PDF and PS document formats. What Is 'tren'? -- 'tren' is a general purpose file and directory renaming tool. Unlike commands like 'mv', 'tren' is particularly well suited for renaming *batches* of files and/or directories with a single command line invocation. 'tren' eliminates the tedium of having to script simpler tools to provide higher-level renaming capabilities. 'tren' is also adept at renaming only *part of an existing file or directory name* either based on a literal string or a regular expression pattern. You can replace any single, group, or all instances of a given string in a file or directory name. 'tren' implements the idea of a *renaming token*. These are special names you can embed in your renaming requests that represent things like the file's original name, its length, date of creation, and so on. There are even renaming tokens that will substitute the content of any environment variable or the results of running a program from a shell back into the new file name. 'tren' can automatically generate *sequences* of file names based on their dates, lengths, times within a given date, and so on. In fact, sequences can be generated on the basis of any of the file's 'stat' information. Sequence numbers can be ascending or descending and the count can start at any initial value. Counting can take place in one of several internally defined counting alphabets (decimal, hex, octal, alpha, etc.) OR you can define your own counting alphabet. This allows you to create sequences in any base (2 or higher please :) using any symbol set for the count. 'tren' is written in pure Python and requires Python version 2.6.x or later. It is known to run on various Unix-like variants (FreeBSD, Linux, MacOS X) as well as Windows. It will also take advantage of 'win32all' Python extensions on a Windows system, if they are present. - Complete details of all fixes, changes, and new features can be found in the WHATSNEW.txt and documentation files included in the distribution. A FreeBSD port has been submitted as well. -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/
ANN: GMPY 1.14 and GMPY2 alpha1 released
Everyone, I'm pleased to annouce the release of both a new production and experimental release of GMPY. GMPY is a wrapper for the MPIR or GMP multiple-precision arithmetic library. The experimental release (GMPY2) now includes support for the MPFR floating-point library. GMPY is available for download from: http://code.google.com/p/gmpy/ Production release -- GMPY 1.14 is the updated stable release. A memory leak was fixed so it is highly recommended that all user upgrade to this version. In addition to a few other bug fixes, GMPY 1.14 is compatible with the changes to the hashing code in Python 3.2a4. The 64-bit Windows installer for Python 3.2 should only be used with 3.2a4 and later. Even though my primary development focus has shifted to GMPY2 (see below), GMPY 1.X will continue to receive bug and compatibility fixes. Experimental release To simplify the codebase, allow for changes in the API, and support simultaneous installation, the development version has been renamed to GMPY2. The following is list of changes in GMPY2: In 2.0.0a0 -- * support for a mutable integer type xmpz * removal of random number functions * xmpz supports slices for setting/clearing bits * some methods have been renamed (scan1 - bit_scan1) * support for Python prior to 2.6 has been removed * support for all division modes has been added * ceiling - round to +Infinity * floor - round to -Infinity * truncate - round to zero * 2exp - division by a power of 2 * support is_even() and is_odd() In 2.0.0a1 -- * support for the MPFR floating point library If you use GMPY regularly, please test GMPY2. There have been several requests asking for a mutable integer and I am curious if there are real-world performance improvements. Please report any issues! casevh -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/
Re: Distribution of Python Scripts
On 11/18/2010 11:48 AM, tazz_ben wrote: Hi Folks - I'm an experienced programmer, but this is my first app with python, so I apologize for any stupidity on my part. So I've written/still working on a command line tool written in Python. It talks to a web service, so there really isn't anything in it that is dependent on a particular os. My goal is to distribute the script as a tool to my users so they can use it (they are on multiple platforms), the vast majority are not programmers. So, I'm looking for a way to distribute the app that takes care of details like them having python installed, or having an older version etc. From reading some docs it sounds like Squeeze did exactly this back in the day, but it is ancient and I'm writing in 2.7. Just to be clear I could care less if they see the source code, my goal is all around ease of use. I would love to have one distribution file for all platforms, but I can live with making three if that's the only option. So, what's my options. The only one I could find was cxFreeze: http://cx-freeze.sourceforge.net/ regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon 2011 Atlanta March 9-17 http://us.pycon.org/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On 11/18/2010 5:35 PM, Mark Wooding wrote: John Naglena...@animats.com writes: Python is defined by what a naive interpreter with late binding and dynamic name lookups, like CPython, can easily implement. Simply emulating the semantics of CPython with generated code doesn't help all that much. Indeed. Because you can monkey patch Python objects from outside the class, a local compiler, like a JIT, can't turn name lookups into hard bindings. Nor can it make reliable decisions about the types of objects. But it /can/ make guesses. A dynamic runtime doesn't have to predict everything right in advance; it only has to predict most things sort of well enough, and fix up the things it got wrong before anyone notices. For example, A Python compiler could inline a function call if it makes a note to recompile the calling function if the called function is modified. Most functions aren't redefined, so this is probably a pretty good guess. That adds a sizable performance penalty. Short of global program analysis, the compiler can't tell when code for the hard cases needs to be generated. The right approach is to guess that things are going to be done the easy way, and then detect when the guess is wrong. That's been done successfully for Self and JavaScript. It's not easy. See this talk on JaegerMonkey: http://blog.cdleary.com/2010/09/picing-on-javascript-for-fun-and-profit/ The effort needed to do that for Javascript is justified by the size of the installed base. The Unladen Swallow people had plans to go in that direction, but they underestimated the size of the job. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Python has a really hidden encapsulation?
In article 87fwvw214g@gmail.com, Arnaud Delobelle arno...@gmail.com wrote: Are you referring to the fact that in Python, if an attribute is a property, the __dict__ lookup on the instance is not performed? As in: class A: ... @property ... def x(self): return 42 ... a = A() a.__dict__['x'] = 24 a.x 42 a.__dict__['x'] 24 This is documented, but I actually don't know the reason for it. Because otherwise you would be able to overwrite the property with a value. -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ Look, it's your affair if you want to play with five people, but don't go calling it doubles. --John Cleese anticipates Usenet -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On 11/18/2010 7:57 PM, Martin Gregorie wrote: [...] .pyc files are an optional extra: in a *nix environment they are not produced if you use a hash-bang line to execute the source file: they only appear if you run the source with a python file.py command. Close, but no cigar. They are only created (assuming file permissions permit) when the module is *imported*. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon 2011 Atlanta March 9-17 http://us.pycon.org/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: what's the precision of fractions.Fraction?
On Thu, 18 Nov 2010 20:08:00 +0100, Stefan Sonnenberg-Carstens wrote: If you need it really *precise*, cast your Fractions into type Decimal: It's actually the opposite. Decimal has many advantages, but being really precise is not one of them. It suffers the exact same issues re precision and round-off as binary floating point. It is easy to demonstrate that there are numbers which cannot be represented precisely as Decimals no matter how many decimal places you use, but can be represented exactly as Fractions. Fraction can exactly represent every Decimal, but Decimal cannot represent exactly every Fraction. We're not talking about weird edge cases either, but simple numbers that you're likely to come across every day: from decimal import Decimal one_ninth = Decimal(1)/Decimal(9) two_thirds = Decimal(2)/Decimal(3) one_ninth*6 == two_thirds False from fractions import Fraction one_ninth = Fraction(1, 9) two_thirds = Fraction(2, 3) one_ninth*6 == two_thirds True Still not convinced? f = Fraction(1) + Fraction(1, 10**10) f != 1 True (which is virtually instantaneous, by the way) compared to the *much* slower: d = Decimal(1) + Decimal(1)/Decimal(10**10) d != 1 False Yes, I could try to set the Decimal context to 100,000 decimal places -- and just as easily defeat it again by adding one more to the exponent. In my opinion, the Fraction module is one of the least appreciated and underused modules in the standard library -- and I include myself in that. It really is a joy, and I don't use it anywhere near enough. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Fri, 19 Nov 2010 00:26:41 +, MRAB wrote: On 19/11/2010 00:07, Steven D'Aprano wrote: On Thu, 18 Nov 2010 14:21:47 +, Martin Gregorie wrote: I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. You're mistaken then about Python, because it does have a separate compilation phase that runs before the program can be executed. Where do you think the .pyc files come from, and what did you think the compile() function did? It just happens automatically, rather than manually. [snip] I think what he means is that you don't need explicitly to compile and then run. So if I write a small shell script to act as a wrapper around the usual compile/execute cycle of (say) C, my C programs turn into scripts, but if I ignore the wrapper and run the compile/execute commands manually, they turn back into not-scripts? -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: What was your strategy?
On Sun, 14 Nov 2010, Paul Rubin wrote: Jorge Biquez jbiq...@icsmx.com writes: I was wondering if you can share what was the strategy you followed to master Python (Yes I know I have to work hard study and practice a lot). 1. Read the tutorial http://docs.python.org/tutorial/ 2. Start writing code, and encounter various issues as usually happens. 3. Refer to the other reference manuals, web search, and ask questions in the newsgroup as you run into issues. 4. After a while you'll have hit most of the usual issues and learned how to deal with them, and how to find resolution for new issues that might come up. That's about as close to mastery as one normally reaches in this world. Python is a pretty easy language if you have a reasonable programming background when you first start with it. I think the way it's currently organized, it may not be so great for self-study if you're not already a programmer. I mean did you use special books, special sites, Nah. Wow, exactly same strategy by me. Do you think it should be GPLed, by chance ;-) ? I find this way of learning to be a bit hard (it must have helped that I was no beginner), but somehow none other option came to my head when I was approaching Python some years ago. I guess I'm not a good follower of various written rules of engagement. So, after tutorial I jumped over the standard Python docs (module index, plus library language references) until I found whatever was needed at the moment. So choice of strategy depends on choice maker. BTW, I think it was very important in my case to have specific program in mind, begging me to write it in Python. So learning was more exciting thanks to this. I came to Python from some other languages, of which only C retains it's value to me nowadays. I consider myself kind of departed from Pythonland, in search of some other, maybe better alternatives - but it is quite possible Python will join C. I'm undecided, as I've not tried 3.x yet. As a side note, I'm not quite sure Python is good for beginners. Yes, it is very simple and easy to grasp. And yes, it is a bit too simple, maybe? So a beginner learns to think in terms of nails and hammers, but may never hear of screwdrivers in his programing life. I may be wrong but, thinking of it, I feel it was good I have been exposed to Pascal and C (and few other things) long before Python. I would advise Python to casual/Sunday programers, knowing there is big chance they will never learn more than this, so Python is their best option IMHO. But in case of serious/serial ;-) programing, I would save Python for second or third language. I mean, I perceive it as rather one way to do it language and forcing this one way on unformed programer doesn't look good. No offence. See? I'm still here. Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did rm -rif on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com ** -- http://mail.python.org/mailman/listinfo/python-list
dpkt
Hi I'm looking to find some help working with dpkt. As you probably know, there really isn't any documentation, and not being astute at Python as of yet leaves me with a lot of gaps. Is there anyone here that can point me in a direction toward writing some test code for parsing gre packets? Thanks for the input; I appreciate the help. Jack -- http://mail.python.org/mailman/listinfo/python-list
Re: dpkt
Am 19.11.2010 06:39, schrieb Verde Denim: Hi I'm looking to find some help working with dpkt. As you probably know, there really isn't any documentation, and not being astute at Python as of yet leaves me with a lot of gaps. Is there anyone here that can point me in a direction toward writing some test code for parsing gre packets? Thanks for the input; I appreciate the help. Jack There is. It is inside the code (not the obvious way). http://www.monkey.org/~dugsong/dpkt/pydoc/public/dpkt-module.html (May be outdated) attachment: stefan_sonnenberg.vcf-- http://mail.python.org/mailman/listinfo/python-list
Re: Changing the EAX register with Python
dutche dut...@gmail.com wrote: Hi folks, I have a unusual question here. How can I change the value of EAX register under python under Linux?? As paimei does under Windows. My project is to have a python program that loads a C program and sets a breakpoint at some address, and then with this breakpoint I change the EAX register and then continue the program execution. With Windows and paimei I did that, but under Linux I don't know yet. You will need to find a Linux application equivalent to PaiMei. Your question is not how can I change EAX, your question is where can I find a Linux debugger that can be controlled from Python? I don't know the answer to that. gdb is quite powerful, and you can certainly control it by connecting to its stdin and stdout connections. -- Tim Roberts, t...@probo.com Providenza Boekelheide, Inc. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sending changed parameters into nested generators
On Nov 12, 10:52 pm, John O'Hagan resea...@johnohagan.com wrote:
On Sat, 13 Nov 2010, Steven D'Aprano wrote:
On Fri, 12 Nov 2010 09:47:26 +, John O'Hagan wrote:
I have a generator function which takes as arguments another generator
and a dictionary of other generators like this:
def modgen(gen, gendict):
for item in gen():
for k, v in gendict:
do_something_called_k(item, v.next())
yield item
[snip]
If anyone's still reading :) , how can I send new values to arbitrary
sub- generators?
I have a headache after reading your problem :(
I think it's a good time to point you at the Zen, particularly these five
maxims:
Beautiful is better than ugly.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
If the implementation is hard to explain, it's a bad idea.
I'm afraid that your nested generators inside another generator idea
fails all of those... it's not elegant (beautiful), it's complicated,
it's nested, and the implementation is hard to explain.
You could probably replace generators with full-blown iterators, but I
wonder what you're trying to accomplish that is so complicated that it
needs such complexity to solve it. What are you actually trying to
accomplish? Can you give a simple example of what practical task you hope
to perform? I suspect there's probably a more elegant way to solve the
problem.
I hope there is!
The project not practical but artistic; it's a real-time musical composition
program.
A (simplified) description: one module contains number-list generating
functions, others contain functions designed to filter and modify the number
lists produced, according to various parameters. Each such stream of number
lists is assigned a musical meaning (e.g. pitch, rhythm, volume, etc) and they
are combined to produce representations of musical phrases, which are sent to
a backend which plays the music as it is produced, and makes PDF scores. Each
such instrument runs as a separate thread, so several can play together in a
coordinated fashion.
All the compositional interest lies in the selection of number-list generators
and how their output is modified. For example, if I say Play every third note
up an octave it's not very interesting, compared to Play every nth note up
an interval of m, where n and m vary according to some pattern. It gets even
more interesting when that pattern is a function of x and y, which also vary
according to another pattern, and so on.
To that end, I have each parameter of each modifier set by another generator,
such that the value may change with each iteration. This may continue
recursively, until at some level we give a parameter a simple value.
That's all working, but I also want it to be interactive. Each thread opens a
terminal where new options can be entered, but so far it only works, as I
mentioned, for changing the values in a top-level mutable object.
I might first suggest this, although I have some caveats to add:
def genfilter(evaluator, **param_sources):
while True:
params = {}
for param, gen in param_sources.iteritems():
params[param] = gen.next()
yield evaluator(**params)
You can then do things like:
def concat(in1, in2):
return str(in1)+|+str(in2)
a = (i for i in range(1,5)) # generator based on a list
b = (2*i for i in xrange(1,5)) # 'pure' generator
c = genfilter(concat, in1=a, in2=b)
for x in c:
print x
1|2
2|4
3}6
4|8
or, more relevant to your original question regarding modifying things
mid-stream:
class Mult():
def __init__(self, multiplier):
self.mulitplier = multiplier
def multi(self, val):
return val*self.multiplier
m = Mult(2)
a = (i for i in range(1,10))
b = (i for i in range(1,10))
c = genfilter(m.multi, val=b)
d = genfilter(concat, in1=a, in2=c)
d.next()
1|2
d.next()
2|4
m.multiplier = 3
d.next()
3|9
Obviously more complex and useful examples could be made, but
hopefully you get the idea: don't focus on modifying the generators,
instead modify the action of the functions which take the generators
as inputs.
But a real problem with this whole strategy is that a generator's
next() function is called every time it is evaluated. If the
relationship between your various generators forms a rooted tree,
that's not a problem, but I would think the relationships form a
directed acyclic graph, and in that case, you end up 'double
incrementing' nodes in a way you don't want:
m = Mult(2)
a = (i for i in range(1,10))
b = genfilter(m.multi, val=a)
c = genfilter(concat, in1=a, in2=b)
for x in c:
print x
1|4
3|8
5|12
7|16
And that's not an unnatural situation for your application: you might
have a generator that produces a note, and then three other distinct
genfilters that you want to use to create three instruments which form
a chord based on that note. With the
Re: Does Pygoogle allows for advanced search options?
On Wed, Nov 17, 2010 at 11:54 PM, neocortex pmi...@gmail.com wrote:
Hello All,
Can anyone help me with the Pygoogle:
from pygoogle import pygoogle
You're apparently talking about http://code.google.com/p/pygoogle/
rather than http://pygoogle.sourceforge.net/ ; the former appears to
have borrowed the latter's name without permission. Oi!
word = u'something'
request_word = word.encode('utf-8')
request = ('%s+site:.edu' % request_word)
g = pygoogle(request)
g.get_result_count()
Now, I realized that domain restriction works (site:.edu etc.), but I
would like to be able to control for language too. Is that possible
with the Pygoogle? If not, how can I make that happen?
The library doesn't seem to have built-in support for filtering by
language (and Google lacks a search query-string-based operator for
that), but it looks like you could implement that feature by adding an
lr parameter with an appropriate value to the query `args`
dictionary. See the lr? entry under Web Search Specific Arguments
on http://code.google.com/apis/websearch/docs/reference.html , and
lines 68 102 of pygoogle.py.
Cheers,
Chris
--
Research for random person = too much idle time
http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pickle in a POST/GET request give EOFError
Le 17/11/2010 18:52, geremy condra a écrit :
On Wed, Nov 17, 2010 at 6:44 AM, Romaric DEFAUXr...@audaxis.com wrote:
Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
Hi everybody !
First time I write to this mailing list :)
I started writing in python last week, that's probably why I can't
understand the following problem...
I create a list called web_site_list.
This list contain dictionaries called web_site.
And some values in this dictionaries are list too.
I do that in a function and I return this :
return pickle.dumps(web_site_list)
This is working fine :)
If I do :
print %s % pickle.loads(system.get_web_site_list())
I've got the right stuffs. For example it returns :
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
I send this to a web service. I send it like that :
#I put it in params
def system_updateweb_site(server, login, password):
params = {}
params['login'] = login
params['password'] = password
params['action'] = 'updateweb_site'
params['servername'] = get_servername()
params['hosted_web_site'] = get_web_site_list()
return call_system_ws(server, params)
#Here's how I send it (I tried in GET and POST)
def call_system_ws(host, params):
query_string = urllib.urlencode(params)
#GET
# f = urllib.urlopen(http://%s/ws?%s; % (host, query_string))
#POST
f = urllib.urlopen(http://%s/ws; % (host), query_string)
result = f.readline().strip()
if result == 'ERROR':
msg = f.readline().strip()
return (False, msg)
return (True, result)
On the server side :
if action == 'updateweb_site':
if not (fields.has_key('servername') and
fields.has_key('hosted_web_site')):
raise WSError('missing parameter :
servername or hosted_web_site')
log ('ERROR : missing parameter :
servername or hosted_web_site')
else:
servername=g.db.escape_string(fields['servername'])
hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
output =
systemserver.updateweb_site(cursor, servername, hosted_web_site)
In systemserver.py :
def updateweb_site(cursor, host, hosted_web_site):
web_site_list = pickle.loads(hosted_web_site)
return %s % (web_site_list)
I catch this error :*
type 'exceptions.EOFError'*:
args = ()
message = ''
Why ?
If I just print hosted_web_site, I get this on my web page :
(lp0\n(dp1\nS\'documentroot_size\'\np2\nS\'120\'\np3\nsS\'servername\'\np4\nS\'default\'\np5\nsS\'client\'\np6\nS\'undefined\'\np7\nsS\'documentroot\'\np8\nS\'/var/www/\'\np9\nsS\'client_contact\'\np10\ng7\nsS\'serveralias\'\np11\n(lp12\nsa.
It's the pickled view of
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
Can someone help me please ? I spend my afternoon to google to try to find
a solution...
Thanks in advance !!!
Romaric Defaux
After entirely rewrite my code to not use Web service but socket (a real
client/server program) I finally found the problem... And it's not linked to
the POST or GET method...
It's because of that :
g.db.escape_string(fields['hosted_web_site'])
(escape_string is the function in MySQLdb library)
It escapes the simple quote of the pickled object, and break it...
It's good to know, NEVER escape a pickled object :)
Romaric Defaux
I'm not sure I understand what you're doing here, but I trust you've
read about and understand the security problems with pickle?
Geremy Condra
I read quickly the security problems with pickle. But I don't feel
concern about that because I run my program in a private network, not
over internet. And now I use socket to communicate on a non-standard
port, not anymore web service on the 80 port. If I plan to run it
through wan, I will encrypt datas for sure with SSL or something like
that :)
Romaric Defaux
smime.p7s
Description: S/MIME Cryptographic Signature
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pickle in a POST/GET request give EOFError
Hi,
you can use json for passing list and dict .
Pickle is dangerous .
Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
(using stdlib json in python = 2.6 or simplejson in python 2.6)
Regards
Le Thu, 18 Nov 2010 09:29:00 +0100,
Romaric DEFAUX r...@audaxis.com a écrit :
Le 17/11/2010 18:52, geremy condra a écrit :
On Wed, Nov 17, 2010 at 6:44 AM, Romaric DEFAUXr...@audaxis.com wrote:
Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
Hi everybody !
First time I write to this mailing list :)
I started writing in python last week, that's probably why I can't
understand the following problem...
I create a list called web_site_list.
This list contain dictionaries called web_site.
And some values in this dictionaries are list too.
I do that in a function and I return this :
return pickle.dumps(web_site_list)
This is working fine :)
If I do :
print %s % pickle.loads(system.get_web_site_list())
I've got the right stuffs. For example it returns :
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
I send this to a web service. I send it like that :
#I put it in params
def system_updateweb_site(server, login, password):
params = {}
params['login'] = login
params['password'] = password
params['action'] = 'updateweb_site'
params['servername'] = get_servername()
params['hosted_web_site'] = get_web_site_list()
return call_system_ws(server, params)
#Here's how I send it (I tried in GET and POST)
def call_system_ws(host, params):
query_string = urllib.urlencode(params)
#GET
# f = urllib.urlopen(http://%s/ws?%s; % (host, query_string))
#POST
f = urllib.urlopen(http://%s/ws; % (host), query_string)
result = f.readline().strip()
if result == 'ERROR':
msg = f.readline().strip()
return (False, msg)
return (True, result)
On the server side :
if action == 'updateweb_site':
if not (fields.has_key('servername') and
fields.has_key('hosted_web_site')):
raise WSError('missing parameter :
servername or hosted_web_site')
log ('ERROR : missing parameter :
servername or hosted_web_site')
else:
servername=g.db.escape_string(fields['servername'])
hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
output =
systemserver.updateweb_site(cursor, servername, hosted_web_site)
In systemserver.py :
def updateweb_site(cursor, host, hosted_web_site):
web_site_list = pickle.loads(hosted_web_site)
return %s % (web_site_list)
I catch this error :*
type 'exceptions.EOFError'*:
args = ()
message = ''
Why ?
If I just print hosted_web_site, I get this on my web page :
(lp0\n(dp1\nS\'documentroot_size\'\np2\nS\'120\'\np3\nsS\'servername\'\np4\nS\'default\'\np5\nsS\'client\'\np6\nS\'undefined\'\np7\nsS\'documentroot\'\np8\nS\'/var/www/\'\np9\nsS\'client_contact\'\np10\ng7\nsS\'serveralias\'\np11\n(lp12\nsa.
It's the pickled view of
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
Can someone help me please ? I spend my afternoon to google to try to find
a solution...
Thanks in advance !!!
Romaric Defaux
After entirely rewrite my code to not use Web service but socket (a real
client/server program) I finally found the problem... And it's not linked
to
the POST or GET method...
It's because of that :
g.db.escape_string(fields['hosted_web_site'])
(escape_string is the function in MySQLdb library)
It escapes the simple quote of the pickled object, and break it...
It's good to know, NEVER escape a pickled object :)
Romaric Defaux
I'm not sure I understand what you're doing here, but I trust you've
read about and understand the security problems with pickle?
Geremy Condra
I read quickly the security problems with pickle. But I don't feel
concern about that because I run my program in a private network, not
over internet. And now I use socket to communicate on a non-standard
port, not anymore web service on the 80 port. If I plan to run it
through wan, I will encrypt datas for sure with SSL or something like
that :)
Romaric Defaux
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to read such file and sumarize the data?
thank you Martin. You are right. But the elapsed time is also okay for me. And i would like to assume that the total CPU time equals to the number of CPUs multiply the elapsed time. As to the number you mentioned, it is the 'process id', so it will be no problem to identify each job. Huiksy On Nov 18, 12:38 am, Martin Gregorie mar...@address-in-sig.invalid wrote: On Wed, 17 Nov 2010 13:45:58 -0800, huisky wrote: Say I have following log file, which records the code usage. I want to read this file and do the summarize how much total CPU time consumed for each user. Two points you should think about: - I don't think you can extract CPU time from this log: you can get the process elapsed time and the number of CPUs each run has used, but you can't calculate CPU time from those values since you don't know how the process spent waiting for i/o etc. - is the first (numeric) part of the first field on the line a process id? If it is, you can match start and stop messages on the value of the first field provided that this value can never be shared by two processes that are both running. If you can get simultaneous duplicates, then you're out of luck because you'll never be able to match up start and stop lines. Is Python able to do so or say easy to achieve this?, anybody can give me some hints, appricate very much! Sure. There are two approaches possible: - sort the log on the first two fields and then process it with Python knowing that start and stop lines will be adjacent - use the first field as the key to an array and put the start time and CPU count in that element. When a matching stop line is found you, retrieve the array element, calculate and output or total the usage figure for that run and delete the array element. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: How to read such file and sumarize the data?
the number before @ is the process id in the linux server and it is identical. So i do NOT think distinguish each job's starting and ending time is difficult in this case. Huisky On Nov 17, 11:38 pm, Tim Harig user...@ilthio.net wrote: On 2010-11-17, huisky hui...@gmail.com wrote: I want to read this file and do the summarize how much total CPU time consumed for each user. Is Python able to do so or say easy to achieve this?, anybody can give me some hints, appricate very much! The question is, is the information you want available in the data. Example log file. ** LSTC license server version 224 started at Sun Dec 6 18:56:48 2009 using configuration file /usr/local/lstc/server_data xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 18:57:40 15...@trofast3.marin.ntnu.no completed Sun Dec 6 19:42:55 xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=2 started Sun Dec 6 20:17:02 15...@trofast3.marin.ntnu.no completed Sun Dec 6 20:26:03 xyz 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 21:01:17 18...@trofast2.marin.ntnu.no completed Sun Dec 6 21:01:28 tanhoi 5...@iimt-tanhoi-w.ivt.ntnu.no LS-DYNA_971 NCPU=1 started Mon Dec 7 09:31:00 5...@iimt-tanhoi-w.ivt.ntnu.no presumed dead Mon Dec 7 10:36:48 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 13:14:47 18...@trofast2.marin.ntnu.no completed Mon Dec 7 13:24:07 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 14:21:34 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 14:28:42 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:31:48 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:32:06 I see starts, completes, kills, and presumed deads. The question is can the starts be matched to the completes and kills either from the numbers before @ or from a combination of the address and NCPU. You will need to figure out whether or not you want to count the presumed deads in your calculations. Assuming that the starts and stops can be corrilated, it is a simple matter of finding the pairs and using the datetime module to find the difference in time between them. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to read such file and sumarize the data?
Thank you very much for your reply. I think you just count the total number of NCPU used for each user. And it does NOT show how much time used for each user. Huisky On Nov 18, 12:10 am, Steve Holden st...@holdenweb.com wrote: On 11/17/2010 4:45 PM, huisky wrote: Say I have following log file, which records the code usage. I want to read this file and do the summarize how much total CPU time consumed for each user. Is Python able to do so or say easy to achieve this?, anybody can give me some hints, appricate very much! Example log file. ** I'm assuming the following (unquoted) data is in file data.txt: LSTC license server version 224 started at Sun Dec 6 18:56:48 2009 using configuration file /usr/local/lstc/server_data xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 18:57:40 15...@trofast3.marin.ntnu.no completed Sun Dec 6 19:42:55 xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=2 started Sun Dec 6 20:17:02 15...@trofast3.marin.ntnu.no completed Sun Dec 6 20:26:03 xyz 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 21:01:17 18...@trofast2.marin.ntnu.no completed Sun Dec 6 21:01:28 tanhoi 5...@iimt-tanhoi-w.ivt.ntnu.no LS-DYNA_971 NCPU=1 started Mon Dec 7 09:31:00 5...@iimt-tanhoi-w.ivt.ntnu.no presumed dead Mon Dec 7 10:36:48 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 13:14:47 18...@trofast2.marin.ntnu.no completed Mon Dec 7 13:24:07 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 14:21:34 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 14:28:42 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:31:48 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:32:06 The line wrapping being wrong shouldn't affect the logic. $ cat data.py lines = open(data.txt).readlines() from collections import defaultdict c = defaultdict(int) for line in lines: ls = line.split() if len(ls) 3 and ls[3].startswith(NCPU=): amt = int(ls[3][5:]) c[ls[0]] += amt for key, value in c.items(): print key, :, value $ python data.py xyz : 4 tanhoi : 1 sabril : 6 regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon 2011 Atlanta March 9-17 http://us.pycon.org/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Pickle in a POST/GET request give EOFError
Thanks for the tip, I'll do that :)
Le 18/11/2010 09:43, Michael Ricordeau a écrit :
Hi,
you can use json for passing list and dict .
Pickle is dangerous .
Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
(using stdlib json in python= 2.6 or simplejson in python 2.6)
Regards
Le Thu, 18 Nov 2010 09:29:00 +0100,
Romaric DEFAUXr...@audaxis.com a écrit :
Le 17/11/2010 18:52, geremy condra a écrit :
On Wed, Nov 17, 2010 at 6:44 AM, Romaric DEFAUXr...@audaxis.com wrote:
Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
Hi everybody !
First time I write to this mailing list :)
I started writing in python last week, that's probably why I can't
understand the following problem...
I create a list called web_site_list.
This list contain dictionaries called web_site.
And some values in this dictionaries are list too.
I do that in a function and I return this :
return pickle.dumps(web_site_list)
This is working fine :)
If I do :
print %s % pickle.loads(system.get_web_site_list())
I've got the right stuffs. For example it returns :
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
I send this to a web service. I send it like that :
#I put it in params
def system_updateweb_site(server, login, password):
params = {}
params['login'] = login
params['password'] = password
params['action'] = 'updateweb_site'
params['servername'] = get_servername()
params['hosted_web_site'] = get_web_site_list()
return call_system_ws(server, params)
#Here's how I send it (I tried in GET and POST)
def call_system_ws(host, params):
query_string = urllib.urlencode(params)
#GET
# f = urllib.urlopen(http://%s/ws?%s; % (host, query_string))
#POST
f = urllib.urlopen(http://%s/ws; % (host), query_string)
result = f.readline().strip()
if result == 'ERROR':
msg = f.readline().strip()
return (False, msg)
return (True, result)
On the server side :
if action == 'updateweb_site':
if not (fields.has_key('servername') and
fields.has_key('hosted_web_site')):
raise WSError('missing parameter :
servername or hosted_web_site')
log ('ERROR : missing parameter :
servername or hosted_web_site')
else:
servername=g.db.escape_string(fields['servername'])
hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
output =
systemserver.updateweb_site(cursor, servername, hosted_web_site)
In systemserver.py :
def updateweb_site(cursor, host, hosted_web_site):
web_site_list = pickle.loads(hosted_web_site)
return %s % (web_site_list)
I catch this error :*
type 'exceptions.EOFError'*:
args = ()
message = ''
Why ?
If I just print hosted_web_site, I get this on my web page :
(lp0\n(dp1\nS\'documentroot_size\'\np2\nS\'120\'\np3\nsS\'servername\'\np4\nS\'default\'\np5\nsS\'client\'\np6\nS\'undefined\'\np7\nsS\'documentroot\'\np8\nS\'/var/www/\'\np9\nsS\'client_contact\'\np10\ng7\nsS\'serveralias\'\np11\n(lp12\nsa.
It's the pickled view of
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
Can someone help me please ? I spend my afternoon to google to try to find
a solution...
Thanks in advance !!!
Romaric Defaux
After entirely rewrite my code to not use Web service but socket (a real
client/server program) I finally found the problem... And it's not linked to
the POST or GET method...
It's because of that :
g.db.escape_string(fields['hosted_web_site'])
(escape_string is the function in MySQLdb library)
It escapes the simple quote of the pickled object, and break it...
It's good to know, NEVER escape a pickled object :)
Romaric Defaux
I'm not sure I understand what you're doing here, but I trust you've
read about and understand the security problems with pickle?
Geremy Condra
I read quickly the security problems with pickle. But I don't feel
concern about that because I run my program in a private network, not
over internet. And now I use socket to communicate on a non-standard
port, not anymore web service on the 80 port. If I plan to run it
through wan, I will encrypt datas for sure with SSL or something like
that :)
Romaric Defaux
smime.p7s
Description: S/MIME Cryptographic Signature
--
http://mail.python.org/mailman/listinfo/python-list
I have a question about JavaFit
Hello, I was searching online to find more info about JavaFit and I came across your information. Can you tell me, are you still involved with JavaFit? If you are, how are things going for you? Please let me know. Sincerely, Bill -- http://mail.python.org/mailman/listinfo/python-list
Re: String multi-replace
Sorin Schwimmer wrote: I have to eliminate diacritics in a fairly large file. Inspired by http://code.activestate.com/recipes/81330/, I came up with the following code: Here's the unicode.translate()-based approach spelt out: http://effbot.org/zone/unicode-convert.htm Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Some syntactic sugar proposals
Steven D'Aprano steve-remove-t...@cybersource.com.au writes:
On Wed, 17 Nov 2010 16:31:40 +, Mark Wooding wrote:
But I don't think that's the big problem with this proposal. The real
problem is that it completely changes the evaluation rule for the
conditional expression. (The evaluation rule is already pretty screwy:
Python is consistently left-to-right -- except here.)
Not quite...
1+2*3
7
(1+2)*3
9
You're wrong. Python evaluates these left-to-right, as I said.
Parentheses override operator associativity; they don't affect
evaluation order at all.
Consider:
def say(x):
print 'seen %s' % x
return x
print say(1) + say(2) * say(3)
print (say(1) + say(2)) * say(3)
Run this program and you get
seen 1
seen 2
seen 3
7
seen 1
seen 2
seen 3
9
So definitely left-to-right. Translating into reverse-Polish, say with
Dijkstra's shunting-yard algorithm, is enlightening: you get
1 2 3 * +
for the first and
1 2 + 3 *
for the second. This preserves evaluation order; indeed, this is a
general property of the shunting-yard algorithm.
Finally, I quote from the language reference (5.13 of the 2.5 version),
just to show that (this time, at least) I'm not trying to impose
unfamiliar terminology, and that Python is defined to behave like this
and I'm not relying on implementation-specific details. Alas, it also
highlights a genuine inconsistency, but one which might be considered
tolerable.
: 5.13 Evaluation order
: =
:
: Python evaluates expressions from left to right. Notice that while
: evaluating an assignment, the right-hand side is evaluated before the
: left-hand side.
:
: In the following lines, expressions will be evaluated in the
: arithmetic order of their suffixes:
:
: expr1, expr2, expr3, expr4
: (expr1, expr2, expr3, expr4)
: {expr1: expr2, expr3: expr4}
: expr1 + expr2 * (expr3 - expr4)
: func(expr1, expr2, *expr3, **expr4)
: expr3, expr4 = expr1, expr2
So the above example is /explicitly/ dealt with in the language
reference, if only you'd cared to look.
Not everything needs to be a one liner. If you need this, do it the old-
fashioned way:
t = foo()
if not pred(t): t = default_value
I already explained how to write it as a one-liner:
t = (lambda y: y if pred(y) else default_value)(foo())
-- [mdw]
--
http://mail.python.org/mailman/listinfo/python-list
Re: I have a question about JavaFit
We are a technical discussion maillinglist. About a computer programming language. Named Python. An unrelated computer programming language which happens to be named Java also exists, but is not this mailinglist's topic of discussion. Some computer programmers personally consume a lot of coffee, but aside from that, as a group, this maillinglist's members have nothing to do with Javalution Coffee Company or their JavaFit coffee products. To put it simply, you're barking up entirely the wrong tree. How exactly were you led to believe that python-list@python.org had anything to do with JavaFit in the first place? Regards, Chris, random maillinglist subscriber On Tue, Nov 16, 2010 at 4:45 PM, Bill Fishbaugher b...@kingofmarketing.net wrote: Hello, I was searching online to find more info about JavaFit and I came across your information. Can you tell me, are you still involved with JavaFit? If you are, how are things going for you? Please let me know. Sincerely, Bill -- http://mail.python.org/mailman/listinfo/python-list
Re: Pickle in a POST/GET request give EOFError
Hi again,
I try simplejson, but I have a problem. (I just replace pickle
everywhere by simplejson)
Here's a test I did :
# python
Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type help, copyright, credits or license for more information.
import simplejson
disk_list = []
disk = {}
disk['name'] = disk1
disk['size'] = 52428800
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}]
disk = {}
disk['name'] = disk2
disk['size'] = 647648
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}, {'name': 'disk2', 'size': '647648'}]
disk_list_json = simplejson.dumps(disk_list)
print disk_list_json
[{name: disk1, size: 52428800}, {name: disk2, size: 647648}]
disk_list2 = simplejson.loads(disk_list_json)
print disk_list2
[{u'name': u'disk1', u'size': u'52428800'}, {u'name': u'disk2', u'size':
u'647648'}]
print repr(disk_list_json)
'[{name: disk1, size: 52428800}, {name: disk2, size:
647648}]'
Explanation :
I dumps with json a list of dictionnaries into disk_list_json.
When I loads it, I don't get my list of dictionnaries like before (see
disk_list2). It adds u letter everywhere.
Why ?
Thanks for help !
Romaric Defaux
Le 18/11/2010 09:43, Michael Ricordeau a écrit :
Hi,
you can use json for passing list and dict .
Pickle is dangerous .
Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
(using stdlib json in python= 2.6 or simplejson in python 2.6)
Regards
Le Thu, 18 Nov 2010 09:29:00 +0100,
Romaric DEFAUXr...@audaxis.com a écrit :
Le 17/11/2010 18:52, geremy condra a écrit :
On Wed, Nov 17, 2010 at 6:44 AM, Romaric DEFAUXr...@audaxis.com wrote:
Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
Hi everybody !
First time I write to this mailing list :)
I started writing in python last week, that's probably why I can't
understand the following problem...
I create a list called web_site_list.
This list contain dictionaries called web_site.
And some values in this dictionaries are list too.
I do that in a function and I return this :
return pickle.dumps(web_site_list)
This is working fine :)
If I do :
print %s % pickle.loads(system.get_web_site_list())
I've got the right stuffs. For example it returns :
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
I send this to a web service. I send it like that :
#I put it in params
def system_updateweb_site(server, login, password):
params = {}
params['login'] = login
params['password'] = password
params['action'] = 'updateweb_site'
params['servername'] = get_servername()
params['hosted_web_site'] = get_web_site_list()
return call_system_ws(server, params)
#Here's how I send it (I tried in GET and POST)
def call_system_ws(host, params):
query_string = urllib.urlencode(params)
#GET
# f = urllib.urlopen(http://%s/ws?%s; % (host, query_string))
#POST
f = urllib.urlopen(http://%s/ws; % (host), query_string)
result = f.readline().strip()
if result == 'ERROR':
msg = f.readline().strip()
return (False, msg)
return (True, result)
On the server side :
if action == 'updateweb_site':
if not (fields.has_key('servername') and
fields.has_key('hosted_web_site')):
raise WSError('missing parameter :
servername or hosted_web_site')
log ('ERROR : missing parameter :
servername or hosted_web_site')
else:
servername=g.db.escape_string(fields['servername'])
hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
output =
systemserver.updateweb_site(cursor, servername, hosted_web_site)
In systemserver.py :
def updateweb_site(cursor, host, hosted_web_site):
web_site_list = pickle.loads(hosted_web_site)
return %s % (web_site_list)
I catch this error :*
type 'exceptions.EOFError'*:
args = ()
message = ''
Why ?
If I just print hosted_web_site, I get this on my web page :
(lp0\n(dp1\nS\'documentroot_size\'\np2\nS\'120\'\np3\nsS\'servername\'\np4\nS\'default\'\np5\nsS\'client\'\np6\nS\'undefined\'\np7\nsS\'documentroot\'\np8\nS\'/var/www/\'\np9\nsS\'client_contact\'\np10\ng7\nsS\'serveralias\'\np11\n(lp12\nsa.
It's the pickled view of
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
Can someone help me please ? I spend my afternoon to google to try to find
a solution...
Thanks in advance !!!
Romaric Defaux
After entirely rewrite my code to not use Web service but socket (a real
Re: Pickle in a POST/GET request give EOFError
When loading from json datas with simplejson, strings are unicode .
Le Thu, 18 Nov 2010 11:22:30 +0100,
Romaric DEFAUX r...@audaxis.com a écrit :
Hi again,
I try simplejson, but I have a problem. (I just replace pickle
everywhere by simplejson)
Here's a test I did :
# python
Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type help, copyright, credits or license for more information.
import simplejson
disk_list = []
disk = {}
disk['name'] = disk1
disk['size'] = 52428800
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}]
disk = {}
disk['name'] = disk2
disk['size'] = 647648
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}, {'name': 'disk2', 'size': '647648'}]
disk_list_json = simplejson.dumps(disk_list)
print disk_list_json
[{name: disk1, size: 52428800}, {name: disk2, size: 647648}]
disk_list2 = simplejson.loads(disk_list_json)
print disk_list2
[{u'name': u'disk1', u'size': u'52428800'}, {u'name': u'disk2', u'size':
u'647648'}]
print repr(disk_list_json)
'[{name: disk1, size: 52428800}, {name: disk2, size:
647648}]'
Explanation :
I dumps with json a list of dictionnaries into disk_list_json.
When I loads it, I don't get my list of dictionnaries like before (see
disk_list2). It adds u letter everywhere.
Why ?
Thanks for help !
Romaric Defaux
Le 18/11/2010 09:43, Michael Ricordeau a écrit :
Hi,
you can use json for passing list and dict .
Pickle is dangerous .
Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
(using stdlib json in python= 2.6 or simplejson in python 2.6)
Regards
Le Thu, 18 Nov 2010 09:29:00 +0100,
Romaric DEFAUXr...@audaxis.com a écrit :
Le 17/11/2010 18:52, geremy condra a écrit :
On Wed, Nov 17, 2010 at 6:44 AM, Romaric DEFAUXr...@audaxis.com wrote:
Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
Hi everybody !
First time I write to this mailing list :)
I started writing in python last week, that's probably why I can't
understand the following problem...
I create a list called web_site_list.
This list contain dictionaries called web_site.
And some values in this dictionaries are list too.
I do that in a function and I return this :
return pickle.dumps(web_site_list)
This is working fine :)
If I do :
print %s % pickle.loads(system.get_web_site_list())
I've got the right stuffs. For example it returns :
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
I send this to a web service. I send it like that :
#I put it in params
def system_updateweb_site(server, login, password):
params = {}
params['login'] = login
params['password'] = password
params['action'] = 'updateweb_site'
params['servername'] = get_servername()
params['hosted_web_site'] = get_web_site_list()
return call_system_ws(server, params)
#Here's how I send it (I tried in GET and POST)
def call_system_ws(host, params):
query_string = urllib.urlencode(params)
#GET
# f = urllib.urlopen(http://%s/ws?%s; % (host, query_string))
#POST
f = urllib.urlopen(http://%s/ws; % (host), query_string)
result = f.readline().strip()
if result == 'ERROR':
msg = f.readline().strip()
return (False, msg)
return (True, result)
On the server side :
if action == 'updateweb_site':
if not (fields.has_key('servername')
and
fields.has_key('hosted_web_site')):
raise WSError('missing
parameter :
servername or hosted_web_site')
log ('ERROR : missing
parameter :
servername or hosted_web_site')
else:
servername=g.db.escape_string(fields['servername'])
hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
output =
systemserver.updateweb_site(cursor, servername, hosted_web_site)
In systemserver.py :
def updateweb_site(cursor, host, hosted_web_site):
web_site_list = pickle.loads(hosted_web_site)
return %s % (web_site_list)
I catch this error :*
type 'exceptions.EOFError'*:
args = ()
message = ''
Why ?
If I just print hosted_web_site, I get this on my web page :
(lp0\n(dp1\nS\'documentroot_size\'\np2\nS\'120\'\np3\nsS\'servername\'\np4\nS\'default\'\np5\nsS\'client\'\np6\nS\'undefined\'\np7\nsS\'documentroot\'\np8\nS\'/var/www/\'\np9\nsS\'client_contact\'\np10\ng7\nsS\'serveralias\'\np11\n(lp12\nsa.
It's the
Re: strange subprocess behavior when calling ps
Roger Davis wrote: Completely off topic but I think the try clause could be rewritten that way: ... Don't use bare except clause, you're masking syntax errors for instance, which will be flagged as 'unexpected error in generation In a more general manner, if something unexpected happens it's better to just let the exception raise uncought. If you want to handle some errors, meaning you're kindof expecting them then add a explicit clause (like you did with KeyboardInterrupt). JM PS : except Exception : will catch most of the exceptions (all inheriting from that class). It's better than using a bare except : clause. (Exception won't catch SyntaxError) Thanks for the suggestion JM, it is off-topic and, although I will first just say that the exception mechanism is *not* one of the reasons I use Python (and stop there with regard to the whole exception mechanism and various usage strategies in general), I do have a few specific questions about a couple of your statements if you don't mind following up. First, inserting a syntax error into my existing code does not hide a SyntaxError exception as you have stated: % cat pid.py #!/usr/bin/python import os import sys import subprocess def main(): psargs= [/bin/ps, -e] try: ps= subprocess.Popen(psargs, stdout=subprocess.PIPE, close_fds=True) psout= ps.communicate()[0] pslines= psout.splitlines() if pslines not good Python talky-talk for line in pslines: print %s % line except KeyboardInterrupt: print Keyboard interrupt received -- terminating. sys.stdout.flush() sys.exit(-1) except: print %s: unexpected error in generation of system process list % prognm sys.stdout.flush() sys.exit(-1) main() % ./pid.py File ./pid.py, line 14 if pslines not good Python talky-talk ^ SyntaxError: invalid syntax It appears that the interpreter is catching the syntax error before the code is even executed. That's right, 'cause my example is approximative. You would need to import that file. A better example would be to fire a NameError exception. In the above code of fix the syntax error, but replace for line in pslines by for line in psLines. note the typo, very common mistake. Then see how the error is hidden by your handler. Second, python.org's exception hierarchy documentation (Section 6.1 at http://docs.python.org/library/exceptions.html) shows all exception types except SystemExit, KeyboardInterrupt and GeneratorExit as being descendants of Exception. This includes SyntaxError, a child of StandardError which is itself a child of Exception. So, if I say 'except Exception:' then isn't that clause going to process any child exception type of Exception (including SyntaxError) that I haven't already covered in a separate except clause? You're absolutely right. Using Exception is bad, using bare except is even worse. Exception is slightly better than except because you can at least get the exception instance that has been raised, print it in your handler, so you don't loose information (well, with a bare except you can still inspect some sys objects to get info about the last exception but it's a bit hackish). (Except of course that my interpreter doesn't seem to treat a syntax error as any kind of exception at all!) Finally, and this does not apply to your comments in particular, in researching around about exception handling I often see the usage except Exception, e: suggested, but can't for the life of me figure out what the heck the ', e' part does. Can anybody explain what this means and why it might be desirable (or not)? Thanks! Exception is the class, e is an instance of that class that has been actually fired, it contains informations, most of the time a message. This syntax has changed in python 3 I think, that would mean that you're right to be confused, this python 2 syntax is ... strange... import sys # what if someone writes sys.hihi ? try: raise AttributeError(sys object has no attribute 'hihi') except Exception, e: print e.args print e.__class__ print e.message print e.__doc__ output: (sys object has no attribute 'hihi',) type 'exceptions.AttributeError' sys object has no attribute 'hihi' Attribute not found. Apart from all these details, just keep in mind this golden rule: - Don't use bare except clause. And you'll be fine. JM -- http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
Tim Harig wrote:
On 2010-11-18, noydb noyd...@gmail.com wrote:
I have an executable that I want to run within python code. The exe
requires an input text file, the user to click a 'compute' button, and
then the exe calculates several output values, one of which I want to
capture into a variable. Can I use Python to supply the input file,
execute the exe and capture the output value, like such that the exe
really doesn't need to be 'seen'? Or, would the user still have to
click the 'compute' button?
Any code snippets or guidance would be very much appreciated. I have
found that
import os
os.system('C:\xTool\stats_hall.exe')
will run the exe. And, maybe these execl/execle/execlp/etc functions
might be what I need for adding in the argument, but documentation
seems to indicate that these do not return output. ??
If you are not already, I would highly suggest using Python3 with the
subprocess module:
http://docs.python.org/py3k/library/subprocess.html
It puts everything in one place and supercedes the exec* functions which
where a PITA. You can 95% of what you need simply using
subprocess.Popen(). There are several examples from this group in the past
few days; but, the process looks something like this:
Python 3.1.2 (r312:79147, Oct 9 2010, 00:16:06)
[GCC 4.4.4] on linux2
Type help, copyright, credits or license for more information.
import subprocess
pig = subprocess.Popen([/usr/games/pig], stdin=subprocess.PIPE)
result = pig.communicate(input=bThis is sample text.\n)
Isthay isway amplesay exttay.
Suggesting subprocess is a good idea, *highly* suggesting python3 is
questionable. The above code works in python 2. Many libraries (those
included batteries) have not been ported yet to python 3.
Py3 is a better core language than py2, but for now, less featured.
JM
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pickle in a POST/GET request give EOFError
Sorry for this stupid question, but how can I convert unicode to string.
I ask that because I hit an error (perhaps a bug) in mod_wsgi, and that
crash the apache process :
TypeError: sequence of string values expected, value of type unicode found
RuntimeError: generator ignored GeneratorExit
child pid 16211 exit signal Segmentation fault (11)
Romaric Defaux
Le 18/11/2010 11:26, Michael Ricordeau a écrit :
When loading from json datas with simplejson, strings are unicode .
Le Thu, 18 Nov 2010 11:22:30 +0100,
Romaric DEFAUXr...@audaxis.com a écrit :
Hi again,
I try simplejson, but I have a problem. (I just replace pickle
everywhere by simplejson)
Here's a test I did :
# python
Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type help, copyright, credits or license for more information.
import simplejson
disk_list = []
disk = {}
disk['name'] = disk1
disk['size'] = 52428800
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}]
disk = {}
disk['name'] = disk2
disk['size'] = 647648
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}, {'name': 'disk2', 'size': '647648'}]
disk_list_json = simplejson.dumps(disk_list)
print disk_list_json
[{name: disk1, size: 52428800}, {name: disk2, size: 647648}]
disk_list2 = simplejson.loads(disk_list_json)
print disk_list2
[{u'name': u'disk1', u'size': u'52428800'}, {u'name': u'disk2', u'size':
u'647648'}]
print repr(disk_list_json)
'[{name: disk1, size: 52428800}, {name: disk2, size:
647648}]'
Explanation :
I dumps with json a list of dictionnaries into disk_list_json.
When I loads it, I don't get my list of dictionnaries like before (see
disk_list2). It adds u letter everywhere.
Why ?
Thanks for help !
Romaric Defaux
Le 18/11/2010 09:43, Michael Ricordeau a écrit :
Hi,
you can use json for passing list and dict .
Pickle is dangerous .
Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
(using stdlib json in python= 2.6 or simplejson in python 2.6)
Regards
Le Thu, 18 Nov 2010 09:29:00 +0100,
Romaric DEFAUXr...@audaxis.com a écrit :
Le 17/11/2010 18:52, geremy condra a écrit :
On Wed, Nov 17, 2010 at 6:44 AM, Romaric DEFAUXr...@audaxis.comwrote:
Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
Hi everybody !
First time I write to this mailing list :)
I started writing in python last week, that's probably why I can't
understand the following problem...
I create a list called web_site_list.
This list contain dictionaries called web_site.
And some values in this dictionaries are list too.
I do that in a function and I return this :
return pickle.dumps(web_site_list)
This is working fine :)
If I do :
print %s % pickle.loads(system.get_web_site_list())
I've got the right stuffs. For example it returns :
[{'documentroot_size': '120', 'servername': '---default---', 'client':
'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
'serveralias': []}]
I send this to a web service. I send it like that :
#I put it in params
def system_updateweb_site(server, login, password):
params = {}
params['login'] = login
params['password'] = password
params['action'] = 'updateweb_site'
params['servername'] = get_servername()
params['hosted_web_site'] = get_web_site_list()
return call_system_ws(server, params)
#Here's how I send it (I tried in GET and POST)
def call_system_ws(host, params):
query_string = urllib.urlencode(params)
#GET
# f = urllib.urlopen(http://%s/ws?%s; % (host, query_string))
#POST
f = urllib.urlopen(http://%s/ws; % (host), query_string)
result = f.readline().strip()
if result == 'ERROR':
msg = f.readline().strip()
return (False, msg)
return (True, result)
On the server side :
if action == 'updateweb_site':
if not (fields.has_key('servername') and
fields.has_key('hosted_web_site')):
raise WSError('missing parameter :
servername or hosted_web_site')
log ('ERROR : missing parameter :
servername or hosted_web_site')
else:
servername=g.db.escape_string(fields['servername'])
hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
output =
systemserver.updateweb_site(cursor, servername, hosted_web_site)
In systemserver.py :
def updateweb_site(cursor, host, hosted_web_site):
web_site_list = pickle.loads(hosted_web_site)
return %s % (web_site_list)
I catch this error :*
type 'exceptions.EOFError'*:
args = ()
message = ''
Why ?
If I just print hosted_web_site, I get this on my
Re: Pickle in a POST/GET request give EOFError
Hi answer to myself :)
With str(unicode_string) !
Le 18/11/2010 12:23, Romaric DEFAUX a écrit :
Sorry for this stupid question, but how can I convert unicode to string.
I ask that because I hit an error (perhaps a bug) in mod_wsgi, and
that crash the apache process :
TypeError: sequence of string values expected, value of type unicode
found
RuntimeError: generator ignored GeneratorExit
child pid 16211 exit signal Segmentation fault (11)
Romaric Defaux
Le 18/11/2010 11:26, Michael Ricordeau a écrit :
When loading from json datas with simplejson, strings are unicode .
Le Thu, 18 Nov 2010 11:22:30 +0100,
Romaric DEFAUXr...@audaxis.com a écrit :
Hi again,
I try simplejson, but I have a problem. (I just replace pickle
everywhere by simplejson)
Here's a test I did :
# python
Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type help, copyright, credits or license for more information.
import simplejson
disk_list = []
disk = {}
disk['name'] = disk1
disk['size'] = 52428800
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}]
disk = {}
disk['name'] = disk2
disk['size'] = 647648
disk_list.append(disk)
print disk_list
[{'name': 'disk1', 'size': '52428800'}, {'name': 'disk2', 'size':
'647648'}]
disk_list_json = simplejson.dumps(disk_list)
print disk_list_json
[{name: disk1, size: 52428800}, {name: disk2, size:
647648}]
disk_list2 = simplejson.loads(disk_list_json)
print disk_list2
[{u'name': u'disk1', u'size': u'52428800'}, {u'name': u'disk2',
u'size':
u'647648'}]
print repr(disk_list_json)
'[{name: disk1, size: 52428800}, {name: disk2, size:
647648}]'
Explanation :
I dumps with json a list of dictionnaries into disk_list_json.
When I loads it, I don't get my list of dictionnaries like before
(see
disk_list2). It adds u letter everywhere.
Why ?
Thanks for help !
Romaric Defaux
Le 18/11/2010 09:43, Michael Ricordeau a écrit :
Hi,
you can use json for passing list and dict .
Pickle is dangerous .
Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
(using stdlib json in python= 2.6 or simplejson in python 2.6)
Regards
Le Thu, 18 Nov 2010 09:29:00 +0100,
Romaric DEFAUXr...@audaxis.com a écrit :
Le 17/11/2010 18:52, geremy condra a écrit :
On Wed, Nov 17, 2010 at 6:44 AM, Romaric
DEFAUXr...@audaxis.comwrote:
Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
Hi everybody !
First time I write to this mailing list :)
I started writing in python last week, that's probably why I can't
understand the following problem...
I create a list called web_site_list.
This list contain dictionaries called web_site.
And some values in this dictionaries are list too.
I do that in a function and I return this :
return pickle.dumps(web_site_list)
This is working fine :)
If I do :
print %s % pickle.loads(system.get_web_site_list())
I've got the right stuffs. For example it returns :
[{'documentroot_size': '120', 'servername': '---default---',
'client':
'undefined', 'documentroot': '/var/www/', 'client_contact':
'undefined',
'serveralias': []}]
I send this to a web service. I send it like that :
#I put it in params
def system_updateweb_site(server, login, password):
params = {}
params['login'] = login
params['password'] = password
params['action'] = 'updateweb_site'
params['servername'] = get_servername()
params['hosted_web_site'] = get_web_site_list()
return call_system_ws(server, params)
#Here's how I send it (I tried in GET and POST)
def call_system_ws(host, params):
query_string = urllib.urlencode(params)
#GET
# f = urllib.urlopen(http://%s/ws?%s; % (host,
query_string))
#POST
f = urllib.urlopen(http://%s/ws; % (host),
query_string)
result = f.readline().strip()
if result == 'ERROR':
msg = f.readline().strip()
return (False, msg)
return (True, result)
On the server side :
if action == 'updateweb_site':
if not
(fields.has_key('servername') and
fields.has_key('hosted_web_site')):
raise
WSError('missing parameter :
servername or hosted_web_site')
log ('ERROR : missing
parameter :
servername or hosted_web_site')
else:
servername=g.db.escape_string(fields['servername'])
hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
output =
systemserver.updateweb_site(cursor, servername, hosted_web_site)
In systemserver.py :
def updateweb_site(cursor, host, hosted_web_site):
web_site_list = pickle.loads(hosted_web_site)
return %s % (web_site_list)
I catch this error :*
type 'exceptions.EOFError'*:
args = ()
message = ''
Why ?
If I just print
Re: Does Pygoogle allows for advanced search options?
The library doesn't seem to have built-in support for filtering by language (and Google lacks a search query-string-based operator for that), but it looks like you could implement that feature by adding an lr parameter with an appropriate value to the query `args` dictionary. See the lr? entry under Web Search Specific Arguments onhttp://code.google.com/apis/websearch/docs/reference.html, and lines 68 102 of pygoogle.py. From those lines, it can be concluded that lr=lang_?? is not supported, unfortunately. And, when I execute, I get an error. Is there any other way to get Google search from Python? I really need only number of hits. Best, PM -- http://mail.python.org/mailman/listinfo/python-list
Re: String multi-replace
On 18/11/2010 04:30, Benjamin Kaplan wrote:
On Wed, Nov 17, 2010 at 11:21 PM, Sorin Schwimmersx...@yahoo.com wrote:
Hi All,
I have to eliminate diacritics in a fairly large file.
Inspired by http://code.activestate.com/recipes/81330/, I came up with the
following code:
#! /usr/bin/env python
import re
nodia={chr(196)+chr(130):'A', # mamaliga
chr(195)+chr(130):'A', # A^
chr(195)+chr(142):'I', # I^
chr(195)+chr(150):'O', # OE
chr(195)+chr(156):'U', # UE
chr(195)+chr(139):'A', # AE
chr(197)+chr(158):'S',
chr(197)+chr(162):'T',
chr(196)+chr(131):'a', # mamaliga
chr(195)+chr(162):'a', # a^
chr(195)+chr(174):'i', # i^
chr(195)+chr(182):'o', # oe
chr(195)+chr(188):'u', # ue
chr(195)+chr(164):'a', # ae
chr(197)+chr(159):'s',
chr(197)+chr(163):'t'
}
name=R\xc3\xa2\xc5\x9fca
regex = re.compile((%s) % |.join(map(re.escape, nodia.keys(
print regex.sub(lambda mo: dict[mo.string[mo.start():mo.end()]], name)
But it won't work; I end up with:
Traceback (most recent call last):
File multirep.py, line 25, inmodule
print regex.sub(lambda mo: dict[mo.string[mo.start():mo.end()]], name)
File multirep.py, line 25, inlambda
print regex.sub(lambda mo: dict[mo.string[mo.start():mo.end()]], name)
TypeError: 'type' object is not subscriptable
What am I doing wrong?
Thanks for your advice,
SxN
dict is a type, not a dict. Your dict is called nodia. I'm guess
that's what you meant to use.
Could I also suggest that you use:
mo.group()
instead of:
mo.string[mo.start():mo.end()]
--
http://mail.python.org/mailman/listinfo/python-list
with HTTPConnection as conn:
Hi! Would it be possible to use a with statement with an HTTPConnection object? I know it is not possible at this moment, it doesn't implement an __exit__ method, at least in version 3.1.1. I was wondering if the __exit__ method should do something more than conn.close(), something really hard or impossible, or if there are more requisites to use the with statement beside of implementing that method. So far it seems easy, useful and not done yet, so I'm probably mistaken about something, and wondering what may that be. Thank you. -- http://mail.python.org/mailman/listinfo/python-list
Re: with HTTPConnection as conn:
On Thu, Nov 18, 2010 at 3:46 AM, trylks try...@gmail.com wrote: Hi! Would it be possible to use a with statement with an HTTPConnection object? Yes, via http://docs.python.org/library/contextlib.html#contextlib.closing from httplib import HTTPConnection from contextlib import closing with closing(HTTPConnection(...)) as conn: # do whatever # connection now closed Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
How to correctly pass “pointer-to-pointer” into DLL via ctypes?
Hello.
I have a DLL that allocates memory and returns it. Function in DLL is like this:
void Foo( unsigned char** ppMem, int* pSize )
{
* pSize = 4;
* ppMem = malloc( * pSize );
for( int i = 0; i * pSize; i ++ ) (* pMem)[ i ] = i;
}
Also, i have a python code that access this function from my DLL:
from ctypes import *
Foo = windll.mydll.Foo
Foo.argtypes = [ POINTER( POINTER( c_ubyte ) ), POINTER( c_int ) ]
mem = POINTER( c_ubyte )()
size = c_int( 0 )
Foo( byref( mem ), byref( size ) ]
print size, mem[ 0 ], mem[ 1 ], mem[ 2 ], mem[ 3 ]
I'm expecting that print will show 4 0 1 2 3 but it shows 4 221 221
221 221 O_O. Any hints what i'm doing wrong?
--
http://mail.python.org/mailman/listinfo/python-list
Re: MATLAB to Python?
On Nov 17, 10:04 am, MATLABdude matlab.d...@mbnet.fi wrote: Thanks guys! I still have some problems. I made the code somewhat simpler by dropping off some parts of it (xx, f). Now my Python code calculates 8 different values for lambda. The problem is that I don't know where the minus sign comes from. Also the values themselves are not identical compared to the values of the MATLAB program. nonhomog.py | nonhomog.m -0.774244159818 | 0.9976 -0.823595831818 | 0.9969 -0.774374006419 | 0.5464 -1.0| -54.3440 -0.774244172803 | 0.9976 -0.774244289666 | 0.9976 -0.823595831818 | 0.9969 -1.0| 0.9674 The new Python code can be found here: http://pastebin.com/c1UvtXxu -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
John Nagle na...@animats.com wrote in message news:4ce37e01$0$1666$742ec...@news.sonic.net... On 11/16/2010 10:24 PM, swapnil wrote: AFAIK, the merging plan was approved by Guido early this year. I guess Google is expecting the community to drive the project from here on. That was the whole idea for merging it to mainline. From my last conversation with Collin, they are targeting Python 3.3 I think it's dead. They're a year behind on quarterly releases. The last release was Q3 2009. The project failed to achieve its stated goal of a 5x speedup. Not even close. More like 1.5x (http://www.python.org/dev/peps/pep-3146) There must have been good reasons to predict a 5x increase. But why did it take so long to find out the approach wasn't going anywhere? Assuming the 5x speedup was shown to be viable (ie. performing the same benchmarks, on the same data, can be done that quickly in any other language, and allowing for the overheads associated with Python's dynamic nature), then what went wrong? (I've had a look at the benchmarks, with a view to trying some on other languages, and they seem an extraordinarily difficult bunch to work with.) The Google blog at http://groups.google.com/group/unladen-swallow/browse_thread/thread/f2011129c4414d04; says, as of November 8, 2010: So Google has pulled the plug on Unladen Swallow. It looks like they underestimated the difficulty of speeding up the CPython model. The performance improvement achieved was so low that cluttering up CPython with a JIT system and LLVM probably is a lose. LLVM. Ok, that explains a lot. (LLVM is a huge, complex system). -- Bartc -- http://mail.python.org/mailman/listinfo/python-list
Re: argparse subparser problem
On 2010-11-16 14:35:01 -0500, Neal Becker said:
I want to have subparsers, but I also want to be able to say:
myprogram --version
and get the version #
Would it not be possible to print the version number and then exit the
program before building the subparser?
---
import argparse
def stop():
pass
parser = argparse.ArgumentParser()
parser.add_argument ('--version', action='store_true')
# if --version print version
# sys.exit(0)
subparsers = parser.add_subparsers()
parser_stop = subparsers.add_parser ('stop')
parser_stop.add_argument ('stop', action='store_true')
parser_stop.set_defaults (func=stop)
opt = parser.parse_args (['--version'])
--
--
./Sven
--
http://mail.python.org/mailman/listinfo/python-list
Re: MATLAB to Python?
On 18/11/2010 12:16, MATLABdude wrote: On Nov 17, 10:04 am, MATLABdudematlab.d...@mbnet.fi wrote: Thanks guys! I still have some problems. I made the code somewhat simpler by dropping off some parts of it (xx, f). Now my Python code calculates 8 different values for lambda. The problem is that I don't know where the minus sign comes from. Also the values themselves are not identical compared to the values of the MATLAB program. nonhomog.py | nonhomog.m -0.774244159818 | 0.9976 -0.823595831818 | 0.9969 -0.774374006419 | 0.5464 -1.0| -54.3440 -0.774244172803 | 0.9976 -0.774244289666 | 0.9976 -0.823595831818 | 0.9969 -1.0| 0.9674 The new Python code can be found here: http://pastebin.com/c1UvtXxu Don't use: from math import * from numpy import * from scipy import * because you'll never be sure where a name comes from! Also, check for accidental integer division. (In Python 2 integer / integer returns integer.) -- http://mail.python.org/mailman/listinfo/python-list
Re: Pickle in a POST/GET request give EOFError
On Thu, 2010-11-18 at 11:22 +0100, Romaric DEFAUX wrote: Hi again, I try simplejson, but I have a problem. (I just replace pickle everywhere by simplejson) Explanation : I dumps with json a list of dictionnaries into disk_list_json. When I loads it, I don't get my list of dictionnaries like before (see disk_list2). It adds u letter everywhere. Correct; I strongly disagree with the assertion to use JSON over Pickle. Pickle is superior for serialization - JSON does not preserve type. Use Pickle when possible. JSON is best for communicating simple data to stupid clients [web browsers]. JSON doesn't know about dates, it doesn't know about string vs. unicode, etc... -- Adam Tauno Williams awill...@whitemice.org -- http://mail.python.org/mailman/listinfo/python-list
Re: What was your strategy?
In article mailman.1048.1289936253.2218.python-l...@python.org, Steve Holden st...@holdenweb.com wrote: On 11/16/2010 2:22 PM, Lou Pecora wrote: I'll jump in and recommend the book Python in a Nutshell by Martelli. It may be a little dated now, but it covers many Python topics in good detail without becoming a bloated reference. Nicely written. It's still the first book I reach for after 6 years of Python coding and it rarely disappoints. +1 It's encyclopedic. Indeed. I hope Martelli updates it. I'd buy another copy right away. -- -- Lou Pecora -- http://mail.python.org/mailman/listinfo/python-list
Re: String multi-replace
On 2:59 PM, Sorin Schwimmer wrote: Steven D'Aprano: the original file is 139MB (that's the typical size for it). Eliminating diacritics is just a little toping on the cake; the processing is something else. Thanks anyway for your suggestion, SxN PS Perhaps I should have mention that I'm on Python 2.7 In the message you were replying to, Steven had a much more important suggestion to make than the size one, and you apparently didn't notice it. Chris made a similar implication. I'll try a third time. The file is obviously encoded, and you know the encoding. Judging from the first entry in your table, it's in utf-8. If so, then your approach is all wrong. Treating it as a pile of bytes, and replacing pairs is likely to get you in trouble, since it's quite possible that you may get a match with the last byte of one character and the first byte of another one. If you substitute such a match, you'll make a hash of the whole region, and quite likely end up with a byte stream that is no longer even utf-8. Fortunately, you can solve that problem, and simplify your code greatly in the bargain, by doing something like what was suggested by Steven. Change your map of encoded bytes into unicode_nodia, using decode(utf-8) on the keys, and u on the values Read in each line of the file, decode it to the unicode it represents, and do a simple translate once it's valid unicode. Assuming the line is in utf-8, use uni = line.decode(utf-8) newuni = uni.trans(unicode_nodia) newutf8 = newuni.encode(utf-8) incidentally, to see what a given byte pair in your table is, you can do something like: import unicodedata a = chr(196)+chr(130) unicodedata.name(a.decode(utf-8)) 'LATIN CAPITAL LETTER A WITH BREVE' DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Thu, 18 Nov 2010 01:16:34 +, MRAB wrote: I'd probably say that a script is a program which is normally not interactive: you just set it up, start it, and let it do its work (a batch script, for example). It's also written in a language primarily designed for convenience rather than speed (Want to manipulate large chunks of text? Fine! :-)). I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: What was your strategy?
+2 I also agree. - Braden Faulkner -Original Message- From: Lou Pecora pec...@anvil.nrl.navy.mil Sender: python-list-bounces+brf256=gmail@python.org Date: Thu, 18 Nov 2010 09:05:59 To: python-list@python.org Subject: Re: What was your strategy? In article mailman.1048.1289936253.2218.python-l...@python.org, Steve Holden st...@holdenweb.com wrote: On 11/16/2010 2:22 PM, Lou Pecora wrote: I'll jump in and recommend the book Python in a Nutshell by Martelli. It may be a little dated now, but it covers many Python topics in good detail without becoming a bloated reference. Nicely written. It's still the first book I reach for after 6 years of Python coding and it rarely disappoints. +1 It's encyclopedic. Indeed. I hope Martelli updates it. I'd buy another copy right away. -- -- Lou Pecora -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Twisted on Windows
Hello All, First off I must say that Twisted is a very nice event driven I/O package indeed. Thanks to all the developers who have contributed to it, as it's made my life much easier. Now for my question... I have a custom server application, and I have it structured as such: MyServerApp/ -- root directory server.py foo/ __init__.py factory.py In my server.py file, which is a Twistd config file that can be executed with twistd -ny server.py, I have the following import line: from foo.factory import MyServerFactory When I run 'twistd -ny server.py' on my Linux machine from within the root MyServerApp directory, all works as expected. However, when I try to do the same thing on a Windows machine using the twistd script that gets installed, I get an error saying that foo.factory cannot be found. However, if I modify my server.py file on Windows to just create the factory and start the reactor, I can run it just fine with 'python server.py' even though it still has the import line for MyServerFactory. Bottom line is, on Windows, the python executable can find my custom module in a sub directory but the twistd script cannot. Any ideas why this is? -- Thanks! Bryan -- http://mail.python.org/mailman/listinfo/python-list
Re: with HTTPConnection as conn:
On Thu, 18 Nov 2010 12:46:07 +0100 trylks try...@gmail.com wrote: Hi! Would it be possible to use a with statement with an HTTPConnection object? I know it is not possible at this moment, it doesn't implement an __exit__ method, at least in version 3.1.1. I was wondering if the __exit__ method should do something more than conn.close(), something really hard or impossible, or if there are more requisites to use the with statement beside of implementing that method. So far it seems easy, useful and not done yet, so I'm probably mistaken about something, and wondering what may that be. It's possibly one of those things that haven't been implemented simply because no one thought about it yet, or bothered enough to do it. In other words, you can open a feature request at http://bugs.python.org, and even provide a patch. Regards Antoine. -- http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
Hi JM, Thank you very much for your followup explanation! Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On 2010-11-18, Martin Gregorie mar...@address-in-sig.invalid wrote: On Thu, 18 Nov 2010 01:16:34 +, MRAB wrote: I'd probably say that a script is a program which is normally not interactive: you just set it up, start it, and let it do its work (a batch script, for example). It's also written in a language primarily designed for convenience rather than speed (Want to manipulate large chunks of text? Fine! :-)). I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. I use 'script' to refer to programs that are used to automate things that would otherwise be done by a person typing commands. IOW, what I write using Python aren't scripts. They're programs. -- Grant Edwards grant.b.edwardsYow! All of life is a blur at of Republicans and meat! gmail.com -- http://mail.python.org/mailman/listinfo/python-list
Re: with HTTPConnection as conn:
Antoine Pitrou solip...@pitrou.net writes: On Thu, 18 Nov 2010 12:46:07 +0100 trylks try...@gmail.com wrote: Hi! Would it be possible to use a with statement with an HTTPConnection object? I know it is not possible at this moment, it doesn't implement an __exit__ method, at least in version 3.1.1. I was wondering if the __exit__ method should do something more than conn.close(), something really hard or impossible, or if there are more requisites to use the with statement beside of implementing that method. So far it seems easy, useful and not done yet, so I'm probably mistaken about something, and wondering what may that be. It's possibly one of those things that haven't been implemented simply because no one thought about it yet, or bothered enough to do it. Also, contextlib.closing is readily available: with contextlib.closing(HTTPConnection(...)) as conn: ... -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Thu, 18 Nov 2010 15:41:51 +, Grant Edwards wrote: On 2010-11-18, Martin Gregorie mar...@address-in-sig.invalid wrote: On Thu, 18 Nov 2010 01:16:34 +, MRAB wrote: I'd probably say that a script is a program which is normally not interactive: you just set it up, start it, and let it do its work (a batch script, for example). It's also written in a language primarily designed for convenience rather than speed (Want to manipulate large chunks of text? Fine! :-)). I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. I use 'script' to refer to programs that are used to automate things that would otherwise be done by a person typing commands. Its a long-standing UNIXism - it applies to any language implementation where you can add #!/path/to/interpreter as the first line of the source file, make the file executable with chmod and run it by treating the file name as a command. IOW, what I write using Python aren't scripts. They're programs. Anything with one or more executable lines is a program: #!/usr/bin/python print \nHello, Python World\n -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
what's the precision of fractions.Fraction?
I do a recursive evaluation of an expression involving fractions and unsurprisingly the numerator and denominator grows pretty quickly. After 10-20 iterations the number of digits in the numerator and denominator (as integers) reaches 80-100. And I'm wondering until what point I can trust the result since I'm using fractions.Fraction for the whole procedure. Are Fraction's infinite precision? Or will I get some sort of an exception if python is not able to represent the numerator and/or denominator as integers? http://www.python.org/dev/peps/pep-3141/ was not very helpful in this regard nor was http://docs.python.org/library/fractions.html Any ideas? Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown -- http://mail.python.org/mailman/listinfo/python-list
Distribution of Python Scripts
Hi Folks - I'm an experienced programmer, but this is my first app with python, so I apologize for any stupidity on my part. So I've written/still working on a command line tool written in Python. It talks to a web service, so there really isn't anything in it that is dependent on a particular os. My goal is to distribute the script as a tool to my users so they can use it (they are on multiple platforms), the vast majority are not programmers. So, I'm looking for a way to distribute the app that takes care of details like them having python installed, or having an older version etc. From reading some docs it sounds like Squeeze did exactly this back in the day, but it is ancient and I'm writing in 2.7. Just to be clear I could care less if they see the source code, my goal is all around ease of use. I would love to have one distribution file for all platforms, but I can live with making three if that's the only option. So, what's my options. -- http://mail.python.org/mailman/listinfo/python-list
Re: Twisted on Windows
On Nov 18, 9:58 am, Bryan Richardson btri...@gmail.com wrote: Hello All, First off I must say that Twisted is a very nice event driven I/O package indeed. Thanks to all the developers who have contributed to it, as it's made my life much easier. Now for my question... I have a custom server application, and I have it structured as such: MyServerApp/ -- root directory server.py foo/ __init__.py factory.py In my server.py file, which is a Twistd config file that can be executed with twistd -ny server.py, I have the following import line: from foo.factory import MyServerFactory When I run 'twistd -ny server.py' on my Linux machine from within the root MyServerApp directory, all works as expected. However, when I try to do the same thing on a Windows machine using the twistd script that gets installed, I get an error saying that foo.factory cannot be found. However, if I modify my server.py file on Windows to just create the factory and start the reactor, I can run it just fine with 'python server.py' even though it still has the import line for MyServerFactory. Bottom line is, on Windows, the python executable can find my custom module in a sub directory but the twistd script cannot. Any ideas why this is? When you run a .py file, python adds the directory containing that .py file to the front of sys.path. So when you run server.py, MyServerApp/ is added to sys.path and the foo package can be found. This happens on Linux and Windows. When you run twistd, the .py file you're running is /usr/bin/twistd or C:\Python26\Scripts\twistd or something else along those lines. So Python adds /usr/bin or C:\Python26\Scripts to sys.path. This doesn't help you find the foo package at all. On Linux, when not running as root, twistd adds the current working directory to sys.path. So if your working directory is MyServerApp, then the foo package can be found. When running as root, or when running on Windows, twistd does not add the working directory to sys.path. So with all that in mind, the solution should be pretty clear - just set PYTHONPATH to include MyServerApp. This variation of twistd behavior is pretty confusing, and I think a future version may drop the sys.path manipulation entirely, so that it behaves consistently in all configurations. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: what's the precision of fractions.Fraction?
On Thu, Nov 18, 2010 at 9:27 AM, Daniel Fetchinson fetchin...@googlemail.com wrote: I do a recursive evaluation of an expression involving fractions and unsurprisingly the numerator and denominator grows pretty quickly. After 10-20 iterations the number of digits in the numerator and denominator (as integers) reaches 80-100. And I'm wondering until what point I can trust the result since I'm using fractions.Fraction for the whole procedure. Are Fraction's infinite precision? Or will I get some sort of an exception if python is not able to represent the numerator and/or denominator as integers? Since fractions are represented as a pair of integers, and since python integers have unlimited precision, I would guess that fractions also have unlimited precision. You could check the code if you want to be absolutely sure. Cheers, Ian -- http://mail.python.org/mailman/listinfo/python-list
ANN: GMPY 1.14 and GMPY2 alpha1 released
Everyone, I'm pleased to annouce the release of both a new production and experimental release of GMPY. GMPY is a wrapper for the MPIR or GMP multiple-precision arithmetic library. The experimental release (GMPY2) now includes support for the MPFR floating-point library. GMPY is available for download from: http://code.google.com/p/gmpy/ Production release -- GMPY 1.14 is the updated stable release. A memory leak was fixed so it is highly recommended that all user upgrade to this version. In addition to a few other bug fixes, GMPY 1.14 is compatible with the changes to the hashing code in Python 3.2a4. The 64-bit Windows installer for Python 3.2 should only be used with 3.2a4 and later. Even though my primary development focus has shifted to GMPY2 (see below), GMPY 1.X will continue to receive bug and compatibility fixes. Experimental release To simplify the codebase, allow for changes in the API, and support simultaneous installation, the development version has been renamed to GMPY2. The following is list of changes in GMPY2: In 2.0.0a0 -- * support for a mutable integer type xmpz * removal of random number functions * xmpz supports slices for setting/clearing bits * some methods have been renamed (scan1 - bit_scan1) * support for Python prior to 2.6 has been removed * support for all division modes has been added * ceiling - round to +Infinity * floor - round to -Infinity * truncate - round to zero * 2exp - division by a power of 2 * support is_even() and is_odd() In 2.0.0a1 -- * support for the MPFR floating point library If you use GMPY regularly, please test GMPY2. There have been several requests asking for a mutable integer and I am curious if there are real-world performance improvements. Please report any issues! casevh -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On Thu, Nov 18, 2010 at 4:24 AM, BartC b...@freeuk.com wrote: John Nagle na...@animats.com wrote in message news:4ce37e01$0$1666$742ec...@news.sonic.net... On 11/16/2010 10:24 PM, swapnil wrote: AFAIK, the merging plan was approved by Guido early this year. I guess Google is expecting the community to drive the project from here on. That was the whole idea for merging it to mainline. From my last conversation with Collin, they are targeting Python 3.3 I think it's dead. They're a year behind on quarterly releases. The last release was Q3 2009. The project failed to achieve its stated goal of a 5x speedup. Not even close. More like 1.5x (http://www.python.org/dev/peps/pep-3146) There must have been good reasons to predict a 5x increase. But why did it take so long to find out the approach wasn't going anywhere? It's not that it wasn't going anywhere. Improvements were accomplished. Also, LLVM needed to be modified more than a little bit for the purpose. Also, the developers were constrained a bit by the need to preserve compatibility with some pretty weird code - like methods being replaced mid-loop. It's not a small project. -- http://mail.python.org/mailman/listinfo/python-list
Re: regenerating unicodedata for py2.7 using py3 makeunicodedata.py?
Thanks for the confirmation Martin! Do you think, it the mentioned omission of the character names of some CJK ranges in unicodedata intended, or should it be reported to the tracker? It's certainly a bug. So a bug report would be appreciated, but much more so a patch. Ideally, the patch would either be completely forward-compatible (should the CJK ranges change in future Unicode versions), or at least have a safe-guard to detect that the data file is getting out of sync with the C implementation. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
I do hope that it isn't dead but it unfortunately seems that way? - Braden Faulkner -- http://mail.python.org/mailman/listinfo/python-list
Re: What was your strategy?
Lou Pecora pec...@anvil.nrl.navy.mil writes: I'll jump in and recommend the book Python in a Nutshell by Martelli. It's encyclopedic. Indeed. I hope Martelli updates it. I'd buy another copy right away. It's a great book but not a starting point for beginners. It's definitely worth having for more advanced users. -- http://mail.python.org/mailman/listinfo/python-list
Re: what's the precision of fractions.Fraction?
I do a recursive evaluation of an expression involving fractions and unsurprisingly the numerator and denominator grows pretty quickly. After 10-20 iterations the number of digits in the numerator and denominator (as integers) reaches 80-100. And I'm wondering until what point I can trust the result since I'm using fractions.Fraction for the whole procedure. Are Fraction's infinite precision? Or will I get some sort of an exception if python is not able to represent the numerator and/or denominator as integers? Since fractions are represented as a pair of integers, and since python integers have unlimited precision, I would guess that fractions also have unlimited precision. You could check the code if you want to be absolutely sure. Sounds reasonable, thanks a lot, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On 11/18/2010 4:24 AM, BartC wrote: John Nagle na...@animats.com wrote in message news:4ce37e01$0$1666$742ec...@news.sonic.net... On 11/16/2010 10:24 PM, swapnil wrote: AFAIK, the merging plan was approved by Guido early this year. I guess Google is expecting the community to drive the project from here on. That was the whole idea for merging it to mainline. From my last conversation with Collin, they are targeting Python 3.3 I think it's dead. They're a year behind on quarterly releases. The last release was Q3 2009. The project failed to achieve its stated goal of a 5x speedup. Not even close. More like 1.5x (http://www.python.org/dev/peps/pep-3146) There must have been good reasons to predict a 5x increase. For Java, adding a JIT improved performance by much more than that. Hard-code compilers for LISP have done much better than 5x. The best Java and LISP compilers approach the speed of C, while CPython is generally considered to be roughly 60 times slower than C. So 5x probably looked like a conservative goal. For Google, a company which buys servers by the acre, a 5x speedup would have a big payoff. Assuming the 5x speedup was shown to be viable (ie. performing the same benchmarks, on the same data, can be done that quickly in any other language, and allowing for the overheads associated with Python's dynamic nature), then what went wrong? Python is defined by what a naive interpreter with late binding and dynamic name lookups, like CPython, can easily implement. Simply emulating the semantics of CPython with generated code doesn't help all that much. Because you can monkey patch Python objects from outside the class, a local compiler, like a JIT, can't turn name lookups into hard bindings. Nor can it make reliable decisions about the types of objects. That adds a sizable performance penalty. Short of global program analysis, the compiler can't tell when code for the hard cases needs to be generated. So the hard-case code, where you figure out at run-time, for ever use of +, whether + is addition or concatenation, has to be generated every time. Making that decision is far slower than doing an add. Shed Skin, which analyzes the entire program, including libraries, on every compilation, can figure out the types of objects and generate much faster code. Shed Skin has some heavy restrictions, many of which could be lifted if more work went into that effort. That's one approach that might work. I've referred to this problem as gratuitous hidden dynamism. Most things which could be changed dynamically in a Python program usually aren't. This has been pointed out many times by many people. There's even a PhD thesis on the topic. Without a few restrictions, so that a compiler can at least tell when support for the hard cases is needed, Python cannot be compiled well. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: what's the precision of fractions.Fraction?
Am 18.11.2010 17:27, schrieb Daniel Fetchinson:
I do a recursive evaluation of an expression involving fractions and
unsurprisingly the numerator and denominator grows pretty quickly.
After 10-20 iterations the number of digits in the numerator and
denominator (as integers) reaches 80-100. And I'm wondering until what
point I can trust the result since I'm using fractions.Fraction for
the whole procedure. Are Fraction's infinite precision? Or will I get
some sort of an exception if python is not able to represent the
numerator and/or denominator as integers?
http://www.python.org/dev/peps/pep-3141/ was not very helpful in this
regard nor was http://docs.python.org/library/fractions.html
Any ideas?
Cheers,
Daniel
I'm not sure what you really want to know.
Fractions are, by their math definition, of unlimited precision.
For example, PI/2 is accurate (with endless precision as PI is a
trancendent number),
but 3.1415926/2 (the decimal representation) is not.
So is fraction.Fraction(1,3) of unlimited precision, thus giving this:
from fractions import Fraction
one_tenth=Fraction(1,10)
a = one_tenth+one_tenth
a
Fraction(1, 5)
a = one_tenth+one_tenth+one_tenth
a
Fraction(3, 10)
now compare to standard floats:
one_tenth=1.0/10
a = one_tenth+one_tenth
a
0.2
a = one_tenth+one_tenth+one_tenth
a
0.30004
(this error is caused by the internal representation of floats as binary
numbers, see No. 14 in Python tut).
I think you really want to know, how large numerator/denominator can be.
As python has bignum support (type=long), the only limit is the one your
virtual memory system
offers, which is almost always approx. the amount of ram.
If you need it really *precise*, cast your Fractions into type Decimal:
from decimal import Decimal
one_tenth=Fraction(1,10)
three_tenth = one_tenth*3
d = Decimal(three_tenth.numerator)/Decimal(three_tenth.denominator)
d
Decimal('0.3')
The operator '/' (__div__) is properly over-written.
You can even tweak the precision getcontext().prec by setting it to a
higher value.
Cheers
attachment: stefan_sonnenberg.vcf--
http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On 18/11/2010 17:50, brf...@gmail.com wrote: I do hope that it isn't dead but it unfortunately seems that way? - Braden Faulkner It looks like we'll have to get used to a laden one. Maybe if we had two, and a strand of creeper... -- http://mail.python.org/mailman/listinfo/python-list
Re: I have a question about JavaFit
On 18/11/2010 10:05, Chris Rebert wrote: We are a technical discussion maillinglist. About a computer programming language. Named Python. An unrelated computer programming language which happens to be named Java also exists, but is not this mailinglist's topic of discussion. Some computer programmers personally consume a lot of coffee, but aside from that, as a group, this maillinglist's members have nothing to do with Javalution Coffee Company or their JavaFit coffee products. To put it simply, you're barking up entirely the wrong tree. How exactly were you led to believe that python-list@python.org had anything to do with JavaFit in the first place? Regards, Chris, random maillinglist subscriber On Tue, Nov 16, 2010 at 4:45 PM, Bill Fishbaugher b...@kingofmarketing.net wrote: Hello, I was searching online to find more info about JavaFit and I came across your information. Can you tell me, are you still involved with JavaFit? If you are, how are things going for you? Please let me know. Sincerely, Bill Is a JavaFit what you have when you drink waay too much coffee? -- http://mail.python.org/mailman/listinfo/python-list
Re: I have a question about JavaFit
Am 18.11.2010 20:17, schrieb Ben James: On 18/11/2010 10:05, Chris Rebert wrote: We are a technical discussion maillinglist. About a computer programming language. Named Python. An unrelated computer programming language which happens to be named Java also exists, but is not this mailinglist's topic of discussion. Some computer programmers personally consume a lot of coffee, but aside from that, as a group, this maillinglist's members have nothing to do with Javalution Coffee Company or their JavaFit coffee products. To put it simply, you're barking up entirely the wrong tree. How exactly were you led to believe that python-list@python.org had anything to do with JavaFit in the first place? Regards, Chris, random maillinglist subscriber On Tue, Nov 16, 2010 at 4:45 PM, Bill Fishbaugher b...@kingofmarketing.net wrote: Hello, I was searching online to find more info about JavaFit and I came across your information. Can you tell me, are you still involved with JavaFit? If you are, how are things going for you? Please let me know. Sincerely, Bill Is a JavaFit what you have when you drink waay too much coffee? Java FIT : Java Framework for integrated testing http://fit.c2.com/ Last update to site Oct 2007 There seems (ed) to be a python port. Status unknown. I guess what bill is looking for is test driven development. This thingy seems really strange. attachment: stefan_sonnenberg.vcf-- http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
I will use 2.5. I tried your suggestion, with this code import subprocess pig = subprocess.Popen([C:\Halls\hallbig2.exe], stdin=subprocess.PIPE, stdout=subprocess.PIPE) result = pig.communicate(input='C:\Halls\Input\Ea39j.txt')[-1] #I need to capture the, what I think is the, last output print result print pig.returncode None 0 So the tuple is empty. ?? The exe executes fine and returns output in th exe tool itself. The python script seems to execute fine, no errors, '...returned exit code 0'. Any ideas/suggestions? -- http://mail.python.org/mailman/listinfo/python-list
Changing the EAX register with Python
Hi folks, I have a unusual question here. How can I change the value of EAX register under python under Linux?? As paimei does under Windows. My project is to have a python program that loads a C program and sets a breakpoint at some address, and then with this breakpoint I change the EAX register and then continue the program execution. With Windows and paimei I did that, but under Linux I don't know yet. Any ideas? Thank you -- http://mail.python.org/mailman/listinfo/python-list
Re: Twisted on Windows
Thanks Jean-Paul, I added the following to my server.py file and things work perfectly on both Windows and Linux now. sys.path.insert(0, os.getcwd()) On Thu, Nov 18, 2010 at 9:52 AM, Jean-Paul Calderone calderone.jeanp...@gmail.com wrote: On Nov 18, 9:58 am, Bryan Richardson btri...@gmail.com wrote: Hello All, First off I must say that Twisted is a very nice event driven I/O package indeed. Thanks to all the developers who have contributed to it, as it's made my life much easier. Now for my question... I have a custom server application, and I have it structured as such: MyServerApp/ -- root directory server.py foo/ __init__.py factory.py In my server.py file, which is a Twistd config file that can be executed with twistd -ny server.py, I have the following import line: from foo.factory import MyServerFactory When I run 'twistd -ny server.py' on my Linux machine from within the root MyServerApp directory, all works as expected. However, when I try to do the same thing on a Windows machine using the twistd script that gets installed, I get an error saying that foo.factory cannot be found. However, if I modify my server.py file on Windows to just create the factory and start the reactor, I can run it just fine with 'python server.py' even though it still has the import line for MyServerFactory. Bottom line is, on Windows, the python executable can find my custom module in a sub directory but the twistd script cannot. Any ideas why this is? When you run a .py file, python adds the directory containing that .py file to the front of sys.path. So when you run server.py, MyServerApp/ is added to sys.path and the foo package can be found. This happens on Linux and Windows. When you run twistd, the .py file you're running is /usr/bin/twistd or C:\Python26\Scripts\twistd or something else along those lines. So Python adds /usr/bin or C:\Python26\Scripts to sys.path. This doesn't help you find the foo package at all. On Linux, when not running as root, twistd adds the current working directory to sys.path. So if your working directory is MyServerApp, then the foo package can be found. When running as root, or when running on Windows, twistd does not add the working directory to sys.path. So with all that in mind, the solution should be pretty clear - just set PYTHONPATH to include MyServerApp. This variation of twistd behavior is pretty confusing, and I think a future version may drop the sys.path manipulation entirely, so that it behaves consistently in all configurations. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: what's the precision of fractions.Fraction?
On 11/18/2010 11:27 AM, Daniel Fetchinson wrote: I do a recursive evaluation of an expression involving fractions and unsurprisingly the numerator and denominator grows pretty quickly. After 10-20 iterations the number of digits in the numerator and denominator (as integers) reaches 80-100. And I'm wondering until what point I can trust the result since I'm using fractions.Fraction for the whole procedure. Are Fraction's infinite precision? 'Unbounded' (except by memory) would be be more accurate. Or will I get some sort of an exception if python is not able to represent the numerator and/or denominator as integers? I believe you could eventually get MemoryError. But you might get tired of waiting first. I tried i=0 n=2 while True: print(i) n *= n i += 1 The 30th doubling (I believe to a billion bit or 128 megabyte number, on a gigabyte machine) took a few minutes and I killed the program after that as it tried to page everything else out to disk. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: String multi-replace
On Wed, 2010-11-17 at 21:12 -0800, Sorin Schwimmer wrote:
Thanks for your answers.
Benjamin Kaplan: of course dict is a type... silly me! I'll blame it on the
time (it's midnight here).
Chris Rebert: I'll have a look.
Thank you both,
SxN
Forgive me if this is off the track. I haven't followed the thread. I do
have a little module that I believe does what you attempted to do:
multiple substitutions using a regular expression that joins a bunch of
targets with '|' in between. Whether or not you risk unintended
translations as Dave Angel pointed out where the two characters or one
of your targets join coincidentally you will have to determine. If so
you can't use this approach. If, on the other hand, your format is safe
it'll work just fine. Use like this:
import translator
t = translator.Translator (nodia.items ())
t (name) # Your example
'Rasca'
Frederic
class Translator:
Will translate any number of targets, handling them correctly if some overlap.
Making Translator
T = Translator (definitions, [eat = 1])
'definitions' is a sequence of pairs: ((target, substitute),(t2, s2), ...)
'eat = True' will make an extraction filter that lets only the replaced targets pass.
Definitions example: (('a','A'),('b','B'),('ab','ab'),('abc','xyz'),
('\x0c', 'page break'), ('\r\n','\n'), (' ','\t')) # ('ab','ab') see Tricks.
Order doesn't matter.
Testing
T.test (). Translates the definitions and prints the result. All targets
must look like the substitutes as defined. If a substitute differs, it has been
affected by the translation. (E.g. 'A'|'A' ... 'page break'|'pAge BreAk').
If this is not intended---the effect can be useful---protect the
affected substitute by translating it to itself. See Tricks.
Running
translation = T (source)
Tricks
Deletion: ('target', '')
Exception: (('\n',''), ('\n\n','\n\n')) # Eat LF except paragraph breaks.
Exception: (('\n', '\r\n'), ('\r\n',\r\n')) # Unix to DOS, would leave DOS unchanged
Translation cascade:
# Unwrap paragraphs, Unix or DOS, restoring inter-word space if missing,
Mark_LF = Translator ((('\n','+LF+'),('\r\n','+LF+'),('\n\n','\n\n'),('\r\n\r\n','\r\n\r\n')))
# Pick positively identifiable mark for end of lines in either Unix or MS-DOS.
Single_Space_Mark = Translator (((' +LF+', ' '),('+LF+', ' '),('-+LF+', '')))
no_lf_text = Single_Space_Mark (Mark_LF (text))
Translation cascade:
reptiles = T_latin_english (T_german_latin (reptilien))
Limitations
1. The number of substitutions and the maximum size of input depends on the respective
capabilities of the Python re module.
2. Regular expressions will not work as such but will be handled literally.
Author:
Frederic Rentsch (i...@anthra-norell.ch).
def __init__ (self, definitions, eat = 0):
'''
definitions: a sequence of pairs of strings. ((target, substitute), (t, s), ...)
eat: False (0) means translate: unaffected data passes unaltered.
True (1) means extract: unaffected data doesn't pass (gets eaten).
Extraction filters typically require substitutes to end with some separator,
else they fuse together. (E.g. ' ', '\t' or '\n')
'eat' is an attribute that can be switched anytime.
'''
self.eat = eat
self.compile_sequence_of_pairs (definitions)
def compile_sequence_of_pairs (self, definitions):
'''
Argument 'definitions' is a sequence of pairs:
(('target 1', 'substitute 1'), ('t2', 's2'), ...)
Order doesn't matter.
'''
import re
self.definitions = definitions
targets, substitutes = zip (*definitions)
re_targets = [re.escape (item) for item in targets]
re_targets.sort (reverse = True)
self.targets_set = set (targets)
self.table = dict (definitions)
regex_string = '|'.join (re_targets)
self.regex = re.compile (regex_string, re.DOTALL)
def __call__ (self, s):
hits = self.regex.findall (s)
nohits = self.regex.split (s)
valid_hits = set (hits) self.targets_set # Ignore targets with illegal re modifiers.
if valid_hits:
substitutes = [self.table [item] for item in hits if item in valid_hits] + [] # Make lengths equal for zip to work right
if self.eat:
return ''.join (substitutes)
else:
zipped = zip (nohits, substitutes)
return ''.join (list (reduce (lambda a, b: a + b, [zipped][0]))) + nohits [-1]
else:
if self.eat:
return ''
else:
return s
def test (self):
'''
Translates the definitions and prints the result. All targets
must look like the substitutes as defined. If a substitute differs,
it has been affected by the translation, indicating a potential
problem, should the substitute occur in the source.
'''
targets_translated = [self (item [0]) for item in self.definitions]
substitutes = [self (item [1])
Re: Changing the EAX register with Python
Am 18.11.2010 21:20, schrieb dutche:
Hi folks, I have a unusual question here.
How can I change the value of EAX register under python under Linux??
As paimei does under Windows.
My project is to have a python program that loads a C program and sets
a breakpoint at some address, and then with this breakpoint I change
the EAX register and then continue the program execution.
With Windows and paimei I did that, but under Linux I don't know yet.
Any ideas?
Thank you
You asked for it:
from ctypes import *
import time
import os
cdll.LoadLibrary('libc.so.6')
libc = CDLL('libc.so.6')
PTRACE_TRACEME = 0
PTRACE_GETREGS = 12
PTRACE_SETREGS = 13
PTRACE_SYSCALL = 24
SYS_WRITE = 4
SYS_IOCTL = 54
class user_regs_struct(Structure):
_fields_ = [
('ebx',c_ulong),
('ecx',c_ulong),
('edx',c_ulong),
('esi',c_ulong),
('edi',c_ulong),
('ebp',c_ulong),
('eax',c_ulong),
('xds',c_ulong),
('xes',c_ulong),
('xfs',c_ulong),
('xgs',c_ulong),
('orig_eax',c_ulong),
('eip',c_ulong),
('xcs',c_ulong),
('eflags',c_ulong),
('esp',c_ulong),
('xss',c_ulong),
]
child = os.fork()
if child == 0:
libc.ptrace(PTRACE_TRACEME,0,None,None)
os.execl('/bin/ls','ls')
else:
while True:
pid,status = os.wait()
if status != 0:
reg = pointer(user_regs_struct())
libc.ptrace(PTRACE_GETREGS,pid,None,reg)
if reg.contents.orig_eax == SYS_IOCTL:
print 'IOCTL ebx,ecx = 0x%0x,0x%0x' %
(reg.contents.ebx,reg.contents.ecx)
# replace IOCTL with SYS_WRITE
reg.contents.orig_eax = SYS_WRITE
libc.ptrace(PTRACE_SETREGS,pid,None,reg)
libc.ptrace(PTRACE_SYSCALL,pid,None,None)
else:
os._exit(0)
from ctypes import *
import time
import os
cdll.LoadLibrary('libc.so.6')
libc = CDLL('libc.so.6')
PTRACE_TRACEME = 0
PTRACE_GETREGS = 12
PTRACE_SETREGS = 13
PTRACE_SYSCALL = 24
SYS_WRITE = 4
SYS_IOCTL = 54
class user_regs_struct(Structure):
_fields_ = [
('ebx',c_ulong),
('ecx',c_ulong),
('edx',c_ulong),
('esi',c_ulong),
('edi',c_ulong),
('ebp',c_ulong),
('eax',c_ulong),
('xds',c_ulong),
('xes',c_ulong),
('xfs',c_ulong),
('xgs',c_ulong),
('orig_eax',c_ulong),
('eip',c_ulong),
('xcs',c_ulong),
('eflags',c_ulong),
('esp',c_ulong),
('xss',c_ulong),
]
child = os.fork()
if child == 0:
libc.ptrace(PTRACE_TRACEME,0,None,None)
os.execl('/bin/ls','ls')
else:
while True:
pid,status = os.wait()
if status != 0:
reg = pointer(user_regs_struct())
libc.ptrace(PTRACE_GETREGS,pid,None,reg)
if reg.contents.orig_eax == SYS_IOCTL:
print 'IOCTL ebx,ecx = 0x%0x,0x%0x' %
(reg.contents.ebx,reg.contents.ecx)
# replace IOCTL with SYS_WRITE
reg.contents.orig_eax = SYS_WRITE
libc.ptrace(PTRACE_SETREGS,pid,None,reg)
libc.ptrace(PTRACE_SYSCALL,pid,None,None)
else:
os._exit(0)attachment: stefan_sonnenberg.vcf--
http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
On 2010-11-18, noydb jenn.du...@gmail.com wrote: import subprocess pig = subprocess.Popen([C:\Halls\hallbig2.exe], stdin=subprocess.PIPE, stdout=subprocess.PIPE) result = pig.communicate(input='C:\Halls\Input\Ea39j.txt')[-1] #I need to capture the, what I think is the, last output From the subprocess documentation: [62]communicate() returns a tuple (stdoutdata, stderrdata). Note that if you want to send data to the process's stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too. By using index [-1] you are accessing the processes stderr stream. I am not really sure why you changed it. It doesn't give you the last output. Index 0 gives you *all* of stdout and index 1 gives you *all* of stderr, period. If you wish to further disect the output to get say the last line, then you will need to parse it separately. print result print pig.returncode None 0 So the tuple is empty. ?? The exe executes fine and returns output in th exe tool itself. The python script seems to execute fine, no errors, '...returned exit code 0'. Any ideas/suggestions? No the tuple contains two items (stdout, stderr). The first is what the program printed to its stdout stream (which is most likely the output you see if you run the command at a terminal/console). The second is what it printed to its stderr stream which is a channel used for out of band data such as error or status messages. In this case, it is None, because you did open stderr as a subprocess.PIPE. -- http://mail.python.org/mailman/listinfo/python-list
Why is pdb messing with STDIN? I can't step through my code, or enter any data accurately.
I'm baffled by the behavior of pdb as I attempt to debug a Twill test.
I'm trying to step through socket.get_fqdn() and my key entries are garbled.
I prefer to use ipdb, but the STDOUT gets messed up and printed at the end
of the ipdb session. That defeats the purpose of a debugger.
So I'm using pdb.
Here is the code:
def v1endpoint(self, port):
Return an OpenID 1.1 OpenIDServiceEndpoint for the server.
import pdb; pdb.set_trace()
base = http://%s:%s; % (socket.getfqdn('127.0.0.1'), port)
ep = OpenIDServiceEndpoint()
ep.claimed_id = base + /id/bob
ep.server_url = base + /openidserver
ep.type_uris = [OPENID_1_1_TYPE]
return ep
Here is the behavior:
--
Ran 2 tests in 0.029s
FAILED (errors=2)
Destroying test database 'default'...
/Users/Bryan/work/osqa/forum/tests.py(96)v1endpoint()
- base = http://%s:%s; % (socket.getfqdn('127.0.0.1'), port)
(Pdb) /Users/Bryan/work/osqa/forum/tests.py(96)v1endpoint()
- base = http://%s:%s; % (socket.getfqdn('127.0.0.1'), port)
# I enter 's' + return no character is displayed
(Pdb) --Call--
/usr/local/lib/python2.7/socket.py(124)getfqdn()
- def getfqdn(name=''):
# I enter 's' + return no character is displayed
(Pdb) /usr/local/lib/python2.7/socket.py(133)getfqdn()
- name = name.strip()
# I enter 's' + return no character is displayed
(Pdb)
# I enter 's' + return no character is displayed
(Pdb) *** NameError: name 'ss' is not defined
# I enter 'l' + return ; it's displayed but not the inputed data aparently
(Pdb) l*** NameError: name 'ss' is not defined
# I enter 'l' + return ; it's displayed but not the inputed data aparently
(Pdb) s
*** NameError: name 'ls' is not defined
# I enter 'name' + return ; it's displayed but not the inputed data
aparently
(Pdb) ne
None of the pdb commands work. Characters are added or not added. It's
driving me nuts.
I'm using Python 2.7 on osX-10.5.8
What is going on?
I've never seen anything like this.
thanks,
Bryan
--
http://mail.python.org/mailman/listinfo/python-list
Re: Changing the EAX register with Python
Here's -a- way to try - not necessarily the best way: 1) Write a little gcc .c with inline assembler, to make your EAX change 2) Call it from CPython using ctypes However, by the time you return from the code that makes your change and resume execution in the CPython interpreter, EAX may well have been changed to something new. Accumulators don't often hold the same value for very long. On Thu, Nov 18, 2010 at 12:20 PM, dutche dut...@gmail.com wrote: Hi folks, I have a unusual question here. How can I change the value of EAX register under python under Linux?? As paimei does under Windows. My project is to have a python program that loads a C program and sets a breakpoint at some address, and then with this breakpoint I change the EAX register and then continue the program execution. With Windows and paimei I did that, but under Linux I don't know yet. Any ideas? Thank you -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
On 2010-11-18, Jean-Michel Pichavant jeanmic...@sequans.com wrote: Tim Harig wrote: If you are not already, I would highly suggest using Python3 with the subprocess module: Suggesting subprocess is a good idea, *highly* suggesting python3 is questionable. The above code works in python 2. Many libraries (those included batteries) have not been ported yet to python 3. Py3 is a better core language than py2, but for now, less featured. I didn't, and I don't, recommend Python3 over Python2 for just any purpose.I recommended Python3's subprocess module over the Python2's subprocess module if that is indeed possible. I happen to be one of those that feels this transition was terribly mis-handled and that there should have been provisions to allow both versions to either be maintained together or to allow modules from both versions to work together. I don't really have a huge preference for either version; but, having to deal with both of them has given the project a black eye. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
On Nov 18, 5:22 pm, Tim Harig user...@ilthio.net wrote: On 2010-11-18, noydb jenn.du...@gmail.com wrote: import subprocess pig = subprocess.Popen([C:\Halls\hallbig2.exe], stdin=subprocess.PIPE, stdout=subprocess.PIPE) result = pig.communicate(input='C:\Halls\Input\Ea39j.txt')[-1] #I need to capture the, what I think is the, last output From the subprocess documentation: [62]communicate() returns a tuple (stdoutdata, stderrdata). Note that if you want to send data to the process's stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too. By using index [-1] you are accessing the processes stderr stream. I am not really sure why you changed it. It doesn't give you the last output. Index 0 gives you *all* of stdout and index 1 gives you *all* of stderr, period. If you wish to further disect the output to get say the last line, then you will need to parse it separately. Okay, I see now. I did run it to start with 0 -- still same result no matter if 0 or -1. So, what is result (stdout, using [0]) in this case? (yes, i know I sound dumb - programming is not my background, obviously). A list, tuple??? How do you access stdout (or is it stdoutdata?) results? I have tried, get errors with all attempts. The exe gui returns several statistical values uopn inputing a text file (containing numerous lines of value frequency) and clicking compute - I want just one of the values. print result print pig.returncode None 0 So the tuple is empty. ?? The exe executes fine and returns output in th exe tool itself. The python script seems to execute fine, no errors, '...returned exit code 0'. Any ideas/suggestions? No the tuple contains two items (stdout, stderr). The first is what the program printed to its stdout stream (which is most likely the output you see if you run the command at a terminal/console). The second is what it printed to its stderr stream which is a channel used for out of band data such as error or status messages. In this case, it is None, because you did open stderr as a subprocess.PIPE. -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On Nov 18, 1:31 pm, John Nagle na...@animats.com wrote: On 11/18/2010 4:24 AM, BartC wrote: John Nagle na...@animats.com wrote in message news:4ce37e01$0$1666$742ec...@news.sonic.net... On 11/16/2010 10:24 PM, swapnil wrote: AFAIK, the merging plan was approved by Guido early this year. I guess Google is expecting the community to drive the project from here on. That was the whole idea for merging it to mainline. From my last conversation with Collin, they are targeting Python 3.3 I think it's dead. They're a year behind on quarterly releases. The last release was Q3 2009. The project failed to achieve its stated goal of a 5x speedup. Not even close. More like 1.5x (http://www.python.org/dev/peps/pep-3146) There must have been good reasons to predict a 5x increase. For Java, adding a JIT improved performance by much more than that. Hard-code compilers for LISP have done much better than 5x. The best Java and LISP compilers approach the speed of C, while CPython is generally considered to be roughly 60 times slower than C. So 5x probably looked like a conservative goal. For Google, a company which buys servers by the acre, a 5x speedup would have a big payoff. Assuming the 5x speedup was shown to be viable (ie. performing the same benchmarks, on the same data, can be done that quickly in any other language, and allowing for the overheads associated with Python's dynamic nature), then what went wrong? Python is defined by what a naive interpreter with late binding and dynamic name lookups, like CPython, can easily implement. Simply emulating the semantics of CPython with generated code doesn't help all that much. Because you can monkey patch Python objects from outside the class, a local compiler, like a JIT, can't turn name lookups into hard bindings. Nor can it make reliable decisions about the types of objects. That adds a sizable performance penalty. Short of global program analysis, the compiler can't tell when code for the hard cases needs to be generated. So the hard-case code, where you figure out at run-time, for ever use of +, whether + is addition or concatenation, has to be generated every time. Making that decision is far slower than doing an add. This isn't completely accurate. It *is* possible to write a JIT compiler for a Python runtime which has fast path code for the common case, the case where the meaning of + doesn't change between every opcode. PyPy has produced some pretty good results with this approach. For those who haven't seen it yet, http://speed.pypy.org/ has some graphs which reflect fairly well on PyPy's performance for benchmarks that are not entirely dissimilar to real world code. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Extension on Windows
Hello,
I am trying to create an extension on Windows and I may be over my
head but I have made it pretty far.
I am trying to create bindings for some libraries which require me to
use Visual Studio 2005.
I set up the spammodule example and in VS set the output file to be a .pyd file.
When I copy that .pyd file into site-packages I can use it just fine
and call system('dir').
Now when I created a 2nd function to wrap a library function I get the
following.
ImportError: DLL load failed: The specified module could not be found.
I added the correct additional include directories and specified the
correct .lib file to get it to compile fine without any errors.
What is going on here? I tried running python with -vvv and got no
meaningful info... it just fails.
What else do I need to do?
Thanks,
~Eric
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
On 2010-11-18, noydb noyd...@gmail.com wrote: On Nov 18, 5:22 pm, Tim Harig user...@ilthio.net wrote: On 2010-11-18, noydb jenn.du...@gmail.com wrote: Okay, I see now. I did run it to start with 0 -- still same result no matter if 0 or -1. So, what is result (stdout, using [0]) in this case? (yes, i know I sound dumb - programming is not my background, obviously). A list, Nope, this one is my fault, I never should have posted being so tired. I was reading posts after being unable to sleep and missed something important about what you said. Sorry about the confusion. tuple??? How do you access stdout (or is it stdoutdata?) results? I have tried, get errors with all attempts. The exe gui returns several GUI Ew. I missed that part. GUIs, on Windows, do not have the standard streams. GUIs are in general, ugly to automate through the GUI itself. I would be much better if the program can be run with command line options, text interface, or if provides an automation object through COM or .Net. As workaround, if you run Python through Windows Script Host, you can open the program with WshShell and automate it, by sending it the keystrokes as you perform the action by typing, with SendKeys() -- http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
On Nov 18, 5:54 pm, noydb noyd...@gmail.com wrote: On Nov 18, 5:22 pm, Tim Harig user...@ilthio.net wrote: On 2010-11-18, noydb jenn.du...@gmail.com wrote: import subprocess pig = subprocess.Popen([C:\Halls\hallbig2.exe], stdin=subprocess.PIPE, stdout=subprocess.PIPE) result = pig.communicate(input='C:\Halls\Input\Ea39j.txt')[-1] #I need to capture the, what I think is the, last output From the subprocess documentation: [62]communicate() returns a tuple (stdoutdata, stderrdata). Note that if you want to send data to the process's stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too. By using index [-1] you are accessing the processes stderr stream. I am not really sure why you changed it. It doesn't give you the last output. Index 0 gives you *all* of stdout and index 1 gives you *all* of stderr, period. If you wish to further disect the output to get say the last line, then you will need to parse it separately. Okay, I see now. I did run it to start with 0 -- still same result no matter if 0 or -1. So, what is result (stdout, using [0]) in this case? (yes, i know I sound dumb - programming is not my background, obviously). A list, tuple??? How do you access stdout (or is it stdoutdata?) results? I have tried, get errors with all attempts. The exe gui returns several statistical values uopn inputing a text file (containing numerous lines of value frequency) and clicking compute - I want just one of the values. print result print pig.returncode None 0 So the tuple is empty. ?? The exe executes fine and returns output in th exe tool itself. The python script seems to execute fine, no errors, '...returned exit code 0'. Any ideas/suggestions? No the tuple contains two items (stdout, stderr). The first is what the program printed to its stdout stream (which is most likely the output you see if you run the command at a terminal/console). The second is what it printed to its stderr stream which is a channel used for out of band data such as error or status messages. In this case, it is None, because you did open stderr as a subprocess.PIPE.- Hide quoted text - - Show quoted text -- Hide quoted text - - Show quoted text - stdout is a file object -- http://mail.python.org/mailman/listinfo/python-list
Re: How to run an EXE, with argument, capture output value
On Nov 18, 6:20 pm, noydb noyd...@gmail.com wrote:
On Nov 18, 5:54 pm, noydb noyd...@gmail.com wrote:
On Nov 18, 5:22 pm, Tim Harig user...@ilthio.net wrote:
On 2010-11-18, noydb jenn.du...@gmail.com wrote:
import subprocess
pig = subprocess.Popen([C:\Halls\hallbig2.exe],
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
result = pig.communicate(input='C:\Halls\Input\Ea39j.txt')[-1] #I need
to capture the, what I think is the, last output
From the subprocess documentation:
[62]communicate() returns a tuple (stdoutdata, stderrdata).
Note that if you want to send data to the process's stdin,
you need to create the Popen object with stdin=PIPE. Similarly,
to get anything other than None in the result tuple, you need
to give stdout=PIPE and/or stderr=PIPE too.
By using index [-1] you are accessing the processes stderr stream. I am
not really sure why you changed it. It doesn't give you the last output.
Index 0 gives you *all* of stdout and index 1 gives you *all* of stderr,
period. If you wish to further disect the output to get say the last
line,
then you will need to parse it separately.
Okay, I see now. I did run it to start with 0 -- still same result no
matter if 0 or -1.
So, what is result (stdout, using [0]) in this case? (yes, i know I
sound dumb - programming is not my background, obviously). A list,
tuple??? How do you access stdout (or is it stdoutdata?) results? I
have tried, get errors with all attempts. The exe gui returns several
statistical values uopn inputing a text file (containing numerous
lines of value frequency) and clicking compute - I want just one
of the values.
print result
print pig.returncode
None
0
So the tuple is empty. ?? The exe executes fine and returns output in
th exe tool itself. The python script seems to execute fine, no
errors, '...returned exit code 0'. Any ideas/suggestions?
No the tuple contains two items (stdout, stderr). The first is what the
program printed to its stdout stream (which is most likely the output you
see if you run the command at a terminal/console). The second is what it
printed to its
stderr stream which is a channel used for out of band data such as error
or
status messages. In this case, it is None, because you did open stderr
as a
subprocess.PIPE.- Hide quoted text -
- Show quoted text -- Hide quoted text -
- Show quoted text -
stdout is a file object- Hide quoted text -
- Show quoted text -
if I do
print pig.communicate()
('', None)
Doesn't that mean it is empty?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Thu, 18 Nov 2010 14:21:47 +, Martin Gregorie wrote: I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. You're mistaken then about Python, because it does have a separate compilation phase that runs before the program can be executed. Where do you think the .pyc files come from, and what did you think the compile() function did? It just happens automatically, rather than manually. For that matter, what about languages with JIT compilers, like Java or, yes, Python? CPython itself has no built-in JIT, but PyPy does, and Psyco is an optimizing JIT compiler for CPython on some hardware. I think the difference between scripts and programs is about as meaningful as the difference between mountains and hills -- there clearly is a difference between *this specific* hill and *that specific* mountain, but any general distinction between hills and mountains is arbitrary and artificial. Maybe, once upon a time, there was a meaningful distinction to be made between purely interpreted languages and purely compiled languages. But today, when interpreted languages have a compilation phase, and compiled languages have the equivalent of eval() and do their compilation at runtime, such a distinction is now fairly arbitrary. Better questions are: * Does the program compile to native machine code or byte code? * How efficient is the machine code, or the virtual machine that executes the byte code? What do you mean by efficient? * Is there an integrated parse-compile-execute cycle, or does the developer have to perform them individually? * To what degree is the syntax of the language aimed at rapid development and/or runtime efficiency? Harder questions to ask, and answer, than Is that language for writing scripts or programs or applications?, but the answers are far more meaningful. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Some syntactic sugar proposals
On Thu, 18 Nov 2010 09:32:23 +, Mark Wooding wrote: [...] You're wrong. Python evaluates these left-to-right, as I said. Parentheses override operator associativity; they don't affect evaluation order at all. Fair enough. I concede your point. [...] Not everything needs to be a one liner. If you need this, do it the old- fashioned way: t = foo() if not pred(t): t = default_value I already explained how to write it as a one-liner: t = (lambda y: y if pred(y) else default_value)(foo()) I didn't say it couldn't be written as a one-liner. I suggested that it was better not to. The costs of the one-liner are: * reduced readability; * requires an increased level of knowledge of the reader (what's lambda do?); * runtime inefficiency (you create a function object, only to use it once then throw it away). The advantages? * one fewer line of code. In my experience, the obsessiveness in which some people look for one- liners is far from helpful, and goes against the spirit of Python. This isn't Perl :) -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On 19/11/2010 00:07, Steven D'Aprano wrote: On Thu, 18 Nov 2010 14:21:47 +, Martin Gregorie wrote: I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. You're mistaken then about Python, because it does have a separate compilation phase that runs before the program can be executed. Where do you think the .pyc files come from, and what did you think the compile() function did? It just happens automatically, rather than manually. [snip] I think what he means is that you don't need explicitly to compile and then run. -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On 19.11.2010 01:26, MRAB wrote: On 19/11/2010 00:07, Steven D'Aprano wrote: On Thu, 18 Nov 2010 14:21:47 +, Martin Gregorie wrote: I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. You're mistaken then about Python, because it does have a separate compilation phase that runs before the program can be executed. Where do you think the .pyc files come from, and what did you think the compile() function did? It just happens automatically, rather than manually. [snip] I think what he means is that you don't need explicitly to compile and then run. What difference does it make? Is 'print Hello' a program or a script? Are you saying, that it depends on whether you have to manually call some compiler? -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Fri, 19 Nov 2010 00:07:05 +, Steven D'Aprano wrote: On Thu, 18 Nov 2010 14:21:47 +, Martin Gregorie wrote: I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. You're mistaken then about Python, because it does have a separate compilation phase that runs before the program can be executed. Where do you think the .pyc files come from, and what did you think the compile() function did? It just happens automatically, rather than manually. Notice my mention of the hash-bang line? That makes it quite clear that I did not mean what you're reading into separate compile phase which must be run because hash-bang means you can feed source into the run-time execution system, and also means that by definition the source contains or references everything a program needs to run and that the source code translation step, if there is one, is fast enough for the time it takes to be negligible. Those aren't necessarily the case with a separate compilation step. .pyc files are an optional extra: in a *nix environment they are not produced if you use a hash-bang line to execute the source file: they only appear if you run the source with a python file.py command. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Advise for numerical programming content (New python user)
Python Users, I am an average Fortran user. I am new to python and I am currently evaluating options and functionalities of numerical programming and related 2d and 3d graphic outputs with python. Kindly share your experience in scientific programming with python like how do you like it, comparison with Fortran and C++. Which version of python + numpy+scipy are compatible with each other or if any other numerical analysis package is available (I am working on windows environment.) Does graphic output like maps, histogram, crossplot, tornado charts is good enough with basic installation or needs some additional packages? Your feedback is valuable for me to start. Thanks Regards Sachin Sachin Kumar Sharma Senior Geoscientist -- http://mail.python.org/mailman/listinfo/python-list
Bunch 2.0 - a dict with a default
Pythonistas:
If everyone likes this post, then the code is a snippet for
community edification. Otherwise, it's a question: How to do this kind
of thing better?
I want a dict() variant that passes these test cases:
map = Map()
assert not(map.has_key('_default'))
map = Map(yo='dude')
assert map['yo'] == 'dude'
assert map.yo == 'dude'
assert None == map['whatever']
assert not(map.has_key('_default'))
map = Map(yo='dude', _default='q')
assert 'q' == map['whatever']
assert not(map.has_key('_default'))
That's like Bunch, but with a default value. (That makes code with
excess if statements less likely.)
So here's the implementation:
def Map(*args, **kwargs):
value = kwargs.get('_default', None)
if kwargs.has_key('_default'): del kwargs['_default']
class _DefMap(dict):
def __init__(self, *a, **kw):
dict.__init__(self, *a, **kw)
self.__dict__ = self
def __getitem__(self, key):
if not self.has_key(key): self[key] = value
return dict.__getitem__(self, key)
return _DefMap(*args, **kwargs)
--
Phlip
http://bit.ly/ZeekLand
--
http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
John Nagle na...@animats.com writes: Python is defined by what a naive interpreter with late binding and dynamic name lookups, like CPython, can easily implement. Simply emulating the semantics of CPython with generated code doesn't help all that much. Indeed. Because you can monkey patch Python objects from outside the class, a local compiler, like a JIT, can't turn name lookups into hard bindings. Nor can it make reliable decisions about the types of objects. But it /can/ make guesses. A dynamic runtime doesn't have to predict everything right in advance; it only has to predict most things sort of well enough, and fix up the things it got wrong before anyone notices. For example, A Python compiler could inline a function call if it makes a note to recompile the calling function if the called function is modified. Most functions aren't redefined, so this is probably a pretty good guess. That adds a sizable performance penalty. Short of global program analysis, the compiler can't tell when code for the hard cases needs to be generated. The right approach is to guess that things are going to be done the easy way, and then detect when the guess is wrong. So the hard-case code, where you figure out at run-time, for ever use of +, whether + is addition or concatenation, has to be generated every time. Making that decision is far slower than doing an add. There's an old trick here called `inline caching'. The first time a function is called, compile it so as to assume that types of things are as you found this time: inline simple methods, and so on. Insert some quick type checks at the top: is this going to work next time? If not, take a trap back into the compiler. The traditional approach is to replace the mispredictions with full dispatches (`monomorphic inline caching'); the clever approach tolerates a few different types, dispatching to optimized code for each (`polymorphic inline caching'), unless there are just too many decision points and you give up. There are time/space tradeoffs to be made here too. Fortunately, you don't have to compile everything super-optimized from the get-go: you can dynamically identify the inner loops which need special attention, and get the compiler to really stare hard at them. The rest of the program might plausibly be left interpreted much of the time for all anyone will care. I've referred to this problem as gratuitous hidden dynamism. Most things which could be changed dynamically in a Python program usually aren't. This is one of the crucial observations for making a dynamic language go fast; the other is that you still have the compiler around if you guessed wrong. An aggressively dynamic runtime has two enormous advantages over batch compilers such as are traditionally used for C: it gets the entire program in one go, and it gets to see the real live data that the program's meant to run against. Given that, I'd expect it to be able to /beat/ a batch compiler in terms of performance. This has been pointed out many times by many people. There's even a PhD thesis on the topic. Without a few restrictions, so that a compiler can at least tell when support for the hard cases is needed, Python cannot be compiled well. This assumes static compilation. It's the wrong approach for a dynamic language like Python. -- [mdw] -- http://mail.python.org/mailman/listinfo/python-list
Re: Bunch 2.0 - a dict with a default
You may want to look at the collections.defaultdict class. It takes in a
factory function for default values.
You can also implement your class by overriding the __missing__ method of
the dict class, rather than overriding the __getitem__.
Both were added in Python 2.5 according to the documentation.
Chris
On Thu, Nov 18, 2010 at 5:45 PM, Phlip phlip2...@gmail.com wrote:
Pythonistas:
If everyone likes this post, then the code is a snippet for
community edification. Otherwise, it's a question: How to do this kind
of thing better?
I want a dict() variant that passes these test cases:
map = Map()
assert not(map.has_key('_default'))
map = Map(yo='dude')
assert map['yo'] == 'dude'
assert map.yo == 'dude'
assert None == map['whatever']
assert not(map.has_key('_default'))
map = Map(yo='dude', _default='q')
assert 'q' == map['whatever']
assert not(map.has_key('_default'))
That's like Bunch, but with a default value. (That makes code with
excess if statements less likely.)
So here's the implementation:
def Map(*args, **kwargs):
value = kwargs.get('_default', None)
if kwargs.has_key('_default'): del kwargs['_default']
class _DefMap(dict):
def __init__(self, *a, **kw):
dict.__init__(self, *a, **kw)
self.__dict__ = self
def __getitem__(self, key):
if not self.has_key(key): self[key] = value
return dict.__getitem__(self, key)
return _DefMap(*args, **kwargs)
--
Phlip
http://bit.ly/ZeekLand
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: Some syntactic sugar proposals
Steven D'Aprano st...@remove-this-cybersource.com.au writes: Not everything needs to be a one liner. If you need this, do it the old- fashioned way: t = foo() if not pred(t): t = default_value I already explained how to write it as a one-liner: t = (lambda y: y if pred(y) else default_value)(foo()) I didn't say it couldn't be written as a one-liner. I suggested that it was better not to. Ahh. I misunderstood the first sentence above as dismissing the possibility. Sorry. I agree that it's not a /nice/ one-liner. ;-) The costs of the one-liner are: * reduced readability; * requires an increased level of knowledge of the reader (what's lambda do?); * runtime inefficiency (you create a function object, only to use it once then throw it away). This last can be obviated by a clever compiler (which, in our case, we have not got). The second could be considered an advantage: it's educational! The advantages? * one fewer line of code. In my experience, the obsessiveness in which some people look for one- liners is far from helpful, and goes against the spirit of Python. This isn't Perl :) Oh, I agree completely. On the other hand, it's just /fun/. Python is a fun language and using its features in playfully unusual ways is enjoyable; like a good pun in a natural language. Just as puns aren't always appropriate in written language, playful code isn't always appropriate either; but that doesn't mean it's never appropriate. (Java has no sense of humour. C++ does have, but it's a bit weird.) -- [mdw] -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
Steven D'Aprano st...@remove-this-cybersource.com.au writes: On Thu, 18 Nov 2010 14:21:47 +, Martin Gregorie wrote: I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. You're mistaken then about Python, because it does have a separate compilation phase that runs before the program can be executed. Where do you think the .pyc files come from, and what did you think the compile() function did? It just happens automatically, rather than manually. I think Martin meant that there's no /explicit/ compilation phase. As a case in point, tcc is an optimizing native-code compiler for C which is so fast at generating code that it has a mode which accepts an initial `#!' line in its input; the documentation refers to C `scripts'. Awk, Javascript and Perl have full compilation phases too, but they're hidden from the user; bash certainly builds a parse tree but I don't know whether it just walks that or has some other kind of internal representation. Maybe, once upon a time, there was a meaningful distinction to be made between purely interpreted languages and purely compiled languages. No, there never was; only between interpreting and compiling implementations. But today, when interpreted languages have a compilation phase, and compiled languages have the equivalent of eval() and do their compilation at runtime, such a distinction is now fairly arbitrary. Better questions are: * Does the program compile to native machine code or byte code? * How efficient is the machine code, or the virtual machine that executes the byte code? What do you mean by efficient? * Is there an integrated parse-compile-execute cycle, or does the developer have to perform them individually? * To what degree is the syntax of the language aimed at rapid development and/or runtime efficiency? Harder questions to ask, and answer, than Is that language for writing scripts or programs or applications?, but the answers are far more meaningful. Maybe. I note that there are Lisp systems (plural!) which compile to native code (only -- with no interpreter at all), but fully support runtime `eval', batch compilation of source files, aggressive optimization for speed and/or space, and so on. So I think that some of the above questions, at least, present false dichotomies: one doesn't have to choose one or the other; one can have cake and eat it too. Also some of the questions are about the language and some are about the implementation. It's important not to get the two confused. -- [mdw] -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On Nov 18, 2:59 pm, Jean-Paul Calderone calderone.jeanp...@gmail.com wrote: On Nov 18, 1:31 pm, John Nagle na...@animats.com wrote: On 11/18/2010 4:24 AM, BartC wrote: John Nagle na...@animats.com wrote in message news:4ce37e01$0$1666$742ec...@news.sonic.net... On 11/16/2010 10:24 PM, swapnil wrote: AFAIK, the merging plan was approved by Guido early this year. I guess Google is expecting the community to drive the project from here on. That was the whole idea for merging it to mainline. From my last conversation with Collin, they are targeting Python 3.3 I think it's dead. They're a year behind on quarterly releases. The last release was Q3 2009. The project failed to achieve its stated goal of a 5x speedup. Not even close. More like 1.5x (http://www.python.org/dev/peps/pep-3146) There must have been good reasons to predict a 5x increase. For Java, adding a JIT improved performance by much more than that. Hard-code compilers for LISP have done much better than 5x. The best Java and LISP compilers approach the speed of C, while CPython is generally considered to be roughly 60 times slower than C. So 5x probably looked like a conservative goal. For Google, a company which buys servers by the acre, a 5x speedup would have a big payoff. Assuming the 5x speedup was shown to be viable (ie. performing the same benchmarks, on the same data, can be done that quickly in any other language, and allowing for the overheads associated with Python's dynamic nature), then what went wrong? Python is defined by what a naive interpreter with late binding and dynamic name lookups, like CPython, can easily implement. Simply emulating the semantics of CPython with generated code doesn't help all that much. Because you can monkey patch Python objects from outside the class, a local compiler, like a JIT, can't turn name lookups into hard bindings. Nor can it make reliable decisions about the types of objects. That adds a sizable performance penalty. Short of global program analysis, the compiler can't tell when code for the hard cases needs to be generated. So the hard-case code, where you figure out at run-time, for ever use of +, whether + is addition or concatenation, has to be generated every time. Making that decision is far slower than doing an add. This isn't completely accurate. It *is* possible to write a JIT compiler for a Python runtime which has fast path code for the common case, the case where the meaning of + doesn't change between every opcode. PyPy has produced some pretty good results with this approach. Right. The key is to be able to dispatch on the type of object once for a given chunk of code, which is possible if you do some kind of flow path analysis on the function/chunk. PyPy is starting to look much better of late, I kind of thought their first approach was wrong (or at least too much) but they seem to have pushed though it. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On 11/18/2010 9:05 PM, Mark Wooding wrote: Maybe, once upon a time, there was a meaningful distinction to be made between purely interpreted languages and purely compiled languages. No, there never was; only between interpreting and compiling implementations. Right. I remember using a C *interpreter* about 25 years ago on Unix. And on Windows, as I remember, the Borland TurboC system let one compile, link, and run with one command. A language is just a language, not an implementation. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
[issue10183] test_concurrent_futures failure on Windows
Brian Quinlan br...@sweetapp.com added the comment: Would you include the entire output of the test run? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10183 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4113] Add custom __repr__ to functools.partial
Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Looks like a reasonable proposal, but coding this in C is a chore. It's not that bad. Most C code is a bit of a chore compared to Python but it really doesn't take much to write a C equivalent of: functools.partial(%r, %s) % (self.func, ', '.join(repr(a) for a in self.args) How hard/inefficient would it be to have 99% of partial coded in C and one stub in functools.py? Let's not do this. There is too little benefit to warrant going down the path of splitting the code across two langauges. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10183] test_concurrent_futures failure on Windows
Antoine Pitrou pit...@free.fr added the comment: Y:\py3k\__svn__\PCbuild\amd64\python_d.exe -m test.regrtest -v test_concurrent_futures == CPython 3.2a4+ (py3k, Nov 18 2010, 11:12:09) [MSC v.1500 64 bit (AMD64)] == Windows-7-6.1.7600 little-endian == Y:\py3k\__svn__\build\test_python_2016 [1/1] test_concurrent_futures test_map (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ok test_map_exception (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ok test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest) ... FAIL test_submit (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ok test_submit_keyword (test.test_concurrent_futures.ProcessPoolExecutorTest) ... o k test_map (test.test_concurrent_futures.ThreadPoolExecutorTest) ... ok test_map_exception (test.test_concurrent_futures.ThreadPoolExecutorTest) ... ok test_map_timeout (test.test_concurrent_futures.ThreadPoolExecutorTest) ... ok test_submit (test.test_concurrent_futures.ThreadPoolExecutorTest) ... ok test_submit_keyword (test.test_concurrent_futures.ThreadPoolExecutorTest) ... ok test_all_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ok test_all_completed_some_already_completed (test.test_concurrent_futures.ProcessP oolWaitTests) ... ok test_first_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ok test_first_completed_one_already_completed (test.test_concurrent_futures.Process PoolWaitTests) ... ok test_first_exception (test.test_concurrent_futures.ProcessPoolWaitTests) ... ok test_first_exception_one_already_failed (test.test_concurrent_futures.ProcessPoo lWaitTests) ... ok test_first_exception_some_already_complete (test.test_concurrent_futures.Process PoolWaitTests) ... ok test_timeout (test.test_concurrent_futures.ProcessPoolWaitTests) ... FAIL test_all_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... ok test_all_completed_some_already_completed (test.test_concurrent_futures.ThreadPo olWaitTests) ... ok test_first_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... ok test_first_completed_one_already_completed (test.test_concurrent_futures.ThreadP oolWaitTests) ... ok test_first_exception (test.test_concurrent_futures.ThreadPoolWaitTests) ... ok test_first_exception_one_already_failed (test.test_concurrent_futures.ThreadPool WaitTests) ... ok test_first_exception_some_already_complete (test.test_concurrent_futures.ThreadP oolWaitTests) ... ok test_timeout (test.test_concurrent_futures.ThreadPoolWaitTests) ... ok test_no_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... o k test_zero_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ok test_no_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... ok test_zero_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... ok test_cancel (test.test_concurrent_futures.FutureTests) ... ok test_cancelled (test.test_concurrent_futures.FutureTests) ... ok test_done (test.test_concurrent_futures.FutureTests) ... ok test_done_callback_already_cancelled (test.test_concurrent_futures.FutureTests) ... ok test_done_callback_already_failed (test.test_concurrent_futures.FutureTests) ... ok test_done_callback_already_successful (test.test_concurrent_futures.FutureTests) ... ok test_done_callback_raises (test.test_concurrent_futures.FutureTests) ... ok test_done_callback_with_cancel (test.test_concurrent_futures.FutureTests) ... ok test_done_callback_with_exception (test.test_concurrent_futures.FutureTests) ... ok test_done_callback_with_result (test.test_concurrent_futures.FutureTests) ... ok test_exception_with_success (test.test_concurrent_futures.FutureTests) ... ok test_exception_with_timeout (test.test_concurrent_futures.FutureTests) ... ok test_repr (test.test_concurrent_futures.FutureTests) ... ok test_result_with_cancel (test.test_concurrent_futures.FutureTests) ... ok test_result_with_success (test.test_concurrent_futures.FutureTests) ... ok test_result_with_timeout (test.test_concurrent_futures.FutureTests) ... ok test_running (test.test_concurrent_futures.FutureTests) ... ok test_context_manager_shutdown (test.test_concurrent_futures.ProcessPoolShutdownT est) ... ok test_del_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ok test_processes_terminate (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ok test_run_after_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) . .. ok test_context_manager_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTe st) ... ok test_del_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... ok test_run_after_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) .. . ok test_threads_terminate (test.test_concurrent_futures.ThreadPoolShutdownTest) ... ok == FAIL: test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest)
[issue10351] Add autocompletion for keys in dictionaries
Valery Khamenya khame...@gmail.com added the comment: Guys, do you expect anythig else from me in respect to this issue? Let me know it before my non-stopable garbage collector wipes all the details from my brain away :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10351 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10351] Add autocompletion for keys in dictionaries
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +facundobatista, georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10351 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
