Re: What's the best way to write this regular expression?

[John Salerno]

After a bit of reading, I've decided to use Beautiful Soup 4, with
lxml as the parser. I considered simply using lxml to do all the work,
but I just got lost in the documentation and tutorials. I couldn't
find a clear explanation of how to parse an HTML file and then
navigate its structure.

The Beautiful Soup 4 documentation was very clear, and BS4 itself is
so simple and Pythonic. And best of all, since version 4 no longer
does the parsing itself, you can choose your own parser, and it works
with lxml, so I'll still be using lxml, but with a nice, clean overlay
for navigating the tree structure.

Thanks for the advice!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: help: confused about python flavors....

[Steven D'Aprano]

On Tue, 06 Mar 2012 20:06:37 -0800, amar Singh wrote:

> Hi,
> 
> I am confused between plain python, numpy, scipy, pylab, matplotlib.

Python is a programming language. It comes standard with many libraries 
for doing basic mathematics, web access, email, etc.

Numpy is a library for doing scientific numerical maths work and fast 
processing of numeric arrays.

Scipy is another library for scientific work. It is separate from, but 
uses, Numpy.

Matplotlib is a project for making graphing and plotting of numeric data 
easy in Python.

Pylab is a project to be Python's version of Matlab: it intends to be an 
integrated bundle of Python the programming language, Numpy, Scipy, and 
Matplotlib all in one easy-to-use application.


> I have high familiarity with matlab, but the computer I use does not
> have it. So moving to python.
> What should I use? and the best way to use it. I will be running
> matlab-like scripts sometimes on the shell prompt and sometimes on the
> command line.

Pylab is intended to be the closest to Matlab, but I don't know how close 
it is. Also, Pylab is NOT compatible with Matlab: its aim is to be an 
alternative to Matlab, not to be a clone. So it cannot run Matlab scripts.

You might also like to look at Sage:

http://www.sagemath.org/

Sage is a Python project aimed to be an alternative to Mathematica.


Ultimately, you will have to look at the packages, see their features, 
perhaps try them for a while (they are all free software, so the only 
cost is your time), and decide for yourself which one meets your needs. 
We can't answer that, because we don't know what you need.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

help: confused about python flavors....

[amar Singh]

Hi,

I am confused between plain python, numpy, scipy, pylab, matplotlib.

I have high familiarity with matlab, but the computer I use does not
have it. So moving to python.
What should I use? and the best way to use it. I will be running
matlab-like scripts sometimes on the shell prompt and sometimes on the
command line.

Please help. Thanks in advance.
-- 
http://mail.python.org/mailman/listinfo/python-list

Get tkinter text to the clipboard

[bugzilla-mail-box]

How can I get something from tkinter gui to another program ?
tkinter on python 3.2 on kde4
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: are int, float, long, double, side-effects of computer engineering?

[rusi]

On Mar 6, 6:11 am, Xah Lee  wrote:
> some additional info i thought is relevant.
>
> are int, float, long, double, side-effects of computer engineering?

It is a bit naive for computer scientists to club integers and reals
as mathematicians do given that for real numbers, even equality is
undecidable!
Mostly when a system like mathematica talks of real numbers it means
computable real numbers which is a subset of mathematical real numbers
(and of course a superset of floats)

See
http://en.wikipedia.org/wiki/Computable_number#Can_computable_numbers_be_used_instead_of_the_reals.3F
-- 
http://mail.python.org/mailman/listinfo/python-list

Tools for refactoring/obfuscation

[Javier]

I am looking for an automated tool for refactoring/obfuscation.
Something that changes names of functions, variables, or which would
merge all the functions of various modules in a single module.
The closest I have seen is http://bicyclerepair.sourceforge.net/

Does somebody know of something that can work from the command line or
simple data structures/text files?, like a python dictionary of functions
{"old":"new",...}

-- 
http://mail.python.org/mailman/listinfo/python-list

Why this recursive import fails?

[INADA Naoki]

I have 4 py files like below. Two __init__.py is empty file.

$ find foo -name "*.py"
foo/lib/lib.py
foo/lib/__init__.py
foo/__init__.py
foo/foo.py

$ cat foo/lib/lib.py
from __future__ import absolute_import
print('lib.py', __name__)
from .. import foo
#import foo.foo

$ cat foo/foo.py
from __future__ import absolute_import
print('foo.py', __name__)
from .lib import lib
#import foo.lib.lib

Then, importing foo.foo or foo.lib.lib fails unexpectedly.

# `from .. import foo` success but `from .lib import lib` fails.
$ python -c "import foo.lib.lib"
('lib.py', 'foo.lib.lib')
('foo.py', 'foo.foo')
Traceback (most recent call last):
  File "", line 1, in 
  File "foo/lib/lib.py", line 3, in 
from .. import foo
  File "foo/foo.py", line 3, in 
from .lib import lib
ImportError: cannot import name lib

# `from .lib import lib` success but `from .. import foo` fails.
$ python -c "import foo.foo"
('foo.py', 'foo.foo')
('lib.py', 'foo.lib.lib')
Traceback (most recent call last):
  File "", line 1, in 
  File "foo/foo.py", line 3, in 
from .lib import lib
  File "foo/lib/lib.py", line 3, in 
from .. import foo
ImportError: cannot import name foo


I can run both with absolute import.
What's wrong about my relative import?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[Roy Smith]

In article 
<12783654.1174.1331073814011.JavaMail.geo-discussion-forums@yner4>,
 John Salerno  wrote:

> I sort of have to work with what the website gives me (as you'll see below), 
> but today I encountered an exception to my RE. Let me just give all the 
> specific information first. The point of my script is to go to the specified 
> URL and extract song information from it.

Rule #1: Don't try to parse XML, HTML, or any other kind of ML with 
regular expressions.

Rule #2: Use a dedicated ML parser.  I like lxml (http://lxml.de/).  
There's other possibilities.

Rule #3: If in doubt, see rule #1.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[Terry Reedy]

On 3/6/2012 6:57 PM, John Salerno wrote:

Also, you're still double-posting.


Grr. I just reported it to Google, but I think if I start to frequent
the newsgroup again I'll have to switch to Thunderbird, or perhaps
I'll just try switching back to the old Google Groups interface. I
think the issue is the new interface.


I am not seeing the double posting, but I use Thunderbird + the 
news.gmane.org mirrors of python-list and others.


--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[Terry Reedy]

On 3/6/2012 6:05 PM, John Salerno wrote:

Anything that allows me NOT to use REs is welcome news, so I look
forward to learning about something new! :)


I should ask though...are there alternatives already bundled with
Python that I could use?


lxml is +- upward compatible with xml.etree in the stdlib.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: are int, float, long, double, side-effects of computer engineering?

[Westley Martínez]

On Tue, Mar 06, 2012 at 04:29:10PM -0500, Calvin Kim wrote:
> On 03/06/2012 01:34 AM, Xah Lee wrote:
> >while what you said is true, but the problem is that 99.99% of
> >programers do NOT know this. They do not know Mathematica. They've
> >never seen a language with such feature. The concept is alien. This is
> >what i'd like to point out and spread awareness.
> >
> I can see your point. But that's not simply true. In my case and
> many others, such issue was addressed during first week of
> introductory programming classes. I was naively thought "computer =
> precision" and I was stunned to find out the inaccuracy of computer
> calculations.
> 
> But as you experienced, I also stumble upon some people (specially
> Java only programmers) who were not aware of it.
> 
> >also, argument about raw speed and fine control vs automatic
> >management, rots with time. Happened with auto memory management,
> >managed code, compilers, auto type conversion, auto extension of
> >array, auto type system, dynamic/scripting languages, etc.
> Maybe it's because I'm not in scientific community, that I learned
> to live with such side-effects. Because 99.99% of computer users and
> programmers can afford to, and willing to lose such small inaccuracy
> billion times in exchange for some performance increase and
> convenience. Although NASA may not accept my application for their
> projects for Mars mission after this posting.

Also remember that double precision is not the maximum.  There exist
standards for triple and quadruple precision formats, as well as other
extended formats.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pickle/unpickle class which has changed

[Gelonida N]

Hi Peter,

A related question.


Is there anyhing like a built in signature which would help to detect,
that one tries to unpickle an object whose byte code has changed?

The idea is to distinguish old and new pickled data and start some
'migration code' fi required


The only thing, that I thought about so far was adding an explicit
version number to each class in order to detect such situations.



On 03/06/2012 02:52 PM, Peter Otten wrote:
> Neal Becker wrote:
> 
>> What happens if I pickle a class, and later unpickle it where the class
>> now has added some new attributes?
> 
> - If the added attributes' values are immutable, provide defaults as class 
> attributes.
> 
> - Implement an appropriate __setstate__() method. The easiest would be
> 
> # untested
> def __setstate__(self, state):
> self.__dict__.update(newattr1=42, newattr2=[])
> self.__dict__.update(state)
> 


-- 
http://mail.python.org/mailman/listinfo/python-list

RE: What's the best way to write this regular expression?

[Prasad, Ramit]

> 
> > Also, you're still double-posting.
> 
> Grr. I just reported it to Google, but I think if I start to frequent the
> newsgroup again I'll have to switch to Thunderbird, or perhaps I'll just
> try switching back to the old Google Groups interface. I think the issue is
> the new interface.
> 
> Sorry.

Oddly, I see no double posting for this thread on my end (email list).

Ramit


Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[John Salerno]

> Also, you're still double-posting.

Grr. I just reported it to Google, but I think if I start to frequent the 
newsgroup again I'll have to switch to Thunderbird, or perhaps I'll just try 
switching back to the old Google Groups interface. I think the issue is the new 
interface.

Sorry.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[Steven D'Aprano]

On Tue, 06 Mar 2012 15:05:39 -0800, John Salerno wrote:

>> Anything that allows me NOT to use REs is welcome news, so I look
>> forward to learning about something new! :)
> 
> I should ask though...are there alternatives already bundled with Python
> that I could use? Now that you mention it, I remember something called
> HTMLParser (or something like that) and I have no idea why I never
> looked into that before I messed with REs.

import htmllib
help(htmllib)

The help is pretty minimal and technical, you might like to google on a 
tutorial or two:

https://duckduckgo.com/html/?q=python%20htmllib%20tutorial

Also, you're still double-posting.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[John Salerno]

Thanks. I'm thinking the choice might be between lxml and Beautiful
Soup, but since BS uses lxml as a parser, I'm trying to figure out the
difference between them. I don't necessarily need the simplest
(html.parser), but I want to choose one that is simple enough yet
powerful enough that I won't have to learn another method later.




On Tue, Mar 6, 2012 at 5:35 PM, Ian Kelly  wrote:
> On Tue, Mar 6, 2012 at 4:05 PM, John Salerno  wrote:
>>> Anything that allows me NOT to use REs is welcome news, so I look forward 
>>> to learning about something new! :)
>>
>> I should ask though...are there alternatives already bundled with Python 
>> that I could use? Now that you mention it, I remember something called 
>> HTMLParser (or something like that) and I have no idea why I never looked 
>> into that before I messed with REs.
>
> HTMLParser is pretty basic, although it may be sufficient for your
> needs.  It just converts an html document into a stream of start tags,
> end tags, and text, with no guarantee that the tags will actually
> correspond in any meaningful way.  lxml can be used to output an
> actual hierarchical structure that may be easier to manipulate and
> extract data from.
>
> Cheers,
> Ian
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[John Salerno]

On Tuesday, March 6, 2012 5:05:39 PM UTC-6, John Salerno wrote:
> > Anything that allows me NOT to use REs is welcome news, so I look forward 
> > to learning about something new! :)
> 
> I should ask though...are there alternatives already bundled with Python that 
> I could use? Now that you mention it, I remember something called HTMLParser 
> (or something like that) and I have no idea why I never looked into that 
> before I messed with REs.
> 
> Thanks.

Also, I just noticed Beautiful Soup, which seems appropriate. I suppose any 
will do, but knowing the pros and cons would help with a decision.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[Ian Kelly]

On Tue, Mar 6, 2012 at 4:05 PM, John Salerno  wrote:
>> Anything that allows me NOT to use REs is welcome news, so I look forward to 
>> learning about something new! :)
>
> I should ask though...are there alternatives already bundled with Python that 
> I could use? Now that you mention it, I remember something called HTMLParser 
> (or something like that) and I have no idea why I never looked into that 
> before I messed with REs.

HTMLParser is pretty basic, although it may be sufficient for your
needs.  It just converts an html document into a stream of start tags,
end tags, and text, with no guarantee that the tags will actually
correspond in any meaningful way.  lxml can be used to output an
actual hierarchical structure that may be easier to manipulate and
extract data from.

Cheers,
Ian
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[John Salerno]

On Tuesday, March 6, 2012 5:05:39 PM UTC-6, John Salerno wrote:
> > Anything that allows me NOT to use REs is welcome news, so I look forward 
> > to learning about something new! :)
> 
> I should ask though...are there alternatives already bundled with Python that 
> I could use? Now that you mention it, I remember something called HTMLParser 
> (or something like that) and I have no idea why I never looked into that 
> before I messed with REs.
> 
> Thanks.

::sigh:: I'm having some trouble with the new Google Groups interface. It seems 
to double post, and in this case didn't post at all. If it did already, I 
apologize. I'll try to figure out what's happening, or just switch to a real 
newsgroup program.

Anyway, my question was about Beautiful Soup. I read on the doc page that BS 
uses a parser, which html.parser and lxml are. So I'm guessing the difference 
between them is that the parser is a little more "low level," whereas BS offers 
a higher level approach to using them? Is BS easier to write code with, while 
still using the power of lxml?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: are int, float, long, double, side-effects of computer engineering?

[Calvin Kim]

On 03/06/2012 01:34 AM, Xah Lee wrote:

while what you said is true, but the problem is that 99.99% of
programers do NOT know this. They do not know Mathematica. They've
never seen a language with such feature. The concept is alien. This is
what i'd like to point out and spread awareness.

I can see your point. But that's not simply true. In my case and many 
others, such issue was addressed during first week of introductory 
programming classes. I was naively thought "computer = precision" and I 
was stunned to find out the inaccuracy of computer calculations.


But as you experienced, I also stumble upon some people (specially Java 
only programmers) who were not aware of it.



also, argument about raw speed and fine control vs automatic
management, rots with time. Happened with auto memory management,
managed code, compilers, auto type conversion, auto extension of
array, auto type system, dynamic/scripting languages, etc.
Maybe it's because I'm not in scientific community, that I learned to 
live with such side-effects. Because 99.99% of computer users and 
programmers can afford to, and willing to lose such small inaccuracy 
billion times in exchange for some performance increase and convenience. 
Although NASA may not accept my application for their projects for Mars 
mission after this posting.




--
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[John Salerno]

> Anything that allows me NOT to use REs is welcome news, so I look forward to 
> learning about something new! :)

I should ask though...are there alternatives already bundled with Python that I 
could use? Now that you mention it, I remember something called HTMLParser (or 
something like that) and I have no idea why I never looked into that before I 
messed with REs.

Thanks.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[John Salerno]

On Tuesday, March 6, 2012 4:52:10 PM UTC-6, Chris Rebert wrote:
> On Tue, Mar 6, 2012 at 2:43 PM, John Salerno  wrote:
> > I sort of have to work with what the website gives me (as you'll see 
> > below), but today I encountered an exception to my RE. Let me just give all 
> > the specific information first. The point of my script is to go to the 
> > specified URL and extract song information from it.
> >
> > This is my RE:
> >
> > song_pattern = re.compile(r'([0-9]{1,2}:[0-9]{2} 
> > [a|p].m.).*?(.*?).*?(.*?)', re.DOTALL)
> 
> I would advise against using regular expressions to "parse" HTML:
> http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags
> 
> lxml is a popular choice for parsing HTML in Python: http://lxml.de
> 
> Cheers,
> Chris

Thanks, that was an interesting read :)

Anything that allows me NOT to use REs is welcome news, so I look forward to 
learning about something new! :)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to write this regular expression?

[Chris Rebert]

On Tue, Mar 6, 2012 at 2:43 PM, John Salerno  wrote:
> I sort of have to work with what the website gives me (as you'll see below), 
> but today I encountered an exception to my RE. Let me just give all the 
> specific information first. The point of my script is to go to the specified 
> URL and extract song information from it.
>
> This is my RE:
>
> song_pattern = re.compile(r'([0-9]{1,2}:[0-9]{2} 
> [a|p].m.).*?(.*?).*?(.*?)', re.DOTALL)

I would advise against using regular expressions to "parse" HTML:
http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags

lxml is a popular choice for parsing HTML in Python: http://lxml.de

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pickle/unpickle class which has changed

[Peter Otten]

Neal Becker wrote:

> Peter Otten wrote:
> 
>> Steven D'Aprano wrote:
>> 
>>> On Tue, 06 Mar 2012 07:34:34 -0500, Neal Becker wrote:
>>> 
 What happens if I pickle a class, and later unpickle it where the class
 now has added some new attributes?
>>> 
>>> Why don't you try it?
>>> 
>>> py> import pickle
>>> py> class C:
>>> ... a = 23
>>> ...
>>> py> c = C()
>>> py> pickled = pickle.dumps(c)
>>> py> C.b = 42  # add a new class attribute
>>> py> d = pickle.loads(pickled)
>>> py> d.a
>>> 23
>>> py> d.b
>>> 42
>>> 
>>> 
>>> Unless you mean something different from this, adding attributes to the
>>> class is perfectly fine.
>>> 
>>> But... why are you dynamically adding attributes to the class? Isn't
>>> that rather unusual?
>> 
>> The way I understand the problem is that an apparently
>> backwards-compatible change like adding a third dimension to a point with
>> an obvious default breaks when you restore an "old" instance in a script
>> with the "new" implementation:
>> 
> import pickle
> class P(object):
>> ... def __init__(self, x, y):
>> ... self.x = x
>> ... self.y = y
>> ... def r2(self):
>> ... return self.x*self.x + self.y*self.y
>> ...
> p = P(2, 3)
> p.r2()
>> 13
> s = pickle.dumps(p)
> class P(object):
>> ... def __init__(self, x, y, z=0):
>> ... self.x = x
>> ... self.y = y
>> ... self.z = z
>> ... def r2(self):
>> ... return self.x*self.x + self.y*self.y + self.z*self.z
>> ...
> p = P(2, 3)
> p.r2()
>> 13
> pickle.loads(s).r2()
>> Traceback (most recent call last):
>>   File "", line 1, in 
>>   File "", line 7, in r2
>> AttributeError: 'P' object has no attribute 'z'
>> 
>> By default pickle doesn't invoke __init__() and updates __dict__
>> directly. As pointed out in my previous post one way to fix the problem
>> is to implement a __setstate__() method:
>> 
> class P(object):
>> ... def __init__(self, x, y, z=0):
>> ... self.x = x
>> ... self.y = y
>> ... self.z = z
>> ... def r2(self):
>> ... return self.x*self.x + self.y*self.y + self.z*self.z
>> ... def __setstate__(self, state):
>> ... self.__dict__["z"] = 42 # stupid default
>> ... self.__dict__.update(state)
>> ...
> pickle.loads(s).r2()
>> 1777
>> 
>> This keeps working with pickles of the new implementation of P:
>> 
> q = P(3, 4, 5)
> pickle.loads(pickle.dumps(q)).r2()
>> 50
> 
> So if in my new class definition there are now some new attributes, and if
> I did not add a __setstate__ to set the new attributes, I guess then when
> unpickled the instance of the class will simply lack those attributes?

I don't know. If you don't trust the demo try it yourself with the actual 
code you have. Throwing in

obj = pickle.load(...)
print vars(obj)

should help.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help me with weird logging problem

[J]

On Tue, Mar 6, 2012 at 11:19, Vinay Sajip  wrote:
> On Mar 6, 4:09 pm, J  wrote:
>>
>> Any idea what I'm doing wrong?
>
> Levels can be set on loggers as well as handlers, and you're only
> setting levels on the handlers. The default level on the root logger
> is WARNING. A logger checks its level first, and only if the event
> passes that test will it be passed to the handlers (which will also
> perform level tests).
>
> So, a logger.setLevel(logging.DEBUG) should be all you need to add
> before logging anything.
>
> Regards,
>
> Vinay Sajip
> --
> http://mail.python.org/mailman/listinfo/python-list

Thank you very much, Vinay :)

I thought it was something simple like that I had overlooked or misunderstood.

Cheers,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pickle/unpickle class which has changed

[Neal Becker]

Peter Otten wrote:

> Steven D'Aprano wrote:
> 
>> On Tue, 06 Mar 2012 07:34:34 -0500, Neal Becker wrote:
>> 
>>> What happens if I pickle a class, and later unpickle it where the class
>>> now has added some new attributes?
>> 
>> Why don't you try it?
>> 
>> py> import pickle
>> py> class C:
>> ... a = 23
>> ...
>> py> c = C()
>> py> pickled = pickle.dumps(c)
>> py> C.b = 42  # add a new class attribute
>> py> d = pickle.loads(pickled)
>> py> d.a
>> 23
>> py> d.b
>> 42
>> 
>> 
>> Unless you mean something different from this, adding attributes to the
>> class is perfectly fine.
>> 
>> But... why are you dynamically adding attributes to the class? Isn't that
>> rather unusual?
> 
> The way I understand the problem is that an apparently backwards-compatible
> change like adding a third dimension to a point with an obvious default
> breaks when you restore an "old" instance in a script with the "new"
> implementation:
> 
 import pickle
 class P(object):
> ... def __init__(self, x, y):
> ... self.x = x
> ... self.y = y
> ... def r2(self):
> ... return self.x*self.x + self.y*self.y
> ...
 p = P(2, 3)
 p.r2()
> 13
 s = pickle.dumps(p)
 class P(object):
> ... def __init__(self, x, y, z=0):
> ... self.x = x
> ... self.y = y
> ... self.z = z
> ... def r2(self):
> ... return self.x*self.x + self.y*self.y + self.z*self.z
> ...
 p = P(2, 3)
 p.r2()
> 13
 pickle.loads(s).r2()
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 7, in r2
> AttributeError: 'P' object has no attribute 'z'
> 
> By default pickle doesn't invoke __init__() and updates __dict__ directly.
> As pointed out in my previous post one way to fix the problem is to
> implement a __setstate__() method:
> 
 class P(object):
> ... def __init__(self, x, y, z=0):
> ... self.x = x
> ... self.y = y
> ... self.z = z
> ... def r2(self):
> ... return self.x*self.x + self.y*self.y + self.z*self.z
> ... def __setstate__(self, state):
> ... self.__dict__["z"] = 42 # stupid default
> ... self.__dict__.update(state)
> ...
 pickle.loads(s).r2()
> 1777
> 
> This keeps working with pickles of the new implementation of P:
> 
 q = P(3, 4, 5)
 pickle.loads(pickle.dumps(q)).r2()
> 50

So if in my new class definition there are now some new attributes, and if I 
did 
not add a __setstate__ to set the new attributes, I guess then when unpickled 
the instance of the class will simply lack those attributes?

-- 
http://mail.python.org/mailman/listinfo/python-list

Monthly Python Meeting in Madrid (Spain)

[Jesus Cea]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Next Thursday, 8th March.



- -- 
Jesus Cea Avion _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
jabber / xmpp:j...@jabber.org _/_/_/_/  _/_/_/_/_/
.  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBT1Y7SJlgi5GaxT1NAQKOmQQAmRSz5Kk1ZAE5iEBiUiB5XRKVVntPfcA1
FflzHu2ZawULVlcnLMj2mh6USzOSqrRqz5mFZA9RFQWjeN6s1wa/x8hUytUFH90t
BirdqeLjzZMoU1eyRlGggSjqR+VLDqqpxFq8aWbKDC3+t5u+UmZMjvHQo0zBGbcZ
CDcITqWR2Ds=
=e495
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help me with weird logging problem

[Vinay Sajip]

On Mar 6, 4:09 pm, J  wrote:
>
> Any idea what I'm doing wrong?

Levels can be set on loggers as well as handlers, and you're only
setting levels on the handlers. The default level on the root logger
is WARNING. A logger checks its level first, and only if the event
passes that test will it be passed to the handlers (which will also
perform level tests).

So, a logger.setLevel(logging.DEBUG) should be all you need to add
before logging anything.

Regards,

Vinay Sajip
-- 
http://mail.python.org/mailman/listinfo/python-list

Help me with weird logging problem

[J]

Hi,

I'm trying to add a couple log handlers to a program.  The end goal is
to log things at INFO or above to console, and if a -v option is set
to ALSO log everything at DEBUG or above to a file.  However, while I
DO get the log file created, and log messages are appearing there,
Debug messages are not...

Here's some sample code that is essentially the guts of the logging
setup right now, based on what I read in the docs:


#!/usr/bin/python

import logging
import sys

verbose = True

# Set up the logger
console_format = '%(levelname)-8s %(message)s'
console_handler = logging.StreamHandler()
console_handler.setFormatter(logging.Formatter(console_format))
console_handler.setLevel(logging.INFO)
logger = logging.getLogger()
logger.addHandler(console_handler)
if verbose:
verbose_format = '%(asctime)s %(levelname)-8s %(message)s'
verbose_handler = logging.FileHandler('testlog.log')
verbose_handler.setLevel(logging.DEBUG)
verbose_handler.setFormatter(logging.Formatter(verbose_format))
logger.addHandler(verbose_handler)
logging.debug("Setting log level to DEBUG for verbosity")

logging.info("INFO event logged")
logging.warning("WARNING event logged")
logging.error("ERROR event logged")
logging.critical("CRITICAL event logged")
logging.debug("DEBUG event logged")

When I run this I get the following console and log file output:

bladernr@klaatu:~/development/cert-submit-script-improvements/bin$ ./logtest.py
WARNING  WARNING event logged
ERRORERROR event logged
CRITICAL CRITICAL event logged
bladernr@klaatu:~/development/cert-submit-script-improvements/bin$ cat
testlog.log
2012-03-06 11:05:40,297 WARNING  WARNING event logged
2012-03-06 11:05:40,297 ERRORERROR event logged
2012-03-06 11:05:40,298 CRITICAL CRITICAL event logged

So I AM getting logging, but I am not getting the DEBUG level logs in
the log file, nor am I getting INFO level logs on console.

Any idea what I'm doing wrong?
-- 
http://mail.python.org/mailman/listinfo/python-list

python simple web server

[queency jones]

hello pythonist

i'm developing using the simple / basic http server
i got very bad performance regarding to request time .
9 sec for each request replay.

i tried to test this with this:
python -m SimpleHTTPServer 8080
but no better.

any sugestions ? that i can use ?
my final goal is to serv 5 people on the lan network only .
can't i stick with the python server ?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pickle/unpickle class which has changed

[Peter Otten]

Steven D'Aprano wrote:

> On Tue, 06 Mar 2012 07:34:34 -0500, Neal Becker wrote:
> 
>> What happens if I pickle a class, and later unpickle it where the class
>> now has added some new attributes?
> 
> Why don't you try it?
> 
> py> import pickle
> py> class C:
> ... a = 23
> ...
> py> c = C()
> py> pickled = pickle.dumps(c)
> py> C.b = 42  # add a new class attribute
> py> d = pickle.loads(pickled)
> py> d.a
> 23
> py> d.b
> 42
> 
> 
> Unless you mean something different from this, adding attributes to the
> class is perfectly fine.
> 
> But... why are you dynamically adding attributes to the class? Isn't that
> rather unusual?

The way I understand the problem is that an apparently backwards-compatible 
change like adding a third dimension to a point with an obvious default 
breaks when you restore an "old" instance in a script with the "new" 
implementation:

>>> import pickle
>>> class P(object):
... def __init__(self, x, y):
... self.x = x
... self.y = y
... def r2(self):
... return self.x*self.x + self.y*self.y
... 
>>> p = P(2, 3)
>>> p.r2()
13
>>> s = pickle.dumps(p)
>>> class P(object):
... def __init__(self, x, y, z=0):
... self.x = x
... self.y = y
... self.z = z
... def r2(self):
... return self.x*self.x + self.y*self.y + self.z*self.z
... 
>>> p = P(2, 3)
>>> p.r2()
13
>>> pickle.loads(s).r2()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 7, in r2
AttributeError: 'P' object has no attribute 'z'

By default pickle doesn't invoke __init__() and updates __dict__ directly.
As pointed out in my previous post one way to fix the problem is to 
implement a __setstate__() method:

>>> class P(object):
... def __init__(self, x, y, z=0):
... self.x = x
... self.y = y
... self.z = z
... def r2(self):
... return self.x*self.x + self.y*self.y + self.z*self.z
... def __setstate__(self, state):
... self.__dict__["z"] = 42 # stupid default
... self.__dict__.update(state)
... 
>>> pickle.loads(s).r2()
1777

This keeps working with pickles of the new implementation of P:

>>> q = P(3, 4, 5)
>>> pickle.loads(pickle.dumps(q)).r2()
50


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Error with co_filename when loading modules from zip file

[Peter Otten]

Bob Rossi wrote:

> On Tue, Mar 06, 2012 at 02:38:50AM -0800, Vinay Sajip wrote:
>> On Mar 6, 2:40 am, Bob Rossi  wrote:
>> 
>> > Darn it, this was reported in 2007
>> > http://bugs.python.org/issue1180193
>> > and it was mentioned the logging package was effected.
>> >
>> > Yikes.
>> >
>> 
>> I will think about this, but don't expect any quick resolution :-( I
>> think the right fix would be not in the logging package, but in the
>> module loading machinery (as mentioned on that issue).
>> 
>> I wouldn't worry about the performance aspect - once the logging
>> package is loaded, there's no performance impact. That's a tiny one-
>> off hit which you will probably not notice at all.
> 
> OK.
> 
> Do you know where the bytecode gets stored when you load a py
> file from a zip?
> 
> My program can potentially run for hours, from an embedded context,
> and could call into the logger and other py files over and over.
> 
> Are the bytecode files stored in RAM one time, or recomputed each
> time they are needed?

The bytecode is generated once when the module is loaded and kept as part of 
the module object in the sys.modules cache unless you explicitly reload() 
the module. For a long-running program the compilation overhead is 
negligable.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pickle/unpickle class which has changed

[Steven D'Aprano]

On Tue, 06 Mar 2012 07:34:34 -0500, Neal Becker wrote:

> What happens if I pickle a class, and later unpickle it where the class
> now has added some new attributes?

Why don't you try it?

py> import pickle
py> class C:
... a = 23
...
py> c = C()
py> pickled = pickle.dumps(c)
py> C.b = 42  # add a new class attribute
py> d = pickle.loads(pickled)
py> d.a
23
py> d.b
42


Unless you mean something different from this, adding attributes to the 
class is perfectly fine.

But... why are you dynamically adding attributes to the class? Isn't that 
rather unusual?


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pickle/unpickle class which has changed

[Peter Otten]

Neal Becker wrote:

> What happens if I pickle a class, and later unpickle it where the class
> now has added some new attributes?

- If the added attributes' values are immutable, provide defaults as class 
attributes.

- Implement an appropriate __setstate__() method. The easiest would be

# untested
def __setstate__(self, state):
self.__dict__.update(newattr1=42, newattr2=[])
self.__dict__.update(state)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: are int, float, long, double, side-effects of computer engineering?

[Roy Smith]

[intentionally violating the followup-to header]

In article <7ol5r.29957$zd5.14...@newsfe12.iad>,
 Chiron  wrote:

> On Mon, 05 Mar 2012 22:34:46 -0800, Xah Lee wrote:
> 
> > while what you said is true, but the problem is that 99.99% of
> > programers do NOT know this. They do not know Mathematica. They've never
> > seen a
> 
> Could you please offer some evidence to support this claim?  Most of the 
> programmers I've ever run into, were quite familiar with the notion that 
> many aspects of their languages were artifacts of hardware limitations.

While I doubt Xah's claim that 99.99% of programmers are unaware of 
this, oddly enough, we ran across an instance of this just yesterday.  
We have a test problem we give to job candidates:

> Write a program that counts the number of times the subsequence "1, 2, 3" 
> appears in the first 100,000 digits of pi. Note "subsequence" does not imply 
> consecutive digits. For example, "1, 2, 3" appears in "3.141592653" twice. 
> You may use Google or the link above to find the digits of pi. Please include 
> the correct count when submitting your application.

We had a candidate submit a solution in Python which ran in O(n) time.  
The algorithm was essentially correct, but unfortunately, he used a 
numpy.array to hold some counters, and suffered integer overflow.  Had 
he used regular python lists, he would have been fine.

What's unclear (at the moment) is whether he falls into Xah's camp of 
people who are unaware of such issues (unlikely, IMHO).  Possibly he 
just didn't realize that the numbers in this problem might get big 
enough to overflow a 32-bit int.  Or had been lulled into a false sense 
of security knowing that python does arbitrary precision integer math 
and didn't realize that doesn't extend to numpy.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Error with co_filename when loading modules from zip file

[Bob Rossi]

On Tue, Mar 06, 2012 at 02:38:50AM -0800, Vinay Sajip wrote:
> On Mar 6, 2:40 am, Bob Rossi  wrote:
> 
> > Darn it, this was reported in 2007
> >  http://bugs.python.org/issue1180193
> > and it was mentioned the logging package was effected.
> >
> > Yikes.
> >
> 
> I will think about this, but don't expect any quick resolution :-( I
> think the right fix would be not in the logging package, but in the
> module loading machinery (as mentioned on that issue).
> 
> I wouldn't worry about the performance aspect - once the logging
> package is loaded, there's no performance impact. That's a tiny one-
> off hit which you will probably not notice at all.

OK.

Do you know where the bytecode gets stored when you load a py
file from a zip?

My program can potentially run for hours, from an embedded context,
and could call into the logger and other py files over and over.

Are the bytecode files stored in RAM one time, or recomputed each
time they are needed?

Thanks,
Bob
-- 
http://mail.python.org/mailman/listinfo/python-list

a interesting Parallel Programing Problem: asciify-string

[Xah Lee]

here's a interesting problem that we are discussing at comp.lang.lisp.

〈Parallel Programing Problem: asciify-string〉
http://xahlee.org/comp/parallel_programing_exercise_asciify-string.html

here's the plain text. Code example is emacs lisp, but the problem is
general.

for a bit python relevancy… is there any python compiler that's
parallel-algorithm aware?

---
Parallel Programing Problem: asciify-string

Here's a interesting parallel programing problem.

Problem Description

The task is to change this function so it's parallelable. (code
example in emacs lisp)

(defun asciify-string (inputStr)
  "Make Unicode string into equivalent ASCII ones."
  (setq inputStr (replace-regexp-in-string "á\\|à\\|â\\|ä" "a"
inputStr))
  (setq inputStr (replace-regexp-in-string "é\\|è\\|ê\\|ë" "e"
inputStr))
  (setq inputStr (replace-regexp-in-string "í\\|ì\\|î\\|ï" "i"
inputStr))
  (setq inputStr (replace-regexp-in-string "ó\\|ò\\|ô\\|ö" "o"
inputStr))
  (setq inputStr (replace-regexp-in-string "ú\\|ù\\|û\\|ü" "u"
inputStr))
  inputStr
  )

Here's a more general description of the problem.

You are given a Unicode text file that's a few peta bytes. For certain
characters in the file, they need to be changed to different char.
(For example of practical application, see: IDN homograph attack ◇
Duplicate characters in Unicode.)

One easy solution is to simply use regex, as the above sample code, to
search thru the file sequentially, and perform the transfrom of a
particular set of chars, then repeat for each char chat needs to be
changed.

But your task is to use a algorithm parallelizable. That is, in a
parallel-algorithm aware language (e.g. Fortress), the compiler will
automatically span the computation to multiple processors.

Refer to Guy Steele's video talk if you haven't seen already. See: Guy
Steele on Parallel Programing.
Solution Suggestions

A better way to write it for parallel programing, is to map a char-
transform function to each char in the string. Here's a pseudo-code in
lisp by Helmut Eller:

(defun asciify-char (c)
  (case c
((? ? ? ?) ?a)
((? ? ? ?) ?e)
((? ? ? ?) ?i)
((? ? ? ?) ?o)
((? ? ? ?) ?u)
(t c)))

(defun asciify-string (string) (map 'string #'asciify-string string))

One problem with this is that the function “asciify-char” itself is
sequential, and not 100% parallelizable. (we might assume here that
there are billions of chars in Unicode that needs to be transformed)

It would be a interesting small project, if someone actually use a
parallel-algorithm-aware language to work on this problem, and report
on the break-point of file-size of parallel-algorithm vs sequential-
algorithm.

Anyone would try it? Perhaps in Fortress, Erlang, Ease, Alice, X10, or
other? Is the Clojure parallel aware?

 Xah
-- 
http://mail.python.org/mailman/listinfo/python-list

pickle/unpickle class which has changed

[Neal Becker]

What happens if I pickle a class, and later unpickle it where the class now has 
added some new attributes?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Error with co_filename when loading modules from zip file

[Vinay Sajip]

On Mar 6, 2:40 am, Bob Rossi  wrote:

> Darn it, this was reported in 2007
>  http://bugs.python.org/issue1180193
> and it was mentioned the logging package was effected.
>
> Yikes.
>

I will think about this, but don't expect any quick resolution :-( I
think the right fix would be not in the logging package, but in the
module loading machinery (as mentioned on that issue).

I wouldn't worry about the performance aspect - once the logging
package is loaded, there's no performance impact. That's a tiny one-
off hit which you will probably not notice at all.

Regards,

Vinay Sajip
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: are int, float, long, double, side-effects of computer engineering?

[Chiron]

On Mon, 05 Mar 2012 22:34:46 -0800, Xah Lee wrote:

> while what you said is true, but the problem is that 99.99% of
> programers do NOT know this. They do not know Mathematica. They've never
> seen a

Could you please offer some evidence to support this claim?  Most of the 
programmers I've ever run into, were quite familiar with the notion that 
many aspects of their languages were artifacts of hardware limitations.  
You don't need Mathematica to figure out that (10.0 * 0.1) - 1.0 doesn't 
often equal 0.0.  The moment you try such comparisons with floats, you 
figure it out.  Oh, granted - the *first* time you try it, you might 
spend days trying to understand what's wrong.  But having done that, you 
will never, ever fail to understand about the evils of computer 
engineering.

Anyway, most programmers probably get burned like this early on, if they 
forget that numeric representations in most languages are inaccurate.  
They don't need Mathematica to help them understand.

BTW, for those who don't have access to Mathematica, I highly recommend 
sagemath.  I have no way of making a comparison between the two (I have 
no access to Mathematica), but sagemath is mature, useful, and fast.

-- 
You will be singled out for promotion in your work.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: are int, float, long, double, side-effects of computer engineering?

[Chiron]

On Mon, 05 Mar 2012 17:11:09 -0800, Xah Lee wrote:

Yes.

Why do you ask?  Is this not obvious?

Was this a rhetorical question?

-- 
A girl with a future avoids the man with a past.
-- Evan Esar, "The Humor of Humor"
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: are int, float, long, double, side-effects of computer engineering?

[Russ P.]

On Mar 5, 10:34 pm, Xah Lee  wrote:
> On Mar 5, 9:26 pm, Tim Roberts  wrote:
>
> > Xah Lee  wrote:
>
> > >some additional info i thought is relevant.
>
> > >are int, float, long, double, side-effects of computer engineering?
>
> > Of course they are.  Such concepts violate the purity of a computer
> > language's abstraction of the underlying hardware.  We accept that
> > violation because of performance reasons.  There are, as you point out,
> > languages that do maintain the purity of the abstraction, but that purity
> > is ALWAYS at the expense of performance.
>
> > I would also point out pre-emptively that there is nothing inherently wrong
> > with asking us to accept an impure abstraction in exchange for performance.
> > It is a performance choice that we choose to make.
>
> while what you said is true, but the problem is that 99.99% of
> programers do NOT know this. They do not know Mathematica. They've
> never seen a language with such feature. The concept is alien. This is
> what i'd like to point out and spread awareness.

I seriously doubt that. I think most decent programmers are well aware
of the limitations of floating point math. If properly used, double-
precision arithmetic is more than adequate for the vast majority of
practical scientific and engineering problems.

> also, argument about raw speed and fine control vs automatic
> management, rots with time. Happened with auto memory management,
> managed code, compilers, auto type conversion, auto extension of
> array, auto type system, dynamic/scripting languages, etc.

First of all, "dynamic/scripting languages" are still a long, long way
from being "fast enough" for computationally intensive applications.

Secondly, nothing is stopping anyone from writing a library to
implement rational numbers or infinite-precision arithmetic in python
(or just about any other language). They are just not needed for most
applications.
-- 
http://mail.python.org/mailman/listinfo/python-list