[issue33827] Generators with lru_cache can be non-intuituve

[Michel Albert]

New submission from Michel Albert :

Consider the following code:

# filename: foo.py

from functools import lru_cache


@lru_cache(10)
def bar():
yield 10
yield 20
yield 30


# This loop will work as expected
for row in bar():
print(row)

# This loop will not loop over anything.
# The cache will return an already consumed generator.
for row in bar():
print(row)


This behaviour is natural, but it is almost invisible to the caller of "foo".

The main issue is one of "surprise". When inspecting the output of "foo" it is 
clear that the output is a generator:

>>> import foo
>>> foo.bar()


**Very** careful inspection will reveal that each call will return the same 
generator instance.

So to an observant user the following is an expected behaviour:

>>> result = foo.bar()
>>> for row in result:
...print(row)
...
10
20
30
>>> for row in result:
... print(row)
...
>>>

However, the following is not:

>>> import foo
>>> result = foo.bar()
>>> for row in result:
... print(row)
...
10
20
30
>>> result = foo.bar()
>>> for row in result:
... print(row)
...
>>>


Would it make sense to emit a warning (or even raise an exception) in 
`lru_cache` if the return value of the cached function is a generator?

I can think of situation where it makes sense to combine the two. For example 
the situation I am currently in:

I have a piece of code which loops several times over the same SNMP table. 
Having a generator makes the application far more responsive. And having the 
cache makes it even faster on subsequent calls. But the gain I get from the 
cache is bigger than the gain from the generator. So I would be okay with 
converting the result to a list before storing it in the cache.

What is your opinion on this issue? Would it make sense to add a warning?

--
messages: 319279
nosy: exhuma
priority: normal
severity: normal
status: open
title: Generators with lru_cache can be non-intuituve
type: behavior

___
Python tracker 
<https://bugs.python.org/issue33827>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for "ip_network in ip_network"

[Michel Albert]

Michel Albert added the comment:

Are there any updates on this? Not sure if it's too late again to get it 
applied for the next Python (3.6) release?

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue20825>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for "ip_network in ip_network"

[Michel Albert]

Michel Albert added the comment:

New patch with proposed changes.

--
Added file: http://bugs.python.org/file43537/net-in-net-r6.patch

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue20825>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for "ip_network in ip_network"

[Michel Albert]

Michel Albert added the comment:

Updated patch, taking into account notes from the previous patch-reviews

--
Added file: http://bugs.python.org/file43535/net-in-net-r5.patch

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue20825>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for "ip_network in ip_network"

[Michel Albert]

Michel Albert added the comment:

Test pass properly.

Is there anything else left to do?

Here's the fixed patch (net-in-net-r4.patch)

--
Added file: http://bugs.python.org/file43534/net-in-net-r4.patch

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue20825>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for "ip_network in ip_network"

[Michel Albert]

Michel Albert added the comment:

I just realised that the latest patch on this no longer applies properly. I 
have fixed the issue and I am currently in the process of running the 
unit-tests which takes a while. Once those pass, I'll update some metadata and 
resubmit.

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue20825>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20815] ipaddress unit tests PEP8

[Michel Albert]

Michel Albert added the comment:

It seems the contributor agreement form has been processed. As I understand it, 
the asterisk on my name confirms this.

I also verified that this patch cleanly applies to the most recent revision.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20815
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for ip_network in ip_network

[Michel Albert]

Michel Albert added the comment:

Hi again,

The contribution agreement has been processed, and the patch still cleanly 
applies to the latest revision of branch `default`.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20825
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20826] Faster implementation to collapse consecutive ip-networks

[Michel Albert]

Michel Albert added the comment:

Sorry for the late reply. I wanted to take some time and give a more detailed 
explanation. At least to the best of my abilities :)

I attached a zip-file with my quick-and-dirty test-rig. The zip contains:

 * gendata.py -- The script I used to generate test-data
 * testdata.lst -- My test-data set (for reproducability)
 * tester.py -- A simple script using ``timeit.timeit``.

I am not sure how sensitive the data is I am working with, so I prefer not to 
put any of the real data on a public forum. Instead, I wrote a small script 
which generates a data-set which makes the performance difference visible 
(``gendata.py``). The data which I processed actually created an even worse 
case, but it's difficult to reproduce. In my case, the data-set I used is in 
the file named ``testdata.lst``.

I then ran the operation 5 times using ``timeit`` (tester.py).

Let me also outline an explanation to what it happening:

It is possible, that through one merge operation, a second may become 
possible. For the sake of readability, let's use IPv4 addresses, and consider 
the following list:

[a_1, a_2, ..., a_n, 192.168.1.0/31, 192.168.1.2/32, 192.168.1.3/32, b_1, 
b_2, ..., b_n]

This can be reduced to

[a_1, a_2, ..., a_n, 192.168.1.0/31, 192.168.1.2/31, b_1, b_2, ..., b_n]

Which in turn can then be reduced to:

[a_1, a_2, ..., a_n, 192.168.1.0/30, b_1, b_2, ..., b_n]

The current implementation, sets a boolean (``optimized``) to ``True`` if any 
merge has been performed. If yes, it re-runs through the whole list until no 
optimisation is done. Those re-runs also include [a1..an] and [b1..bn], which 
is unnecessary. With the given data-set, this gives the following result:

Execution time: 48.27790632040014 seconds
./python tester.py  244.29s user 0.06s system 99% cpu 4:04.51 total

With the shift/reduce approach, only as many nodes are visited as necessary. If 
a reduce is made, it backtracks as much as possible, but not further. So in 
the above example, nodes [a1..an] will only be visited once, and [b1..bn] will 
only be visited once the complete optimisation of the example addresses has 
been performed. With the given data-set, this gives the following result:

Execution time: 20.298685277199912 seconds
./python tester.py  104.20s user 0.14s system 99% cpu 1:44.58 total

If my thoughts are correct, both implementations should have a similar 
best-case, but the worst-case differs significantly. I am not well-versed 
with the Big-O notation, especially the best/average/worst case difference. 
Neither am I good at math. But I believe both are strictly speaking O(n). But 
more precisely, O(k*n) where k is proportional the number of reconciliation 
steps needed (that is, if one merger makes another merger possible). But it is 
much smaller in the shift/reduce approach as only as many elements need to be 
revisited as necessary, instead of all of them.

--
Added file: http://bugs.python.org/file34583/testrig.zip

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20826
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for ip_network in ip_network

[Michel Albert]

Michel Albert added the comment:

I made the changes mentioned by r.david.murray

I am not sure if the modifications in ``Doc/whatsnew/3.5.rst`` are correct. I 
tried to follow the notes at the top of the file, but it's not clear to me if 
it should have gone into ``News/Misc`` or into ``Doc/whatsnew/3.5.rst``.

On another side-note: I attached this as an ``-r3`` file, but I could have 
replaced the existing patch as well. Which method is preferred? Replacing 
existing patches on the issue or adding new revisions?

--
Added file: http://bugs.python.org/file34588/net-in-net-r3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20825
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20815] ipaddress unit tests PEP8

[Michel Albert]

Michel Albert added the comment:

Did so already last weekend. I suppose it will take some time to be processed.

I can ping you via a message here once I receive the confirmation.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20815
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for ip_network in ip_network

[Michel Albert]

Michel Albert added the comment:

Yes. I signed it last Friday if I recall correctly.

As I understood it, the way for you to tell if it's done, is that my username 
will be followed by an asterisk.

But I'm not in a hurry. Once I get the confirmation, I can just ping you again 
via a comment here, so you don't need to monitor it yourself.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20825
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for ip_network in ip_network

[Michel Albert]

Michel Albert added the comment:

Here's a new patch implementing both ``subnet_of`` and ``supernet_of``.

It also contains the relevant docs and unit-tests.

--
Added file: http://bugs.python.org/file34292/net-in-net-r2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20825
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for ip_network in ip_network

[Michel Albert]

Michel Albert added the comment:

I second supernet_of and subnet_of. I'll implement it as soon as I get 
around it.

I have been thinking about using ``in`` and ``=`` and, while I initially liked 
the idea for tests, I find both operators too ambiguous.

With ``in`` there's the already mentioned ambiguity of containment/inclusion. 
And ``=`` could mean is a smaller size (has less individual hosts), but could 
also mean that it is a subnet, or even that it is located to the left.

Naming it ``subnet_of`` makes it 100% clear what it does.

Currently, ``a = b`` returns ``True`` if a comes before/lies on the left of b.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20825
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20815] ipaddress unit tests PEP8

[Michel Albert]

Michel Albert added the comment:

I strongly agree with Raymond's points! They are all valid.

I should note, that I submitted this patch to - as mentioned by Nick - 
familiarise myself with the patch submission process. I decided to make 
harmless changes which won't risk braking anything.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20815
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20815] ipaddress unit tests PEP8

[Michel Albert]

Michel Albert added the comment:

Here's a new patch which addresses white-space issues without touching the old 
tests.

--
Added file: http://bugs.python.org/file34265/test_ipaddress_pep8-r3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20815
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for ip_network in ip_network

[Michel Albert]

New submission from Michel Albert:

The ipaddress module always returns ``False`` when testing if a network is 
contained in another network. However, I feel this should be a valid test. No? 
Is there any reason why this is fixed to ``False``?

In case not, here's a patch which implements this test.

Note that by design, IP addresses networks can never overlap half-way. In 
cases where this should return ``False``, you either have a network that lies 
completely to the left, or completely to the right. In the case it should 
return ``True`` the smaller network is always completely bounded by the larger 
network's network- and broadcast address.

I needed to change two containment tests as they were in conflict with this 
change. These tests were ``self.v6net not in self.v6net`` and ``self.v4net not 
in self.v4net``. The reason these two failed is that the new containment test 
checks the target range *including* broadcast and network address. So ``a in 
a`` always returns true.

This could be changed by excluding one of the two boundaries, and by that 
forcing the containee to be smaller than the container. But it would make 
the check a bit more complex, as you would need to add an exception for the 
case that both are identical.

Backwards compatibility is a good question. Strictly put, this would break it. 
However, I can't think of any reason why anyone would expect ``a in a`` to be 
false in the case of IP-Addresses.

Just as a side-note, I currently work at our national network provider and am 
currently implementing a tool dealing with a lot of IP-Addresses. We have run 
into the need to test ``net in net`` a couple of times and ran into bugs 
because the stdlib returns ``False`` where you technically expect it to be 
``True``.

--
components: Library (Lib)
files: net-in-net.patch
keywords: patch
messages: 212550
nosy: exhuma, ncoghlan, pmoody
priority: normal
severity: normal
status: open
title: containment test for ip_network in ip_network
versions: Python 3.5
Added file: http://bugs.python.org/file34266/net-in-net.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20825
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20825] containment test for ip_network in ip_network

[Michel Albert]

Michel Albert added the comment:

Hmm... after thinking about this, I kind of agree. I was about to state 
something about the fact that you could consider networks like an ordered 
set. And use that to justify my addition :) But the more I think about it, the 
more I am okay with your point.

I quickly tested the following:

 a = ip_network('10.0.0.0/24')
 b = ip_network('10.0.0.0/30')
 a = b
True
 b = a
False

Which is wrong when considering containement.

What about an instance-method? Something like ``b.contained_in(a)``?

At least that would be explicit and avoids confusion. Because the existing 
``__lt__`` implementation makes sense in the way it's already used.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20825
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20826] Faster implementation to collapse consecutive ip-networks

[Michel Albert]

New submission from Michel Albert:

This alternative implementation runs over the ``addresses`` collection only 
once, and backtracks only if necessary. Inspired by a shift-reduce approach.

Technically both are O(n), so the best case is always the same. But the old 
implementation runs over the *complete* list multiple times until it cannot 
make any more optimisations. The new implementation only repeats the 
optimisation on elements which require reconciliation.

Tests on a local machine have shown a considerable increase in speed on large 
collections of elements (iirc about twice as fast on average).

--
components: Library (Lib)
files: faster-collapse-addresses.patch
keywords: patch
messages: 212553
nosy: exhuma, ncoghlan, pmoody
priority: normal
severity: normal
status: open
title: Faster implementation to collapse consecutive ip-networks
type: performance
versions: Python 3.5
Added file: http://bugs.python.org/file34267/faster-collapse-addresses.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20826
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20815] ipaddress unit tests PEP8

[Michel Albert]

New submission from Michel Albert:

While I was looking at the source of the ipaddress unit-tests, I noticed a 
couple of PEP8 violations. This patch fixes these (verified using the ``pep8`` 
tool).

There are no behavioural changes. Only white-space.

Test-cases ran successfully before, and after the change was made.

--
components: Tests
files: test_ipaddress_pep8.patch
keywords: patch
messages: 212497
nosy: exhuma, ncoghlan, pmoody
priority: normal
severity: normal
status: open
title: ipaddress unit tests PEP8
versions: Python 3.5
Added file: http://bugs.python.org/file34257/test_ipaddress_pep8.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20815
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20815] ipaddress unit tests PEP8

[Michel Albert]

Michel Albert added the comment:

Thanks for the quick reply!

I did not know the pep8 tool added it's own rules :( I have read PEP8 a long 
while ago and have since relied on the tool to do the right thing. Many of 
it's idiosyncrasies have probably made their way into my blood since :(

And you're right: The PEP is actually explicitly saying that it's okay to leave 
out the white-space to make operator precedence more visible (for reference: 
http://legacy.python.org/dev/peps/pep-0008/#other-recommendations).

I will undo those changes.

Is there anything else that immediately caught your eye so I can address it in 
the update?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20815
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

DNS query against a specific server.

[Michel Albert]

Hi,

``socket.gethostbyname`` sends the DNS resolution query to the DNS server 
specified by the OS. Is there an easy way to send a query to a *different* 
server?

I see that twisted.names allows you to do this, but, having all of twisted as 
dependency to my project when all I need to do is a simple DNS query seems a 
bit extreme. I also found pydns, but that looks fairly outdated and 
unmaintained.

Is there not an actively maintained lightweight solution? If not, I will go 
with twisted.


Cheers,
Mich.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: DNS query against a specific server.

[Michel Albert]

On Monday, 30 September 2013 14:36:34 UTC+2, William Ray Wing  wrote:
 On Sep 30, 2013, at 7:42 AM, Michel Albert *** wrote:
 
 
 
  Hi,
 
  
 
  ``socket.gethostbyname`` sends the DNS resolution query to the DNS server 
  specified by the OS. Is there an easy way to send a query to a *different* 
  server?
 
  
 
  I see that twisted.names allows you to do this, but, having all of twisted 
  as dependency to my project when all I need to do is a simple DNS query 
  seems a bit extreme. I also found pydns, but that looks fairly outdated and 
  unmaintained.
 
  
 
  Is there not an actively maintained lightweight solution? If not, I will go 
  with twisted.
 
  
 
  
 
  Cheers,
 
  Mich.
 
  -- 
 
  https://mail.python.org/mailman/listinfo/python-list
 
 
 
 It isn't pure python, but you would be pretty much guaranteed a maintained 
 solution if you use the name server lookup in your OS.  Something like:
 
 
 
   import subprocess
 
   nsl_reslt = subprocess.Popen(['nslookup', 'insert name nere' ],stderr 
 = subprocess.PIPE, stdout = subprocess.PIPE).communicate()[0]
 
 
 
 
 
 Hope this helps,
 
 Bill

Hmm... I had this option in mind, but opening a subprocess for something as 
small as this seemed a bit error-prone. If something on the system changes, 
nslookup replaced by dig or nslookup output changes for example your 
application will bug out.

Granted, the chance of this happening is slim, but using a fixed-version 
dependency in your setup script gives you a much safer solution IMO.

I know I may be splitting hairs. Any of the mentioned solutions are fine. But I 
am curious to see if something like this is not yet implemented in a more 
standard way. I was surprised to see that ``gethostbyname`` does not take an 
optional parameter for this task.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: DNS query against a specific server.

[Michel Albert]

On Monday, 30 September 2013 14:54:41 UTC+2, Ervin Hegedüs  wrote:
 Hello,
 
 
 
 On Mon, Sep 30, 2013 at 04:42:29AM -0700, Michel Albert wrote:
 
  Hi,
 
  
 
  ``socket.gethostbyname`` sends the DNS resolution query to the DNS server 
  specified by the OS. Is there an easy way to send a query to a *different* 
  server?
 
  
 
  I see that twisted.names allows you to do this, but, having all of twisted 
  as dependency to my project when all I need to do is a simple DNS query 
  seems a bit extreme. I also found pydns, but that looks fairly outdated and 
  unmaintained.
 
  
 
  Is there not an actively maintained lightweight solution? If not, I will go 
  with twisted.
 
  
 
 
 
 there is a dns modul for Python (I don't know is it part of
 
 standard Python library or not), on most Linux distribution you
 
 can find it, eg. in Debian it's called python-dnspython.
 
 
 
 It can handle different nameserver, than OS knows - here is a
 
 sample code:
 
 
 
 
 
 import dns.resolver
 
 
 
 r = dns.resolver.Resolver()
 
 r.namerservers =  ['127.0.0.1']
 
 # or any other IP, in my case I'm using PDNS, which have two
 
 # parts: a recursor and a resolver; recursor allows requests only
 
 # on localhost
 
 
 
 mxservers = r.query(python.org, 'MX').response
 
 
 
 
 
 
 
 
 
 hth,
 
 
 
 
 
 a.


Indeed, this looks much nicer than both twisted or pydns. I think I'll go with 
that one. Thanks a lot!
-- 
https://mail.python.org/mailman/listinfo/python-list

How to display ReST properly on pypi?

[Michel Albert]

In general, I write my README files using the ReST syntax. But when I do, they 
don't show up formatted on pypi (see for example 
https://pypi.python.org/pypi/config_resolver/3.3.0).

How do I get it to be formatted properly?

Also, is there a way to specify that the description field in setup.py is 
formatted as ReST?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to display ReST properly on pypi?

[Michel Albert]

Ah... I understand. Makes sense.

I was confused that it worked properly on github, but not on pypi, so I never 
even thought about checking it in rst2html. I thought I would need to specify 
somewhere manually that it was formatted as ReST instead of plain-text.

Thanks for the info.

On Saturday, 14 September 2013 11:53:12 UTC+2, Chris “Kwpolska” Warrick  wrote:
 On Sat, Sep 14, 2013 at 11:08 AM, Michel Albert wrote:
 
  In general, I write my README files using the ReST syntax. But when I do, 
  they don't show up formatted on pypi (see for example 
  https://pypi.python.org/pypi/config_resolver/3.3.0).
 
 
 
  How do I get it to be formatted properly?
 
 
 
  Also, is there a way to specify that the description field in setup.py is 
  formatted as ReST?
 
  --
 
  https://mail.python.org/mailman/listinfo/python-list
 
 
 
 rst2html crashes with your file.  Also, `py:class` is Sphinx-only and
 
 won’t work with the Cheeseshop and produce garbage in the output.
 
 
 
 [kwpolska@kwpolska-lin config_resolver-3.3.0]% rst2html README.rst
 
 README.rst:67: (WARNING/2) Inline emphasis start-string without end-string.
 
 README.rst:108: (ERROR/3) Unknown interpreted text role py:class.
 
 README.rst:115: (ERROR/3) Unknown interpreted text role py:class.
 
 README.rst:121: (ERROR/3) Unknown interpreted text role py:class.
 
 README.rst:142: (SEVERE/4) Problems with include directive path:
 
 InputError: [Errno 2] No such file or directory: 'CHANGES'.
 
 Exiting due to level-4 (SEVERE) system message.
 
 [kwpolska@kwpolska-lin config_resolver-3.3.0]% echo $?
 
 1
 
 [kwpolska@kwpolska-lin config_resolver-3.3.0]%
 
 
 
 -- 
 
 Chris “Kwpolska” Warrick http://kwpolska.tk
 
 PGP: 5EAAEA16
 
 stop html mail | always bottom-post | only UTF-8 makes sense

-- 
https://mail.python.org/mailman/listinfo/python-list

Making `logging.basicConfig` log to *both* `sys.stderr` and `sys.stdout`?

[Michel Albert]

Hi,

I use python oftentimes to write automation scripts on Linux servers.
And there's a big pattern in my scripts:

- I *always* use `logging` instead of `print` statements.
- I *always* create two stream handlers. One for `sys.stdout` with
level `INFO` and one for `sys.stderr` with level `WARN`

Well, the levels may variate occasionally, but that's only the rare
exception.

The reason I do this is simple: Most automation tasks are run via
cron. With this setup, I can redirect `stdout` to `/dev/null` and
still receive e-mails if things go wrong.

And having two handlers gives me more flexibility in my scripts. In
one case, I used a different color for error messages for example as
this script is run primarily from the shell and having errors stand
out has proven to be a good thing.

Unfortunately this setup makes `logging.basicConfig` pretty useless.
However, I believe that this is something that more people could
benefit from. I also believe, that it just makes sense to send
warnings (and above) to `stderr`, the rest to `stdout`.

So I was thinking: Why does `logging.basicConfig` not behave that
way.

Naturally, I was thinking of writing a patch against the python
codebase and submit it as a suggestion. But before doing so, I would
like to hear your thoughts on this. Does it make sense to you too or
am I on the wrong track? Are there any downsides I am missing?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Making `logging.basicConfig` log to *both* `sys.stderr` and `sys.stdout`?

[Michel Albert]

On Aug 30, 11:45 am, Peter Otten __pete...@web.de wrote:
 Michel Albert wrote:
  I use python oftentimes to write automation scripts on Linux servers.
  And there's a big pattern in my scripts:

  - I *always* use `logging` instead of `print` statements.
  - I *always* create two stream handlers. One for `sys.stdout` with
  level `INFO` and one for `sys.stderr` with level `WARN`

  Well, the levels may variate occasionally, but that's only the rare
  exception.

 How would a call to basicConfig() look like that produces this setup?

I personally see this happen by default (i.e. no additional
parameters). And in case the `stream` parameter is set, /then/ you
would  send all to that stream only.

In my point of view, the call to `basicConfig` is either something
used in only the most mundane usages of the logging package, or it's
mainly used by people that have not yet grokked the logging package
(correct me if I'm wrong). In both cases, I find it useful to redirect
warnings and errors to `stderr` by default.

However, this would also mean that existing code calling this method
would result in different behavior. But only /slightly/ different.
Meaning, you still see the output on the console as expected. But it
gives you the possibility to use standard shell redirection in a way
that makes sense.
-- 
http://mail.python.org/mailman/listinfo/python-list

Using a decorator to *remove* parameters from a call

[Michel Albert]

A small foreword: This might look like a cherrypy-oriented post, and
should therefore go to the cherrypy group, but if you read to the end,
you'll see it's a more basic python problem, with cherrypy only as an
example. ;)

From the decorator PEP (318) I get it that you can /add/ parameters to
a call. Say you do something like this:

@mydeco( foo=bar )
def myfunc( hello ):
   print foo, foo

However, this is not what I would like to do. I would like to take
away one or more attributes using a decorator. My goal is to
centralize the logic associated with that parameter. In my particular
example, I am writing a cherrypy application (more specifically,
turbogears1) and I would like all controller method to accept a lang
and a skin attribute. As a simple, but hands-on example, imagine
code like this (http://python.pastebin.com/f25f2429b):

class MyController(Controller):

@expose()
def index(self, skin=default, lang=en):
set_skin( skin )
set_language( lang )
return Hello skinned world

@expose()
def foo(self, skin=default, lang=en):
set_skin( skin )
set_language( lang )
return bar

This becomes cumbersome however for a large application with many
controllers and methods. Always adding the parameters to the methods
and function calls into the method bodies looks way to repetitive for
my taste.

Currently I solve this by using a cherrypy filter which removes those
parameters from the HTTP-Request and puts them into the session
object. This looked wrong to me though right from the start. Also, the
cherrypy docs advise against the use of filters for application
logic. But this looks pretty much like that to me somewhat. Worse
yet, filters are not supported in Turbogears2 any longer. Which makes
porting my app not straight-forward, as I have to port my filters and
write WSGI middleware instead.

I would prefer a syntax like this (http://python.pastebin.com/
f462bc29c)

class MyController(Controller):

@expose()
@skinnable
@translatable
def index(self):
return Hello skinned world

@expose()
@skinnable
@translatable
def foo(self):
return bar

This, to me has several advantages: The logic is encapsulated in the
application code itself. I do not need to rely on framework specific
features (be it cherrypy or wsgi). This would make the application
more self-contained and hence more future proof.

But right here lies my problem. If you are not familiar with CherryPy,
let me digress for a tiny bit to give you the necessary backgroud:

Inside a controller, an exposed method is reachable via HTTP
requests. It's possible to force request parameters. If that is the
case (as in my case it is), then the call will only be successful, if
all parameters received a vaule and no unknown parameters have been
passed. Assume the following signature:

def index(self)

This will successfully return an HTTP Response when the client
requested the resource on the URL /index. If the client adds a query
string, say /index?lang=en the call will *fail* as this parameter is
unkown. For this to work the signature must read:

def index(self, lang)

or

def index(self, lang=en)

Right end of digression, back to topic:

My ultimate question is: Is it possible to write a decorator that
removes a parameter from a call and return a function without that
parameter? Something along the lines (http://python.pastebin.com/
f257877cd):

def translatable(f):
Note that this code is only an example and will not work!
lang = f.__get_parameter_value__(lang)
f.__remove_parameter__(lang)
do_something_with_lang(lang)
return f

I am aware, that this depends on *when* the decorator call is
executed. If its called whenever the decorated function is called,
then there should be some way to handle this. If it's called at
compile-time, then I suppose it's not going to be possible.

A usage scenario would be this:

# --- definition ---
@translatable
def index(self):
   return _(Hello, lang)

# --- call --
obj.index( lang=de)
obj.index()

As you can see, the method definition does not contain the lang
parameter in the signature, but I want to be able to (optionally) set
it during a call.

I hope I made my ideas clear enough. I would be very positively
surprised if this worked. It would make my application code much
easier to read, understand and follow. The filters currently do a lot
of magic behind-the-scenes and if somebody else needs to jump in and
code on that application they will surely ask themselves where the
** is that * lang variable set? :) And I'd like to make it as
easy as possible for everyone :)

Or, maybe you even have a better solution in mind? I'm open for
suggestions.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Using a decorator to *remove* parameters from a call

[Michel Albert]

On Apr 13, 12:52 pm, Jon Clements jon...@googlemail.com wrote:
 On 13 Apr, 11:11, Michel Albert exh...@gmail.com wrote:



  A small foreword: This might look like a cherrypy-oriented post, and
  should therefore go to the cherrypy group, but if you read to the end,
  you'll see it's a more basic python problem, with cherrypy only as an
  example. ;)

  From the decorator PEP (318) I get it that you can /add/ parameters to
  a call. Say you do something like this:

  @mydeco( foo=bar )
  def myfunc( hello ):
     print foo, foo

  However, this is not what I would like to do. I would like to take
  away one or more attributes using a decorator. My goal is to
  centralize the logic associated with that parameter. In my particular
  example, I am writing a cherrypy application (more specifically,
  turbogears1) and I would like all controller method to accept a lang
  and a skin attribute. As a simple, but hands-on example, imagine
  code like this (http://python.pastebin.com/f25f2429b):

  class MyController(Controller):

      @expose()
      def index(self, skin=default, lang=en):
          set_skin( skin )
          set_language( lang )
          return Hello skinned world

      @expose()
      def foo(self, skin=default, lang=en):
          set_skin( skin )
          set_language( lang )
          return bar

  This becomes cumbersome however for a large application with many
  controllers and methods. Always adding the parameters to the methods
  and function calls into the method bodies looks way to repetitive for
  my taste.

  Currently I solve this by using a cherrypy filter which removes those
  parameters from the HTTP-Request and puts them into the session
  object. This looked wrong to me though right from the start. Also, the
  cherrypy docs advise against the use of filters for application
  logic. But this looks pretty much like that to me somewhat. Worse
  yet, filters are not supported in Turbogears2 any longer. Which makes
  porting my app not straight-forward, as I have to port my filters and
  write WSGI middleware instead.

  I would prefer a syntax like this (http://python.pastebin.com/
  f462bc29c)

  class MyController(Controller):

      @expose()
      @skinnable
      @translatable
      def index(self):
          return Hello skinned world

      @expose()
      @skinnable
      @translatable
      def foo(self):
          return bar

  This, to me has several advantages: The logic is encapsulated in the
  application code itself. I do not need to rely on framework specific
  features (be it cherrypy or wsgi). This would make the application
  more self-contained and hence more future proof.

  But right here lies my problem. If you are not familiar with CherryPy,
  let me digress for a tiny bit to give you the necessary backgroud:

  Inside a controller, an exposed method is reachable via HTTP
  requests. It's possible to force request parameters. If that is the
  case (as in my case it is), then the call will only be successful, if
  all parameters received a vaule and no unknown parameters have been
  passed. Assume the following signature:

  def index(self)

  This will successfully return an HTTP Response when the client
  requested the resource on the URL /index. If the client adds a query
  string, say /index?lang=en the call will *fail* as this parameter is
  unkown. For this to work the signature must read:

  def index(self, lang)

  or

  def index(self, lang=en)

  Right end of digression, back to topic:

 Haven't finished coffee yet, plus it's a bank holiday! [/excuse]

 Can't that be changed to def index(self, **kwdargs) -- then your
 decorators can operate something like set_language(kwdargs.get('lang',
 'en')) and then delete the key

Sure. It could work like that. But that, in my case, would mean that
anyone could pass any parameters into that method. That should not
make any problems, but I prefer enforcing only the parameters I expect
(and /only/ those) to be passed into the method. If only for my own
sanity ;)

 Anyway, back to more coffee!

enjoy!

  My ultimate question is: Is it possible to write a decorator that
  removes a parameter from a call and return a function without that
  parameter? Something along the lines (http://python.pastebin.com/
  f257877cd):

  def translatable(f):
      Note that this code is only an example and will not work!
      lang = f.__get_parameter_value__(lang)
      f.__remove_parameter__(lang)
      do_something_with_lang(lang)
      return f

  I am aware, that this depends on *when* the decorator call is
  executed. If its called whenever the decorated function is called,
  then there should be some way to handle this. If it's called at
  compile-time, then I suppose it's not going to be possible.

  A usage scenario would be this:

  # --- definition ---
  @translatable
  def index(self):
     return _(Hello, lang)

  # --- call --
  obj.index( lang=de)
  obj.index

Re: Using a decorator to *remove* parameters from a call

[Michel Albert]

On Apr 13, 12:45 pm, Aaron Brady castiro...@gmail.com wrote:
 On Apr 13, 5:11 am, Michel Albert exh...@gmail.com wrote:

  A small foreword: This might look like a cherrypy-oriented post, and
  should therefore go to the cherrypy group, but if you read to the end,
  you'll see it's a more basic python problem, with cherrypy only as an
  example. ;)

  From the decorator PEP (318) I get it that you can /add/ parameters to
  a call. Say you do something like this:

  @mydeco( foo=bar )
  def myfunc( hello ):
     print foo, foo
 snip
  I would prefer a syntax like this (http://python.pastebin.com/
  f462bc29c)

  class MyController(Controller):

      @expose()
      @skinnable
      @translatable
      def index(self):
          return Hello skinned world

      @expose()
      @skinnable
      @translatable
      def foo(self):
          return bar

 snip
  more future proof.

 snip

 Sure.  Just pop the keywords you want from the dictionary, if they
 exist.

 def take_ab( fun ):
     def newfun( *ar, **kw ):
         aval= kw.pop( 'a', None )
         bval= kw.pop( 'b', None )
         print( 'aval %r and bval %r popped'% ( aval, bval ) )
         res= fun( *ar, **kw )
         return res
     return newfun

 @take_ab
 def f( c= 42, d= 'some' ):
     print( 'in f, c= %r, d= %r'% ( c, d ) )

 f( a= 'paramA', b= 'paramB', c= 'other number' )

 /Output:

 aval 'paramA' and bval 'paramB' popped
 in f, c= 'other number', d= 'some'

 As you can see, 'a', 'b', and 'c' were passed to 'f', but 'f' only
 received 'c', as well as its default value for 'd'.  So long as you
 only need to pass by keyword, not position, it's pretty easy.  It's
 harder if the parameter you want might have been passed by either, and
 you need to remove it from whichever one it was passed by.

Thanks. That works like a charm. I tried that last night and it gave
me an error. So I must have had a typo in there somewhere. Not enough
sleep I guess ;)
--
http://mail.python.org/mailman/listinfo/python-list

qrcode in python?

[Michel Albert]

I am planning to write a python-module for python as I haven't found
anything on the tubes so far. If anyone has any interesting insight on
this topic, replies are welcome! ;)

Q: What is QR-Code?
A: http://www.denso-wave.com/qrcode/qrstandard-e.html

So far (as of May 21st 2008):

Google  (1 sensible hit): 
http://mail.python.org/pipermail/python-list/2006-May/385258.html
The Cheese shop (nothing): 
http://pypi.python.org/pypi?%3Aaction=searchterm=qrcodesubmit=search
Sourceforge (1 hit - no files/code): http://sourceforge.net/projects/qrcode/

I was planning to make use of qr-code in a web-application which is
written in python. So far I was not yet lucky to find any existing
module. I just contacted the project lead of 
http://sourceforge.net/projects/qrcode/
on this topic as well. Let's see what he replies.

So. any pointers/ideas ?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Binary search tree

[Michel Albert]

On Nov 9, 11:45 pm, Bruno Desthuilliers
[EMAIL PROTECTED] wrote:
 [EMAIL PROTECTED] a écrit :

  Hi,

  I have to get list of URLs one by one and to find the URLs that I have
  more than one time(can't be more than twice).

  I thought to put them into binary search tree, this way they'll be
  sorted and I'll be able to check if the URL already exist.

 What about a set ?

 s = set()
 for url in urls:
if url in s:
  print already have , url
else:
  set.add(url)

Interesting. For this case I usually used dicts. As in:

d = {}
for url in urls:
   if url in d:
  print already have , url
   else:
  d[url] = 1

Now, I can see that this method has some superfluous data (the `1`
that is assigned to the dict). So I suppose this is less memory
efficient. But is this slower then? As both implementations use hashes
of the URL to access the data. Just asking out of curiosity ;)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Using python as primary language

[Michel Albert]

On Nov 8, 8:55 pm, [EMAIL PROTECTED] wrote:
 On Nov 8, 1:52 am, Michel Albert [EMAIL PROTECTED] wrote:



  In our company we are looking for one language to be used as default
  language. So far Python looks like a good choice (slacking behind
  Java). A few requirements that the language should be able cope with
  are:

  * Database access to Sybase.
This seems to be available for python, but the windows-binaries for
  the library
are not available. Self-Compiling them proved to be non-trivial (As
  always
on windows).
  * Easy GUI creation.
Solved using PyQt.
  * Cross Platform (Linux + Windows).
Again, PyQt, solves this
  * Easy deployment.
Solved using py2exe + innosetup
  * Charting (Histograms, Line charts, bar charts, pie charts, ...)
I am currently looking into PyQwt, which looks promising.
  * Report generation with GUI support
reportlab + rml?

  So far, nearly all point seems to be manageable. But I was not yet
  able to find a solution for the report generation. What we would like
  to have is a sort of GUI interface to prepare the reports without
  having to code the positioning. I found reportlab, and it looks like
  it can do all that is needed in terms of output. But you still have to
  code the report. And this is a no go. In context, I found RML and saw
  links to graphical RML editors. But I have not yet found a concrete
  example code, or documentation. What am I missing? Is RML a reportlab
  creation or is it a recognised open standard? If not, is there an open
  standard, that is easily to process with python?

  Any pointers? I would prefer coding Python to coding Java or
  worse. VB ;) which is another contender in our roundup.

 It looks like RML (Reportlab Markup Language) is a type of XML to me.
 It also appears to be a ReportLab invention. Lots of code examples can
 be found here:

 http://developer.reportlab.com/examples.html

 See also:http://www.reportlab.com/rml_index.html

 I'm not sure what you mean by editing a report. I have an
 application that allows users to enter data into a GUI and then it
 takes their data and inputs it into a ReportLab generated PDF.

 Mike

What I meant was that one should be able to draw a report template.
Basically a graphical user interface for RML in this case. I
personally would opt for writing RML or whatever code myself. But it's
impossible to convice my boss. The dialog usually goes like this:

So, did you find a click-and-play editor for reports - Not yet, but
anyway, writing code is more flexible and easier to manage later on -
Hmmm... but it's code! - Sure, but you only write it once for one
report, you can easily re-use code-snippets, modifying the code does
not require one additional program, you just use your text-editor of
choice,... - Okay, but it's CODE!

and this goes on forever. My boss seems to be allergic to writing code
by hand. Which is very frustrating. I'm happy that Qt has the
designer, although it's very easy to code the GUI's too ;)

-- 
http://mail.python.org/mailman/listinfo/python-list

parallel csv-file processing

[Michel Albert]

Currently I am faced with a large computation tasks, which works on a
huge CSV file. As a test I am working on a very small subset which
already contains 2E6 records. The task itself allows the file to be
split however as each computation only involves one line. The
application performing the computation exists already, but it was
never meant to run on such a big dataset.

One thing that is clear, is that it will take a while to compute all
this. So a distributed approach is probably a good idea. There ar a
couple of options for this:

Scenario A ( file is split manually in smaller parts ):
1) Fire up an openmosix/kerrighed cluster, and run one process for
each file part.

Scenario B ( file is split using the application itself ):
2) Again with an openmosix/kerrighed cluster, but only one instance of
the application is run, using parallelpython
3) Using parallelpython without cluster, but using ppserver.py on each
node.

The second case looks most interesting as it is quite flexible. In
this case I would need to address subsets of the CSV file however. And
the default csv.reader class does not allow random-access of the file
(or jumping to a specific line).

What would be the most efficient way to subset a CSV-file. For
example:

f1 = job_server.submit(calc_scores, datafile[0:1000])
f2 = job_server.submit(calc_scores, datafile[1001:2000])
f3 = job_server.submit(calc_scores, datafile[2001:3000])
...

and so on

Obviously this won't work as you cannot access a slice of a csv-file.
Would it be possible to subclass the csv.reader class in a way that
you can somewhat efficiently access a slice? Jumping backwards is not
really necessary, so it's not really random access.

The obvious way is to do the following:

buffer = []
for line in reader:
   buffer.append(line)
   if len(buffer) == 1000:
  f = job_server.submit(calc_scores, buffer)
  buffer = []

f = job_server.submit(calc_scores, buffer)
buffer = []

but would this not kill my memory if I start loading bigger slices
into the buffer variable?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Drawing charts in Qt

[Michel Albert]

On Nov 6, 5:08 pm, David Boddie [EMAIL PROTECTED] wrote:
 On Tue Nov 6 15:46:07 CET 2007, Michel Albert wrote:

 [PyQwt and matplotlib]

  PyQwt looks much more interesting, but I have trouble installing it.
  On my machine it complains that sipconfig has no attribute
  '_pkg_config'.

 Is the configuration script finding the sipconfig file for SIP 3 or SIP 4?

How can I see if it's finding it or not? It does not give me any
helpful output.


  In the end the application should also run on Windows boxes. And I
  suppose as long as I use the right versions and use the precompiled
  binaries, I should get it at least installed. But the thing with the
  version numbers looks like some major lottery game to me. If one of
  the elements in the chain (be it Qt, or Qwt) release new versions and
  the available binary distributions get out of sync, future support
  for the written application becomes foggy.

 I'm not sure what you mean. Can you explain?

Well, I suppose I can manage this. If there really are some version
problems I can always compile everything I need myself.


  Has anyone ever successfully used these graphing libraries with PyQt?
  Or are there other graphing libraries available? In fact, my needs are
  modest. A Line- and Bar-Chart would solve the majority of problems.

 I've installed PyQwt for PyQt4 and tried the examples, but only out of
 curiosity because I wasn't writing an application that needed those
 facilities at the time.

 David


-- 
http://mail.python.org/mailman/listinfo/python-list

Using python as primary language

[Michel Albert]

In our company we are looking for one language to be used as default
language. So far Python looks like a good choice (slacking behind
Java). A few requirements that the language should be able cope with
are:

* Database access to Sybase.
  This seems to be available for python, but the windows-binaries for
the library
  are not available. Self-Compiling them proved to be non-trivial (As
always
  on windows).
* Easy GUI creation.
  Solved using PyQt.
* Cross Platform (Linux + Windows).
  Again, PyQt, solves this
* Easy deployment.
  Solved using py2exe + innosetup
* Charting (Histograms, Line charts, bar charts, pie charts, ...)
  I am currently looking into PyQwt, which looks promising.
* Report generation with GUI support
  reportlab + rml?

So far, nearly all point seems to be manageable. But I was not yet
able to find a solution for the report generation. What we would like
to have is a sort of GUI interface to prepare the reports without
having to code the positioning. I found reportlab, and it looks like
it can do all that is needed in terms of output. But you still have to
code the report. And this is a no go. In context, I found RML and saw
links to graphical RML editors. But I have not yet found a concrete
example code, or documentation. What am I missing? Is RML a reportlab
creation or is it a recognised open standard? If not, is there an open
standard, that is easily to process with python?

Any pointers? I would prefer coding Python to coding Java or
worse. VB ;) which is another contender in our roundup.

-- 
http://mail.python.org/mailman/listinfo/python-list

Drawing charts in Qt

[Michel Albert]

I would like to display some charts in a Qt application. But all the
docs I find online are rather dusty and talk about Qt3. My application
uses Qt4 however. I ran into PyQwt and matplotlib. But the docs of
matplotlib are horrid and the example in their wiki covers Qt3, and
things look quite cryptic to me.

PyQwt looks much more interesting, but I have trouble installing it.
On my machine it complains that sipconfig has no attribute
'_pkg_config'.

In the end the application should also run on Windows boxes. And I
suppose as long as I use the right versions and use the precompiled
binaries, I should get it at least installed. But the thing with the
version numbers looks like some major lottery game to me. If one of
the elements in the chain (be it Qt, or Qwt) release new versions and
the available binary distributions get out of sync, future support
for the written application becomes foggy.

Has anyone ever successfully used these graphing libraries with PyQt?
Or are there other graphing libraries available? In fact, my needs are
modest. A Line- and Bar-Chart would solve the majority of problems.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python + Shoutpy + Twisted Locks

[Michel Albert]

On Oct 6, 4:21 am, Gabriel Genellina [EMAIL PROTECTED] wrote:
 En Fri, 05 Oct 2007 04:55:55 -0300, exhuma.twn [EMAIL PROTECTED] escribi?:

  [...] What I found
  is that libshout is blocking, which should be fine as the whole
  thing runs in it's separate thread. But the application hangs
  nevertheless while streaming. This effectively blocks out the other
  thread that checks the player status, which then fails to append new
  songs to the queue. So only one song is played when streaming.

  The other threads in my application run fine and don't block the rest
  of the app. So I guess, that the main problem is that blocking occurs
  outside the python world and inside the libshout world.

 Only one thread at a time may be executing Python code; the Global
 Interpreter Lock (GIL) ensures the mutual exclusion. Extension modules
 (written in C) may use the macros
 Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release/acquire the GIL
 before/after an external blocking call.
 I don't know libshout, or how you are doing the binding python-libshout,
 but if your analysis is correct it means that the code is not releasing
 the GIL at the appropiate points.

 --
 Gabriel Genellina

Hmmm... ok. I suppose rewriting the whole thing using twisted's
deferreds could then solve the problem. Which are basically nothing
more than callbacks with a weird name ;) Unfortunately this means that
I have to re-think a lot. But in the end I suppose it will pay off.

Thanks for taking the time and reading my little essay Gabriel ;)

-- 
http://mail.python.org/mailman/listinfo/python-list