Re: unit test strategy

[Aaron Brady]

On Sunday, September 16, 2012 3:01:11 PM UTC-5, Steven D'Aprano wrote:
> On Sun, 16 Sep 2012 11:38:15 -0700, Aaron Brady wrote:
> > Here is an example of some repetitive code.
> > 
> > for view_meth in [ dict.items, dict.keys, dict.values ]:
> > dict0= dict( ( k, None ) for k in range( 10 ) ) 
> >   iter0= iter( view_meth( dict0 ) )
> > dict.__setitem__( dict0, 0, 1 )
> > next( iter0 )
> > dict.__setitem__( dict0, 10, 1 )
> > self.assertRaises( IterationError, next, iter0 )
> [...]
> 
> First off, if you have any wish for this to be accepted into the standard 
> library, I suggest you stick to PEP 8.

The code in the zip file on the other thread does conform to PEP 8.
 
> Secondly, this is test code. A bit of repetition is not to be concerned 
> about, clarity is far more important than "Don't Repeat Yourself". The 
> aim isn't to write the fastest, or most compact code, but to have good 
> test coverage with tests which are *obviously* correct (rather than test 
> code which has no obvious bugs, which is very different). If a test 
> fails, you should be able to identify quickly what failed without running 
> a debugger to identify what part of the code failed.
> 
> Thirdly, why are you writing dict.__setitem__( dict0, 0, 1 ) instead of 
> dict0[0] = 1 ?
> 
> 
> [...]
> > Specifically my questions are, is the code condensed beyond legibility? 
> 
> Yes.
> 
> 
> > Should 'chain' execute the test directly, or act as a metaprogram and
> > output the test code into a 2nd file, or both?
> 
> None of the above.
> 
> 
> > Should 'chain' create
> > the iterators in a dictionary, or in the function local variables
> > directly with 'exec'?
> 
> Heavens to Murgatroyd, you can't be serious.
> 
> 
> Here's my attempt at this. Note the features:
> 
> - meaningful names (if a bit long, but that's a hazard of tests)
> - distinct methods for each distinct test case
> - comments explaining what the test code does
> - use of subclassing
> 
> 
> # Untested
> class TestDictIteratorModificationDetection(unittest.TestCase):
[snip]
> def testIteratorFailsAfterSet(self):
> self.iterator_fails_after_modification(dict.__setitem__, 1, 1)
> 
> def testIteratorFailsAfterDel(self):
> self.iterator_fails_after_modification(dict.__delitem__, 1)
> 
> def testIteratorFailsAfterUpdate(self):
> self.iterator_fails_after_modification(dict.update, {5: 1})
> 
> def testIteratorFailsAfterPop(self):
> self.iterator_fails_after_modification(dict.pop, 4)
[snip]
> 
> I think I've got all the methods which can mutate a dictionary. If I 
> missed any, it's easy enough to add a new test method to the class.
[snip]

Well Mr. D'Aprano, I have some serious disagreements with the script you posted.

You did test all the mutating methods; there are 7; but I don't think it's 
enough.  You omitted the test case which revealed the bug originally or other 
pairs of operations; you didn't test on empty sets; you didn't test on "null" 
modifications; you didn't test on multiple iterators in any form; and you 
didn't test for memory leaks which is important with a dynamic structure.  A 
thorough test suite should contain tens if not hundreds of tests for this 
application.

Your script was very easy to understand.  However in the volume I'm advocating, 
something more concise would easier to understand, and D-R-Y becomes applicable 
again.  In a more concise form, the reader could browse what tests are 
executed, find a certain test or determine if it's omitted.

The "best of both" solution is a metaprogram, which is extremely brief, but 
does not run tests, and outputs a full test catalog instead, which is then 
itself scrutinized and run as the real test.  The test script would 
consequently be two files big, and be run in a specific order.  Not all tests 
can be condensed in a metaprogram; the script would contain large sections of 
literal code or it would appear in yet a 3rd file.

> Thirdly, why are you writing dict.__setitem__( dict0, 0, 1 ) instead of 
> dict0[0] = 1 ?

'__setitem__' can be passed to secondary functions whereas square brackets 
cannot.  The 2nd file, the output of the metaprogram, could contain either or 
both.  We could pass 'operator.setitem' as an alternative.

I realize I'm introducing yet a 3rd foreign concept with the patch.  If not 
enough readers approve of it I will have to abandon it, which would be a shame.

OTOH, I appreciate the fact you used my "for view in (dict.items, dict.keys, 
dict.values):" idea.  Also, what is your argument that an unstarted iterator 
should be exempt from invalidation when the set/dict is modified?  It is not 
obvious it should or shouldn't, similar to the behavior of modifying dict 
values but not keys, and I would side against the exemption.  (Perhaps we 
should raise that issue on the other thread.)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: unit test strategy

[Aaron Brady]

On Sunday, September 16, 2012 2:42:09 AM UTC-5, Steven D'Aprano wrote:
> On Fri, 14 Sep 2012 19:59:29 -0700, Aaron Brady wrote:
> 
> 
> 
> > Hello,
> 
> > 
> 
> > I've developing a test script.  There's a lot of repetition.  I want to
> 
> > introduce a strategy for approaching it, but I don't want the program to
> 
> > be discredited because of the test script.
> 
> 
> 
> Test scripts should be simple enough that they don't require test scripts 
> 
> of their own. Or at least, not too many "test-the-test" tests. It is 
> 
> possible to avoid repetition without making convoluted, hard to 
> 
> understand code. Based on the tiny code fragments you give below, I 
> 
> suspect that your test script will need more testing than the code it 
> 
> tests!
> 
> 
> 
> 
> 
> > Therefore, I'd like to know
> 
> > what people's reactions to and thoughts about it are.
> 
> 
> 
> I'd love to make some suggestions, but I have *no idea* what you are 
> 
> talking about. See further comments below:
> 
> 
> 
> 
> 
> > The first strategy I used created an iterator and advanced it between
> 
> > each step:
> 
> 
> 
> What are you using an iterator for? What does this have to do with unit 
> 
> testing?
> 
> 
> 
> So far, your explanation is rather lacking. It's a bit like:
> 
> 
> 
> "I want to create an alarm system for my home, so I put in a screw and 
> 
> tightened it after each step."
> 
> 
> 
> Doesn't really help us understand what you are doing.
> 
> 
> 
> 
> 
> > self.op_chain(range(5), ('add', 5))
> 
> > self.op_chain(range(5), ('add', -2), ('add', -1))
> 
> > self.op_chain(range(5), ('discard', -1), ('add', 5))
> 
> > self.op_chain_ok(range(5), ('update', [0, 1]))
> 
> > Etc.
> 
> 
> 
> Where is the iterator you created? Where are you advancing it? What's 
> 
> op_chain do?
> 
> 
> 
> 
> 
> > I'm considering something more complicated.  'iN' creates iterator N,
> 
> > 'nN' advances iterator N, an exception calls 'assertRaises', and the
> 
> > rest are function calls.
> 
> [...]
> 
> 
> 
> You've proven that even in Python people can write obfuscated code.
> 
> 
> 
> 
> 
> > Do you think the 2nd version is legible?
> 
> 
> 
> Neither version is even close to legible.
> 
> 
> 
> 
> 
> > Could it interfere with the accuracy of the test?
> 
> 
> 
> Who knows? I have no clue what your code is doing, it could be doing 
> 
> *anything*.
> 
> 
> 
> 
> 
> 
> 
> -- 
> 
> Steven

You are forcing me to explain my test code.

Here is an example of some repetitive code.

for view_meth in [ dict.items, dict.keys, dict.values ]:
dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
dict.__setitem__( dict0, 0, 1 )
next( iter0 )
dict.__setitem__( dict0, 10, 1 )
self.assertRaises( IterationError, next, iter0 )

dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
next( iter0 )
dict.__setitem__( dict0, 0, 1 )
next( iter0 )
dict.__setitem__( dict0, 10, 1 )
self.assertRaises( IterationError, next, iter0 )

dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
self.assertRaises( KeyError, dict0.__delitem__, 10 )
next( iter0 )
dict.__delitem__( dict0, 9 )
self.assertRaises( IterationError, next, iter0 )

dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
next( iter0 )
self.assertRaises( KeyError, dict0.__delitem__, 10 )
next( iter0 )
dict.__delitem__( dict0, 9 )
self.assertRaises( IterationError, next, iter0 )


It makes sense to condense it.  However, it can be condensed rather far, as 
follows:

dsi= dict.__setitem__
ddi= dict.__delitem__
KE= KeyError
IE= IterationError
chain(range(10), 'i0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 'n0'))
chain(range(10), 'i0', 'n0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 'n0'))
chain(range(10), 'i0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))
chain(range(10), 'i0', 'n0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))


The parameters to 'chain' correspond 1-to-1 with the statements earlier.  The 
view methods are looped over in the 'chain' method instead.  'op_chain' was an 
earlier version; some midway point in condensing the code could be preferable.

Specifically my questions are, is the code condensed beyond legibility?  Should 
'chain' execute the test directly, or act as a metaprogram and output the test 
code into a 2nd file, or both?  Should 'chain' create the iterators in a 
dictionary, or in the function local variables directly with 'exec'?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: unit test strategy

[Aaron Brady]

On Friday, September 14, 2012 10:32:47 PM UTC-5, David Hutto wrote:
> On Fri, Sep 14, 2012 at 11:26 PM, Dwight Hutto  wrote:
> 
> > On Fri, Sep 14, 2012 at 10:59 PM, Aaron Brady  wrote:
> 
> >> Hello,
> 
> >>
> 
> >> I've developing a test script.  There's a lot of repetition.  I want to 
> >> introduce a strategy for approaching it, but I don't want the program to 
> >> be discredited because of the test script.  Therefore, I'd like to know 
> >> what people's reactions to and thoughts about it are.
> 
> >>
> 
> >> The first strategy I used created an iterator and advanced it between each 
> >> step:
> 
> >
> 
> > That isn't a refined iterator below:
> 
> What I mean is look at the similarities, and the differences, then
> 
> replace the differences with interpolation, in eval even.
> 
> 
> 
> 
> 
> >
> 
> >> self.op_chain(range(5), ('add', 5))
> 
> >> self.op_chain(range(5), ('add', -2), ('add', -1))
> 
> >> self.op_chain(range(5), ('discard', -1), ('add', 5))
> 
> >> self.op_chain_ok(range(5), ('update', [0, 1]))
> 
> >> Etc.
> 
> >>
> 
> >> I'm considering something more complicated.  'iN' creates iterator N, 'nN' 
> >> advances iterator N, an exception calls 'assertRaises', and the rest are 
> >> function calls.
> 
> 
> 
> iN = [N for N in range(0,5)]
> 
> 
> 
> 
> 
> >> dsi= dict.__setitem__
> 
> >> ddi= dict.__delitem__
> 
> >> dsd= dict.setdefault
> 
> >> KE= KeyError
> 
> >> IE= IterationError
> 
> >> self.chain(range(10), 'i0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 
> >> 'n0'))
> 
> >> self.chain(range(10), 'i0', 'n0', (dsd, 0, 0), 'n0', (dsd, 10, 1), 
> >> (IE, 'n0'))
> 
> >> self.chain(range(10), 'i0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 
> >> 'n0'))
> 
> >>
> 
> >> Do you think the 2nd version is legible?  Could it interfere with the 
> >> accuracy of the test?
> 
> 
> 
> Define the 2nd version
> 
> 
> 
> 
> 
> >
> 
> > Show the test, which should show instances of what you want called.
> 
> >
> 
> > I could rewrite the above, but it seems you're more in need of refining
> 
> > your iterations, and the values given within them.
> 
> >
> 
> 
> 
> -- 
> 
> Best Regards,
> 
> David Hutto
> 
> CEO: http://www.hitwebdevelopment.com

Hi David,

I'm interested in your comments, but I had difficulty interpreting them.  What 
I want to know is, do people think that the 2nd version I presented would be a 
more effective test script?

Do you think it would be more useful to run the tests in the function call 
directly?  Or would it be more useful to output a program script and then run 
that?  Is there some risk that the direct test would interfere with the 
results?  And, is the 2nd version legible?  That is, is it easy for other 
programmers to tell what the purpose and actual effects of a given test are?
-- 
http://mail.python.org/mailman/listinfo/python-list

unit test strategy

[Aaron Brady]

Hello,

I've developing a test script.  There's a lot of repetition.  I want to 
introduce a strategy for approaching it, but I don't want the program to be 
discredited because of the test script.  Therefore, I'd like to know what 
people's reactions to and thoughts about it are.

The first strategy I used created an iterator and advanced it between each step:
self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

I'm considering something more complicated.  'iN' creates iterator N, 'nN' 
advances iterator N, an exception calls 'assertRaises', and the rest are 
function calls.
dsi= dict.__setitem__
ddi= dict.__delitem__
dsd= dict.setdefault
KE= KeyError
IE= IterationError
self.chain(range(10), 'i0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 'n0'))
self.chain(range(10), 'i0', 'n0', (dsd, 0, 0), 'n0', (dsd, 10, 1), (IE, 
'n0'))
self.chain(range(10), 'i0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))

Do you think the 2nd version is legible?  Could it interfere with the accuracy 
of the test?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Thursday, August 23, 2012 1:11:14 PM UTC-5, Steven D'Aprano wrote:
> On Thu, 23 Aug 2012 09:49:41 -0700, Aaron Brady wrote:
> 
> 
> 
> [...]
> 
> > The patch for the above is only 40-60 lines.  However it introduces two
> 
> > new concepts.
> 
> > 
> 
> > The first is a "linked list", a classic dynamic data structure, first
> 
> > developed in 1955, cf. http://en.wikipedia.org/wiki/Linked_list . 
> 
> > Linked lists are absent in Python
> 
> 
> 
> They certainly are not. There's merely no named "linked list" class.
> 
> 
> 
> Linked lists are used by collections.ChainMap, tracebacks, xml.dom, 
> 
> Abstract Syntax Trees, and probably many other places. (Well, technically 
> 
> some of these are trees rather than lists.) You can trivially create a 
> 
> linked list:
> 
> 
> 
> x = [a, [b, [c, [d, [e, None]
> 
> 
> 
> is equivalent to a singly-linked list with five nodes. Only less 
> 
> efficient.
>
[snip.]
>
> -- 
> 
> Steven

That's not totally true.  Your formulation is equivalent to a single-linked 
list, but a double-linked list can't be represented as an expression because it 
contains reference cycles.

Single and double-linked lists have the same requirements for inserting a new 
node.  For instance:

[a, [b, [c, [d, [e, None]
[a, [a2, [b, [c, [d, [e, None]]

However, to remove a node, the single-linked list requires iterating over the 
entire list to find the predecessor of the target, whereas the double-linked 
list already contains that information.

[a, [b, [c, [d, [e, None]
[a, [b, [c, [e, None

The difference might be moot in some applications, such as if we only remove 
nodes when we're already iterating.

Memory constraints were more severe 50 years ago when the structure was 
developed.  The difference between one pointer and two in a structure could 
have a big impact in repetition.  The single-linked list admits more easily of 
recursive algorithms as well.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Monday, September 3, 2012 8:59:16 PM UTC-5, Steven D'Aprano wrote:
> On Mon, 03 Sep 2012 21:50:57 -0400, Dave Angel wrote:
> 
> 
> 
> > On 09/03/2012 09:26 PM, Steven D'Aprano wrote:
> 
> 
> 
> >> An unsigned C int can count up to 4,294,967,295. I propose that you say
> 
> >> that is enough iterators for anyone, and use a single, simple, version
> 
> >> counter in the dict and the iterator. If somebody exceeds that many
> 
> >> iterators to a single dict or set,
> 
> > 
> 
> > I think you have the count confused.  it has to be a count of how many
> 
> > changes have been made to the dict or set, not how many iterators exist.
> 
> 
> 
> Oops, yes you are absolutely right. It's a version number, not a count of 
> 
> iterators.
> 
> 
> 
> 
> 
> -- 
> 
> Steven

Hello.  We have a number of proposed solutions so far.

1) Collection of iterators
  a) Linked list
i) Uncounted references
ii) Counted references
iii) Weak references
  b) Weak set
2) Serial index / timestamp
  a) No overflow - Python longs
  b) Overflow - C ints / shorts / chars
  c) Reset index if no iterators left
3) Iterator count
  - Raise exception on set modifications, not iteration

Note, "2b" still leaves the possibility of missing a case and letting an error 
pass silently, as the current behavior does.  The rest catch the error 100% of 
the time.

Anyway, I plan to develop the above patch for the 'dict' class.  Would anyone 
like to take over or help me do it?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Monday, September 3, 2012 3:28:28 PM UTC-5, Dave Angel wrote:
> On 09/03/2012 04:04 PM, Aaron Brady wrote:
> 
> > On Monday, September 3, 2012 2:30:24 PM UTC-5, Ian wrote:
> 
> >> On Sun, Sep 2, 2012 at 11:43 AM, Aaron Brady  wrote:
> 
> >>
> 
> >>> We could use a Python long object for the version index to prevent 
> >>> overflow.  Combined with P. Rubin's idea to count the number of open 
> >>> iterators, most use cases still wouldn't exceed a single word comparison; 
> >>> we could reset the counter when there weren't any.
> 
> >>
> 
> >>
> 
> >> We could use a Python long; I just don't think the extra overhead is
> 
> >>
> 
> >> justified in a data structure that is already highly optimized for
> 
> >>
> 
> >> speed.  Incrementing and testing a C int is *much* faster than doing
> 
> >>
> 
> >> the same with a Python long.
> 
> > I think the technique would require two python longs and a bool in the set, 
> > and a python long in the iterator.
> 
> >
> 
> > One long counts the number of existing (open) iterators.  Another counts 
> > the version.  The bool keeps track of whether an iterator has been created 
> > since the last modification, in which case the next modification requires 
> > incrementing the version and resetting the flag.
> 
> 
> 
> I think you're over-engineering the problem.  it's a bug if an iterator
> 
> is used after some change is made to the set it's iterating over.  We
> 
> don't need to catch every possible instance of the bug, that's what
> 
> testing is for.  The point is to "probably" detect it, and for that, all
> 
> we need is a counter in the set and a counter in the open iterator. 
> 
> Whenever changing the set, increment its count.  And whenever iterating,
> 
> check the two counters.  if they don't agree, throw an exception, and
> 
> destroy the iterator.  i suppose that could be done with a flag, but it
> 
> could just as easily be done by zeroing the pointer to the set.
> 
> 
> 
> I'd figure a byte or two would be good enough for the counts, but a C
> 
> uint would be somewhat faster, and wouldn't wrap as quickly.
> 
> 
> 
> -- 
> 
> 
> 
> DaveA

Hi D. Angel,

The serial index constantly reminds me of upper limits.  I have the same 
problem with PHP arrays, though it's not a problem with the language itself.

The linked list doesn't have a counter, it invalidates iterators when a 
modification is made, therefore it's the "correct" structure in my 
interpretation.  But it does seem more precarious comparatively, IMHO.

Both strategies solve the problem I posed originally, they both involve 
trade-offs, and it's too late to include either in 3.3.0.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Monday, September 3, 2012 2:30:24 PM UTC-5, Ian wrote:
> On Sun, Sep 2, 2012 at 11:43 AM, Aaron Brady  wrote:
> 
> > We could use a Python long object for the version index to prevent 
> > overflow.  Combined with P. Rubin's idea to count the number of open 
> > iterators, most use cases still wouldn't exceed a single word comparison; 
> > we could reset the counter when there weren't any.
> 
> 
> 
> We could use a Python long; I just don't think the extra overhead is
> 
> justified in a data structure that is already highly optimized for
> 
> speed.  Incrementing and testing a C int is *much* faster than doing
> 
> the same with a Python long.

I think the technique would require two python longs and a bool in the set, and 
a python long in the iterator.

One long counts the number of existing (open) iterators.  Another counts the 
version.  The bool keeps track of whether an iterator has been created since 
the last modification, in which case the next modification requires 
incrementing the version and resetting the flag.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Monday, August 27, 2012 2:17:45 PM UTC-5, Ian wrote:
> On Thu, Aug 23, 2012 at 10:49 AM, Aaron Brady  wrote:
> 
> > The patch for the above is only 40-60 lines.  However it introduces two new 
> > concepts.
> 
> 
> 
> Is there a link to the patch?

Please see below.  It grew somewhat during development.

> > The first is a "linked list".
SNIP.
 
> > The second is "uncounted references".  The uncounted references are 
> > references to "set iterators" exclusively, exist only internally to "set" 
> > objects, and are invisible to the rest of the program.  The reason for the 
> > exception is that iterators are unique in the Python Data Model; iterators 
> > consist of a single immutable reference, unlike both immutable types such 
> > as strings and numbers, as well as container types.  Counted references 
> > could be used instead, but would be consistently wasted work for the 
> > garbage collector, though the benefit to programmers' peace of mind could 
> > be significant.
> 
> >
> 
> > Please share your opinion!  Do you agree that the internal list resolves 
> > the inconsistency?  Do you agree with the strategy?  Do you agree that 
> > uncounted references are justified to introduce, or are counted references 
> > preferable?
> 
> 
> 
> This feature is a hard sell as it is; I think that adding uncounted
> 
> references into the mix is only going to make that worse.  May I
> 
> suggest an alternate approach?  Internally tag each set or dict with a
> 
> "version", which is just a C int.  Every time the hash table is
> 
> modified, increment the version.  When an iterator is created, store
> 
> the current version on the iterator.  When the iterator is advanced,
> 
> check that the iterator version matches the dict/set version.  If
> 
> they're not equal, raise an error.
> 
> 
> 
> This should add less overhead than the linked list without any
> 
> concerns about reference counting.  It does introduce a small bug in
> 
> that an error condition could be "missed", if the version is
> 
> incremented a multiple of 2**32 or 2**64 times between iterations --
> 
> but how often is that really likely to occur?  Bearing in mind that
> 
> this error is meant for debugging and not production error handling,
> 
> you could even make the version a single byte and I'd still be fine
> 
> with that.
> 
> 
> 
> Cheers,
> 
> Ian


Hi Ian,

We could use a Python long object for the version index to prevent overflow.  
Combined with P. Rubin's idea to count the number of open iterators, most use 
cases still wouldn't exceed a single word comparison; we could reset the 
counter when there weren't any.  Using the linked list collection, modification 
operations are expensive in rare cases.  Using the version index, iteration is 
expensive in rare cases.

I was more interested in the linked list for conceptual reasons, so I developed 
it further.  Changelog, diff file, test suite, and links are below.  The devs 
should be aware that a competing patch might be developed.  I would be pleased 
to hear what everybody thinks of it!

Linked list with uncounted references implementation for Win32.

Added:
- 'set_clear_ex' and 'set_clear_internal_ex' methods, differ in invalidation 
and conditional invalidation behavior and return type..  The 'set.clear()' 
method and 'tp_clear' type field both called the same method.
- 'set_invalidate_iter_linked' method.  Iterate over the iterators of a set, 
mark them invalid, and clear the list.
- 'setiter_unlink_internal' method.  Remove the iterator from the set's linked 
list of iterators.
- 'IterationError', global.
- New fields:
-- PySetObject: setiterobject *iter_linked.  Pointer to the first element of 
the linked list of the iterators of the set.
-- setiterobject: setiterobject *linked_pred, *linked_succ.  Predecessor and 
successor nodes in the linked list of iterators of the same set.
-- setiterobject: char ob_valid.  Validation status of the iterator.
- Result is compared with original in 'set_intersection_update' and '_multi' to 
determine whether to invalidate the list of iterators.  Asymptotic running time 
is unchanged.
- Pending: add 'tp_clear' field to 'PySetIter_Type'?
- Test script included, 'large numbers' test pending.

6 files changed: { setobject.h, setobject.c, exceptions.c, pyerrors.h, 
python3.def, python33stub.def }.  Test script 'set_iterator_test.py' new.  
Linked list interface and pseudocode 'patch_pseudocode.txt'.
Zip file:
http://home.comcast.net/~castironpi-misc/clpy-0062-set_iterator_patch.zip
Diff file of 3.3.0b2:
http://home.comcast.net/~castironpi-misc/clpy-0062-set_iterator_diff.txt

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Looking for an IPC solution

[Aaron Brady]

On Friday, August 31, 2012 2:22:00 PM UTC-5, Laszlo Nagy wrote:
> There are just so many IPC modules out there. I'm looking for a solution 
> 
> for developing a new a multi-tier application. The core application will 
> 
> be running on a single computer, so the IPC should be using shared 
> 
> memory (or mmap) and have very short response times. But there will be a 
> 
> tier that will hold application state for clients, and there will be 
> 
> lots of clients. So that tier needs to go to different computers. E.g. 
> 
> the same IPC should also be accessed over TCP/IP. Most messages will be 
> 
> simple data structures, nothing complicated. The ability to run on PyPy 
> 
> would, and also to run on both Windows and Linux would be a plus.
> 
> 
> 
> I have seen a stand alone cross platform IPC server before that could 
> 
> serve "channels", and send/receive messages using these channels. But I 
> 
> don't remember its name and now I cannot find it. Can somebody please help?
> 
> 
> 
> Thanks,
> 
> 
> 
> Laszlo

Hi Laszlo,

There aren't a lot of ways to create a Python object in an "mmap" buffer.  
"mmap" is conducive to arrays of arrays.  For variable-length structures like 
strings and lists, you need "dynamic allocation".  The C functions "malloc" and 
"free" allocate memory space, and file creation and deletion routines operate 
on disk space.  However "malloc" doesn't allow you to allocate memory space 
within memory that's already allocated.  Operating systems don't provide that 
capability, and doing it yourself amounts to creating your own file system.  If 
you did, you still might not be able to use existing libraries like the STL or 
Python, because one address might refer to different locations in different 
processes.

One solution is to keep a linked list of free blocks within your "mmap" buffer. 
 It is prone to slow access times and segment fragmentation.  Another solution 
is to create many small files with fixed-length names.  The minimum file size 
on your system might become prohibitive depending on your constraints, since a 
4-byte integer could occupy 4096 bytes on disk or more.  Or you can serialize 
the arguments and return values of your functions, and make requests to a 
central process.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Thursday, August 23, 2012 1:11:14 PM UTC-5, Steven D'Aprano wrote:
> On Thu, 23 Aug 2012 09:49:41 -0700, Aaron Brady wrote:
> 
> 
> 
> [...]
> 
> > The patch for the above is only 40-60 lines.  However it introduces two
> 
> > new concepts.
> 
> > 
> 
> > The first is a "linked list", a classic dynamic data structure, first
> 
> > developed in 1955, cf. http://en.wikipedia.org/wiki/Linked_list . 
> 
> > Linked lists are absent in Python
> 
> 
> 
> They certainly are not. There's merely no named "linked list" class.
> 
> 
> 
> Linked lists are used by collections.ChainMap, tracebacks, xml.dom, 
> 
> Abstract Syntax Trees, and probably many other places. (Well, technically 
> 
> some of these are trees rather than lists.) You can trivially create a 
> 
> linked list:
> 
> 
> 
> x = [a, [b, [c, [d, [e, None]
> 
> 
> 
> is equivalent to a singly-linked list with five nodes. Only less 
> 
> efficient.
> 
> 
> 
> 
> 
> > The second is "uncounted references".  The uncounted references are
> 
> > references to "set iterators" exclusively, exist only internally to
> 
> > "set" objects, and are invisible to the rest of the program.  The reason
> 
> > for the exception is that iterators are unique in the Python Data Model;
> 
> > iterators consist of a single immutable reference, unlike both immutable
> 
> > types such as strings and numbers, as well as container types.  Counted
> 
> > references could be used instead, but would be consistently wasted work
> 
> > for the garbage collector, though the benefit to programmers' peace of
> 
> > mind could be significant.
> 
> 
> 
> The usual way to implement "uncounted references" is by using weakrefs. 
> 
> Why invent yet another form of weakref?
> 
> 
> 
> 
> 
> 
> 
> -- 
> 
> Steven


Hello S. D'Aprano.  Thanks for your support as always.

The semantics of the second collection are equivalent to a WeakSet.  The space 
and time consumption of a WeakSet are higher in comparison to a linked list.  
However, so long as we iterated over it using the C API instead of creating an 
iterator, it would be consistent.  If we dynamically create the WeakSet on 
demand and free it when empty, the space consumption would be lower.  Typical 
use cases don't involve creating thousands of iterators, or rapidly creating 
and destroying them, so the performance impact might not be severe.

Regarding the bare weakrefs, if the iterator's destructor hasn't been called, 
then the pointer is still valid.  If it has been called, then it's not present 
in the list.  Unlike Python classes, the destructors of C extension classes are 
guaranteed to be called.  Therefore there are no points during exection at 
which a node needs to check whether a reference to its neighbor is valid.

Are your concerns based on data integrity, future maintainability, or what?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Saturday, August 18, 2012 9:28:32 PM UTC-5, Aaron Brady wrote:
> On Saturday, August 18, 2012 5:14:05 PM UTC-5, MRAB wrote:
> 
> > On 18/08/2012 21:29, Aaron Brady wrote:
> 
> > > On Friday, August 17, 2012 4:57:41 PM UTC-5, Chris Angelico wrote:
> 
> > >> On Sat, Aug 18, 2012 at 4:37 AM, Aaron Brady  
> > >> wrote:
> 
> > >> > Is there a problem with hacking on the Beta?
> 
> > >> Nope. Hack on the beta, then when the release arrives, rebase your
> 
> > >> work onto it. I doubt that anything of this nature will be changed
> 
> > >> between now and then.
> 
> > >> ChrisA
> 
> > > Thanks Chris, your post was encouraging.
> 
> > > I have a question about involving the 'tp_clear' field of the types.
> 
> > > http://docs.python.org/dev/c-api/typeobj.html#PyTypeObject.tp_clear
> 
> > > '''
> 
> > > ...The tuple type does not implement a tp_clear function, because it’s 
> > > possible to prove that no reference cycle can be composed entirely of 
> > > tuples.
> 
> > > '''
> 
> > > I didn't follow the reasoning in the proof; the premise is necessary but 
> > > IMHO not obviously sufficient.  Nevertheless, the earlier diagram 
> > > contains an overt homogeneous reference cycle.
> 
> > > Reposting: 
> > > http://home.comcast.net/~castironpi-misc/clpy-0062%20set%20iterators.png
> 
> > > In my estimate, the 'tp_traverse' and 'tp_clear' fields of the set 
> > > doesn't need to visit the auxiliary collection; the same fields of the 
> > > iterators don't need to visit the primary set or other iterators; and 
> > > references in the linked list don't need to be included in the iterators' 
> > > reference counts.
> 
> > > Can someone who is more familiar with the cycle detector and cycle 
> > > breaker, help prove or disprove the above?
> 
> > In simple terms, when you create an immutable object it can contain
> 
> > only references to pre-existing objects, but in order to create a cycle
> 
> > you need to make an object refer to another which is created later, so
> 
> > it's not possible to create a cycle out of immutable objects.
> 
> > However, using Python's C API it _is_ possible to create such a cycle,
> 
> > by mutating an otherwise-immutable tuple (see PyTuple_SetItem and
> 
> > PyTuple_SET_ITEM).
> 
> Are there any precedents for storing uncounted references to PyObject's?
> 
> One apparent problematic case is creating an iterator to a set, then adding 
> it to the set.  However the operation is a modification, and causes the 
> iterator to be removed from the secondary list before the set is examined for 
> collection.
> 
> Otherwise, the iterator keeps a counted reference to the set, but the set 
> does not keep a counted reference to the iterator, so the iterator will 
> always be freed first.  Therefore, the set's secondary list will be empty 
> when the set is freed.
> 
> Concurrent addition and deletion of iterators should be disabled, and the 
> iterators should remove themselves from the set's secondary list before they 
> decrement their references to the set.
> 
> Please refresh the earlier diagram; counted references are distinguished 
> separately.  Reposting: 
> http://home.comcast.net/~castironpi-misc/clpy-0062%20set%20iterators.png

The patch for the above is only 40-60 lines.  However it introduces two new 
concepts.

The first is a "linked list", a classic dynamic data structure, first developed 
in 1955, cf. http://en.wikipedia.org/wiki/Linked_list .  Linked lists are 
absent in Python, including the standard library and CPython implementation, 
beyond the weak reference mechanism and garbage collector.  The 
"collections.deque" structure shares some of the linked list interface but uses 
arrays.

The second is "uncounted references".  The uncounted references are references 
to "set iterators" exclusively, exist only internally to "set" objects, and are 
invisible to the rest of the program.  The reason for the exception is that 
iterators are unique in the Python Data Model; iterators consist of a single 
immutable reference, unlike both immutable types such as strings and numbers, 
as well as container types.  Counted references could be used instead, but 
would be consistently wasted work for the garbage collector, though the benefit 
to programmers' peace of mind could be significant.

Please share your opinion!  Do you agree that the internal list resolves the 
inconsistency?  Do you agree with the strategy?  Do you agree that uncounted 
references are justified to introduce, or are counted references preferable?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Saturday, August 18, 2012 5:14:05 PM UTC-5, MRAB wrote:
> On 18/08/2012 21:29, Aaron Brady wrote:
> 
> > On Friday, August 17, 2012 4:57:41 PM UTC-5, Chris Angelico wrote:
> 
> >> On Sat, Aug 18, 2012 at 4:37 AM, Aaron Brady  wrote:
> 
> >>
> 
> >> > Is there a problem with hacking on the Beta?
> 
> >>
> 
> >>
> 
> >>
> 
> >> Nope. Hack on the beta, then when the release arrives, rebase your
> 
> >>
> 
> >> work onto it. I doubt that anything of this nature will be changed
> 
> >>
> 
> >> between now and then.
> 
> >>
> 
> >>
> 
> >>
> 
> >> ChrisA
> 
> >
> 
> > Thanks Chris, your post was encouraging.
> 
> >
> 
> > I have a question about involving the 'tp_clear' field of the types.
> 
> >
> 
> > http://docs.python.org/dev/c-api/typeobj.html#PyTypeObject.tp_clear
> 
> >
> 
> > '''
> 
> > ...The tuple type does not implement a tp_clear function, because it’s 
> > possible to prove that no reference cycle can be composed entirely of 
> > tuples.
> 
> > '''
> 
> >
> 
> > I didn't follow the reasoning in the proof; the premise is necessary but 
> > IMHO not obviously sufficient.  Nevertheless, the earlier diagram contains 
> > an overt homogeneous reference cycle.
> 
> >
> 
> > Reposting: 
> > http://home.comcast.net/~castironpi-misc/clpy-0062%20set%20iterators.png
> 
> >
> 
> > In my estimate, the 'tp_traverse' and 'tp_clear' fields of the set doesn't 
> > need to visit the auxiliary collection; the same fields of the iterators 
> > don't need to visit the primary set or other iterators; and references in 
> > the linked list don't need to be included in the iterators' reference 
> > counts.
> 
> >
> 
> > Can someone who is more familiar with the cycle detector and cycle breaker, 
> > help prove or disprove the above?
> 
> >
> 
> In simple terms, when you create an immutable object it can contain
> 
> only references to pre-existing objects, but in order to create a cycle
> 
> you need to make an object refer to another which is created later, so
> 
> it's not possible to create a cycle out of immutable objects.
> 
> 
> 
> However, using Python's C API it _is_ possible to create such a cycle,
> 
> by mutating an otherwise-immutable tuple (see PyTuple_SetItem and
> 
> PyTuple_SET_ITEM).

Are there any precedents for storing uncounted references to PyObject's?

One apparent problematic case is creating an iterator to a set, then adding it 
to the set.  However the operation is a modification, and causes the iterator 
to be removed from the secondary list before the set is examined for collection.

Otherwise, the iterator keeps a counted reference to the set, but the set does 
not keep a counted reference to the iterator, so the iterator will always be 
freed first.  Therefore, the set's secondary list will be empty when the set is 
freed.

Concurrent addition and deletion of iterators should be disabled, and the 
iterators should remove themselves from the set's secondary list before they 
decrement their references to the set.

Please refresh the earlier diagram; counted references are distinguished 
separately.  Reposting: 
http://home.comcast.net/~castironpi-misc/clpy-0062%20set%20iterators.png
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Friday, August 17, 2012 4:57:41 PM UTC-5, Chris Angelico wrote:
> On Sat, Aug 18, 2012 at 4:37 AM, Aaron Brady  wrote:
> 
> > Is there a problem with hacking on the Beta?
> 
> 
> 
> Nope. Hack on the beta, then when the release arrives, rebase your
> 
> work onto it. I doubt that anything of this nature will be changed
> 
> between now and then.
> 
> 
> 
> ChrisA

Thanks Chris, your post was encouraging.

I have a question about involving the 'tp_clear' field of the types.

http://docs.python.org/dev/c-api/typeobj.html#PyTypeObject.tp_clear

'''
...The tuple type does not implement a tp_clear function, because it’s possible 
to prove that no reference cycle can be composed entirely of tuples. 
'''

I didn't follow the reasoning in the proof; the premise is necessary but IMHO 
not obviously sufficient.  Nevertheless, the earlier diagram contains an overt 
homogeneous reference cycle.

Reposting: 
http://home.comcast.net/~castironpi-misc/clpy-0062%20set%20iterators.png

In my estimate, the 'tp_traverse' and 'tp_clear' fields of the set doesn't need 
to visit the auxiliary collection; the same fields of the iterators don't need 
to visit the primary set or other iterators; and references in the linked list 
don't need to be included in the iterators' reference counts.

Can someone who is more familiar with the cycle detector and cycle breaker, 
help prove or disprove the above?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Thursday, August 16, 2012 9:24:44 PM UTC-5, Steven D'Aprano wrote:
> On Thu, 16 Aug 2012 19:11:19 -0400, Dave Angel wrote:
> 
> 
> 
> > On 08/16/2012 05:26 PM, Paul Rubin wrote:
> 
> >> Dave Angel  writes:
> 
> >>> Everything else is implementation defined.  Why should an
> 
> >>> implementation be forced to have ANY extra data structure to detect a
> 
> >>> static bug in the caller's code?
> 
> >> For the same reason the interpreter checks for type errors at runtime
> 
> >> and raises TypeError, instead of letting the program go into the weeds.
> 
> > 
> 
> > There's an enormous difference between type errors, which affect the low
> 
> > level dispatch, and checking for whether a dict has changed and may have
> 
> > invalidated the iterator.  If we were really going to keep track of what
> 
> > iterators are tracking a given dict or set, why stop there?  Why not
> 
> > check if another process has changed a file we're iterating through?  Or
> 
> > ...
> 
> 
> 
> Which is why Python doesn't do it -- because it is (claimed to be) 
> 
> excessively expensive for the benefit that you would get.
> 
> 
> 
> Not because it is a matter of principle that data integrity is 
> 
> unimportant. Data integrity *is* important, but in the opinion of the 
> 
> people who wrote these particular data structures, the effort required to 
> 
> guarantee correct iteration in the face of mutation is too expensive for 
> 
> the benefit.
> 
> 
> 
> Are they right? I don't know. I know that the list sort method goes to a 
> 
> lot of trouble to prevent code from modifying lists while they are being 
> 
> sorted. During the sort, the list temporarily appears to be empty to 
> 
> anything which attempts to access it. So at least sometimes, the Python 
> 
> developers spend effort to ensure data integrity.
> 
> 
> 
> Luckily, Python is open source. If anyone thinks that sets and dicts 
> 
> should include more code protecting against mutation-during-iteration, 
> 
> they are more than welcome to come up with a patch. Don't forget unit and 
> 
> regression tests, and also a set of timing results which show that the 
> 
> slow-down isn't excessive.

I contribute a patch some time ago.  It wasn't accepted.  However this thread 
seems to show a moderately more favorable sentiment than that one.

Is there a problem with hacking on the Beta?  Or should I wait for the Release? 
 Does anyone want to help me with the changes?  Perhaps P. Rubin could 
contribute the variation he suggested as well.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Thursday, August 16, 2012 9:30:42 PM UTC-5, Paul Rubin wrote:
> Steven D'Aprano  writes:
> 
> > Luckily, Python is open source. If anyone thinks that sets and dicts 
> 
> > should include more code protecting against mutation-during-iteration, 
> 
> > they are more than welcome to come up with a patch. Don't forget unit and 
> 
> > regression tests, and also a set of timing results which show that the 
> 
> > slow-down isn't excessive.
> 
> 
> 
> It could be a debugging option, in which case even a fairly significant
> 
> slowdown is acceptable.

Another possibility is to use the 'gc.get_referrers' mechanism to obtain the 
iterators.


import gc
a= set( ( 0, 1, 2 ) )
b= iter( a )
c= iter( a )
d= iter( a )
print( gc.get_referrers( a ) )

Output:

[, , 
, [others] ]

This approach wouldn't be as time-efficient as a dedicated secondary structure, 
due to the other objects which refer to the set, including variable namespaces.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Thursday, August 16, 2012 8:01:39 PM UTC-5, Paul Rubin wrote:
> Ian Kelly  writes:
> 
> > With regard to key insertion and deletion while iterating over a dict
> 
> > or set, though, there is just no good reason to be doing that
> 
> > (especially as the result is very implementation-specific), and I
> 
> > wouldn't mind a more complete low-level check against it as long as
> 
> > it's not too expensive (which is not clearly the case with the current
> 
> > suggestion at all).
> 
> 
> 
> One possible approach is to freeze the dictionary against modification
> 
> while any iterator is open on it.  You could keep a count of active
> 
> iterators in the dict structure, adjusting it whenever an iterator is
> 
> created or closed/destroyed.

Good point.  Your approach is another consistent solution.

The difference is in where the exception is raised.  Invalidating the iterators 
raises an exception when they're called.  Locking the set/dict raises an 
exception on 'add' and 'remove' calls.

The latter also forces additional statements to delete iterators before they 
leave scope in some cases.  We wouldn't be able to make modifications in a 
'for' suite, even if followed by a 'break', which could be a problem for 
existing code.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: set and dict iteration

[Aaron Brady]

On Thursday, August 16, 2012 6:07:40 PM UTC-5, Ian wrote:
> On Thu, Aug 16, 2012 at 4:55 PM, Ian Kelly  wrote:
> 
> > On Thu, Aug 16, 2012 at 12:00 PM, Aaron Brady  wrote:
> 
> >> The inconsistency is, if we remove an element from a set and add another 
> >> during iteration, the new element might appear later in the iteration, and 
> >> might not, depending on the hash code; therefore comparing the size of the 
> >> set between iterations isn't adequate.  Example:
> 
> >
> 
> > It can be more than just the new element.  For example, here the
> 
> > entire set is repeated (Python 3.2):
> 
> >
> 
> >>>> s = set(range(8, 13))
> 
> >>>> it = iter(s)
> 
> >>>> from itertools import islice
> 
> >>>> list(islice(it, 5))  # avoid exhausting the iterator
> 
> > [8, 9, 10, 11, 12]
> 
> >>>> s.add(13)
> 
> >>>> s.remove(13)
> 
> >>>> list(it)
> 
> > [8, 9, 10, 11, 12]
> 
> >
> 
> > This occurs because the addition of the sixth item triggers a resize
> 
> > of the underlying hash table, and the existing items, which were
> 
> > originally in slots 0-4, are now in slots 8-12.
> 
> 
> 
> Another curious example:
> 
> 
> 
> >>> s = set(range(8, 48, 8))
> 
> >>> s
> 
> {8, 16, 40, 24, 32}
> 
> >>> it = iter(s)
> 
> >>> from itertools import islice
> 
> >>> list(islice(it, 4))
> 
> [8, 16, 40, 24]
> 
> >>> s.add(48)
> 
> >>> s.remove(48)
> 
> >>> list(it)
> 
> [8, 16, 40, 24]
> 
> 
> 
> Hey, what happened to 32?

Good examples.  The former occurs without the 'islice' as well.

s= set( range( 8, 13 ) ) 
it= iter( s ) 
print( [ next( it ) for _ in range( 5 ) ] )
s.add( 13 ) 
s.remove( 13 ) 
print( [ next( it ) for _ in range( 5 ) ] )

Output:

[8, 9, 10, 11, 12]
[8, 9, 10, 11, 12]
-- 
http://mail.python.org/mailman/listinfo/python-list

set and dict iteration

[Aaron Brady]

Hello,

I observed an inconsistency in the behavior of 'set' and 'dict' iterators.  It 
is "by design" according to the docs.

'''
http://docs.python.org/dev/library/stdtypes.html#dict-views

iter(dictview).  Iterating views while adding or deleting entries in the 
dictionary may raise a RuntimeError or fail to iterate over all entries.
'''

The 'set' has the same behavior.  Iteration might also complete successfully.

The inconsistency is, if we remove an element from a set and add another during 
iteration, the new element might appear later in the iteration, and might not, 
depending on the hash code; therefore comparing the size of the set between 
iterations isn't adequate.  Example:

http://home.comcast.net/~castironpi-misc/clpy-0062%20set%20iterators.py  

'''
# py: { 'ver': '3' }

set0= set( ( 1, 2 ) )

iter0= iter( set0 )
print( next( iter0 ) )

set0.add( 3 )
set0.remove( 2 )
print( next( iter0 ) )


print( )

set0= set( ( 6, 7 ) )

iter0= iter( set0 )
print( next( iter0 ) )

set0.add( 8 )
set0.remove( 7 )
print( next( iter0 ) )
'''

Output:

'''
1
3

6
Traceback (most recent call last):
  File [...] line 22, in 
print( next( iter0 ) )
StopIteration
'''

Iteration should behave the same regardless of the contents of the set.  
Continuing iteration over sets and dicts after a modification isn't defined; it 
should unconditionally raise an error.

What's going on, is '8' is added before the position of the iterator due to 
hashing in the second part, but the size doesn't change, so the iterator 
reaches the end of the set after '7' is removed.

The inconsistency isn't easily solved.  One possibility is to use a timestamp 
or other serial index in the object and iterators, and compare them on every 
iteration to determine if a modification has occurred.

Another possibility which the author prefers, is to maintain a secondary 
collection of the iterators of an object, and invalidate them upon 
modification.  The applicable collection structure is a doubly-linked linked 
list, informally depicted:

http://home.comcast.net/~castironpi-misc/clpy-0062%20set%20iterators.png

Upon modification, the set traverses its iterators, setting an 'invalid' flag 
on each; and subsequent calls to any of them raise an 'IterationError'.  Adding 
and removing iterators to and from the secondary list is performed in O( 1 ) 
time with no penalty.

The above example depicted a 'Set'.  'Dicts' have the same anomaly, but the 
solution is ambiguous, since dict values can be changed meaningfully without 
altering the structure of the object.  In the author's opinion, the dict should 
not raise an 'IterationError' on value changes, only key changes like the set, 
but the argument isn't conclusive.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Rich comparison methods don't work in sets?

[Aaron Brady]

On Jun 20, 9:27 am, MRAB  wrote:
> Gustavo Narea wrote:
> > Hello again, everybody.
>
> > Thank you very much for your responses. You guessed right, I didn't
> > use the __hash__ method (and I forgot to mention that, sorry).
>
> > And unfortunately, I think I can't make them hashable, because the
> > objects are compared based on their attributes, which are in turn
> > other kind of objects compared based on other attributes. All these
> > class instances are compared with __eq__/__ne__ and they wrap
> > relatively complex data which would be hard to attempt to represent
> > them unambiguously using a 32-bit integer. That's why I'm afraid I
> > cannot use hashables.
>
> > I guess I'll have to use something like the function of my first
> > post. :(
>
> A hash doesn't have to be unambiguous. It's just a way to reduce the
> number of equality checks that have to be made.
>
> Could you create a hash from a tuple of attributes?
>
> If all else fails you could define a hash function that returns a
> constant.

Ooo sneaky.  +1 fancy.

> You would, however, lose the speed advantage that a hash
> gives in narrowing down the possible matches.

Are there /any/ immutable members of the objects?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 19, 7:00 am, "Rhodri James" 
wrote:
> On Fri, 19 Jun 2009 14:24:34 +0100, Aaron Brady   
> wrote:
>
> > You are not being any help, Rhodri, in your question.
>
> To you, perhaps not.  To me, it has at least had the effect of making
> what you're trying to do (write a pythonic object database) clearer.

I stand corrected.  Some help you were.  I have nothing further to say
about it.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 19, 7:45 am, a...@pythoncraft.com (Aahz) wrote:
> In article 
> ,
> Aaron Brady   wrote:
>
>
>
> >You are not being any help, Rhodri, in your question.  
>
> Maybe not, but honestly, you're getting pretty close to going back in my
> killfile.  Although you're no longer trolling this group, I find your
> writing difficult to read at best; answering questions from people like
> Rhodri is about your only chance to reach people like me.  Even without
> the killfile, I'm skipping about 60-80% of your posts.  It's a free
> Usenet, of course, so you're under no obligation to pay any attention to
> me, but I think you ought to consider the merits of changing your
> opinion of Rhodri's questions.
>
> Side note: the rhetorical trick of inserting a person's name in a
> statement is often intended and received as a mildly patronizing insult.
> It's not at all clear to me whether that was your intent; if not, you
> might want to change your writing style a bit.
> --
> Aahz (a...@pythoncraft.com)           <*>        http://www.pythoncraft.com/
>
> "as long as we like the same operating system, things are cool." --piranha

> Side note: the rhetorical trick of inserting a person's name in a

This is a welcome digression for me.  I wasn't sure that my idea was
being taken seriously.  On a temperature scale from freezing to
boiling, I took Rhodri's message to be somewhere in the single
digits-- quite chilly.  Maybe if it hadn't made me feel so defensive,
I could've just said so.  But, judging from his reply, he got the
message.  

> I find your
> writing difficult to read at best;
...
> I'm skipping about 60-80% of your posts.

My loss.



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Exotic Logics

[Aaron Brady]

On Jun 17, 10:32 am, Aaron Brady  wrote:
> On Jun 17, 10:23 am, Mensanator  wrote:
snip
> > > I think high and low /voltages/, though continuous and approximate,
> > > might satisfy this.
>
> > > There are no such things as electrons,
>
> > I've got a Tesla coil if you'd like to meet some electrons personally.
>
> The Wall Street Journal ran an article about Asian pleasure markets;
> they provide a-- quote-- "perfectly reasonable professional option".

For shame, Lieutenant.  Always cite your source.

http://online.wsj.com/article/SB124416693109987685.html

Last paragraph, second to last sentence.  It was a book review of _The
East, the West, and Sex_ by Richard Bernstein .
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Rich comparison methods don't work in sets?

[Aaron Brady]

On Jun 19, 12:42 pm, Chris Rebert  wrote:
> On Fri, Jun 19, 2009 at 12:02 PM, Gustavo Narea wrote:
> > Hello, everyone.
>
> > I've noticed that if I have a class with so-called "rich comparison"
> > methods
> > (__eq__, __ne__, etc.), when its instances are included in a set,
> > set.__contains__/__eq__ won't call the .__eq__ method of the elements
> > and thus
> > the code below:
snip

> > """
> > obj1 = RichComparisonClass()
> > obj2 = RichComparisonClass()
>
> > set1 = set([obj1])
> > set2 = set([obj2])
>
> > if obj1 == obj2:
> >    print "Objects 1 and 2 are equivalent"
> > else:
> >    print "Objects 1 and 2 aren't equivalent"
>
> > if set1 == set2:
> >    print "Sets 1 and 2 are equivalent"
> > else:
> >    print "Sets 1 and 2 aren't equivalent"
> > """
>
> > Would output:
> > """
> > Objects 1 and 2 are equivalent
> > Sets 1 and 2 aren't equivalent
> > """
>
> > instead of:
> > """
> > Objects 1 and 2 are equivalent
> > Sets 1 and 2 are equivalent
> > """
>
> > How can I work around this? The only solution that comes to my mind is
>
> Note that sets are dict-based and thus use hash tables internally.
> Thus, you must implement a sensible __hash__ method.
> The default __hash__ provided by class `object` uses the object ID for the 
> hash.
>
> Since id(x) == id(y)  iff  x is y, and
> hash(x) != hash(y) implies x != y,
> If you don't implement __hash__, object's implementation causes every
> distinct object of your type to be considered unequal a priori, so it
> doesn't bother to check any further by calling the comparison methods.
>
> Cheers,
> Chris
> --http://blog.rebertia.com

You're using sets for uniqueness and faster lookups than a list or
other collection.  Uniqueness is determined by hash key first, then by
identity (due to an idiosyncrasy of 'RichCompareBool'), then by rich
equality.  If you want to compare to every element, you can't use
sets, because they take short-cuts by omitting many of the
comparisons.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fastest native python database?

[Aaron Brady]

On Jun 17, 8:28 pm, per  wrote:
> hi all,
>
> i'm looking for a native python package to run a very simple data
> base. i was originally using cpickle with dictionaries for my problem,
> but i was making dictionaries out of very large text files (around
> 1000MB in size) and pickling was simply too slow.
>
> i am not looking for fancy SQL operations, just very simple data base
> operations (doesn't have to be SQL style) and my preference is for a
> module that just needs python and doesn't require me to run a separate
> data base like Sybase or MySQL.
>
> does anyone have any recommendations? the only candidates i've seen
> are snaklesql and buzhug... any thoughts/benchmarks on these?
>
> any info on this would be greatly appreciated. thank you

I have one or two.  If the objects you're pickling are all
dictionaries, you could store file names in a master 'shelve' object,
and nested data in the corresponding files.

Otherwise, it may be pretty cheap to write the operations by hand
using ctypes if you only need a few, though that can get precarious
quickly.  Just like life, huh?

Lastly, the 'sqlite3' module's bark is worse than its byte.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 17, 3:53 pm, "Rhodri James" 
wrote:
> On Wed, 17 Jun 2009 16:06:22 +0100, Aaron Brady   
> wrote:
>
>
>
> > On Jun 16, 10:09 am, Mike Kazantsev  wrote:
> >> On Tue, 16 Jun 2009 06:57:13 -0700 (PDT)
>
> >> Aaron Brady  wrote:
> >> > Making the charitable interpretation that this was the extent of c-l-
> >> > py's support and enthusiasm for my idea, I will now go into mourning.
> >> > Death occurred at oh-eight-hundred.  Rest in peace, support &
> >> > enthusiasm.
>
> >> I've read this thread from the beginning, being tempted to insert
> >> remarks about shelve module or ORMs like SQLAlchemy, but that'd be
> >> meaningless without the problem description, which I haven't seen
> >> anywhere. Is it some trick idea like "let's walk on our heads"?
>
> > More like bronze them, or hang them on a tackboard.  You haven't
> > convinced me that it's not a problem, or that it's an easy one.
>
> Unfortunately it's up to you to demonstrate that it is a problem,
> whichever of the many possible 'it's you're talking about.  So far,
> the question "Why would I want to use this?  What's the use case?"
> has gone unanswered, and I'm sure I'm not the only baffled by it.

I can demonstrate it's a problem; many things are.

The subject line says it all.  SQL will persist (and share for IPC,
after a fashion) sets of statically typed tuples.  But we have need
for sets of dynamically typed tuples, in volatile storage at the very
least, or no one would like Python.

Whether we have need for them in persistent storage remains to be
seen.

POSH Python Object SHaring has been at least one student's graduate
thesis.  It couldn't hurt to pursue it if one has time, as with many
things.  It's pretty likely my estimates of end user usefulness of
'KeepDB', say, derive sturdily and steadily from our estimated
popularity of Python.  So much for my motives, though not for my
opportunities.

My implementation has followed two separate paths: one using SQL for a
back-end host; the other using hand-coded bytes on disk, complete with
large file support using views, and on-disk (or in-buffer) memory
management, which incidentally I spoke up about about a year ago.  It
was the topic of an honors lecture in the Data Structures course in my
undergrad.  For the record, I still can't beat a log-N balanced tree
for finding free nodes, but I am operating on the assumption that the
smallest sufficient region is the best to return from 'alloc'.

You are not being any help, Rhodri, in your question.  Its chief
virtue is fidelity of programming, that is, what you write is most
true to what you mean.  If you have an object on disk, and want the
third element of its 'coords' list attribute, it's a waste to do
anything other than find its disk address, then the disk address of
its 'coords' attribute, then the disk address of the third element of
that, such as say, loading or worse, processing, the entire structure;
even if, and in light of parallel considerations about economic market
transparency and efficiency, that is to say, economy, maybe /
especially/ if, you are having a lowly machine make do.  You don't
want to open every box in your basement to find your tennis racket;
you want to open one box, find the racket in the index, then open all
and only the boxes that 'nestedly' contain the racket.  (Yes, that's /
all/ and /only/, or 'if and only if'-- not to be confused with 'one
and only'.)

It's more economic.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regarding Python is scripting language or not

[Aaron Brady]

On Jun 18, 6:07 am, Jochen Schulz  wrote:
> Terry Reedy:
>
> > Jochen Schulz wrote:
>
> >> If, by "object-oriented" you mean "everything has to be put into
> >> classes", then Python is not object-oriented.
>
> > That depends on what you mean by 'put into classes' (and 'everything').
>
> :) What I meant was that in Python you can write code without defining
> your own classes at all. I had Java in mind where you cannot write a
> program without using the keyword 'class'. But I think we agree here.

Numbers have infinite precision.  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regarding Python is scripting language or not

[Aaron Brady]

On Jun 17, 7:38 am, Jean-Michel Pichavant 
wrote:
> abhishek goswami wrote:
> > Hi,
> > I have very basic question about Python that do we consider pyhton as
> > script language.
> > I searched in google but it becomes more confusion for me. After some
> > analysis I came to know that Python support oops .
>
> > Can anyone Guide me that Python is Oject oriented programming language
> > or Script language
>
> > Abhishek Goswami
> > Chennai
> > Phone No -0996227099
>
> > 
> > ICC World Twenty20 England '09 exclusively on YAHOO! CRICKET
> > 
>
> Depends on what you are calling a scripting language.
> Refering to wikipedia,
> "A *scripting language*, *script language* or *extension language* is a
> programming language 
> that allows some control of a single or many software application(s)
> ."
>
> Python is definitely OOP oriented and I don't think it fits in the
> script definition above.
> Python is interpreted and platform independent, but you can still build
> standalone platform dependent binaries if required.
>
> Regarding your last question, I'm not sure scripting and OOP language
> are not compatible, I'm pretty sure you'll be able to find OOP scripting
> language.
>
> Jean-Michel

I'm not the only one that thinks that Java is a joke, though I provoke
flaming, and do not know it fluently.  It's all big words and
handcuffs.

To quote my favorite t.v. show, 'You imply disparity where none
exists.'  You're trying to 'pigeon-hole', where the subjects are
fairly complex, even though not ultimately continuous, but still not
binary.  You might as well be asking whether it's rainy or sunny out.
Even day and night only account for 22-23 hours out of our day.

Speaking of big words, programming languages vary in a number of
dimensions.  Regardless of what surface you use to divide the space,
you'll have data points which aren't quite intuitively aligned with
the rest of their category; not to shirk the burden of proof.  The
high complexity of Python blurs the partition.  You might give it a
center, and some membership density function that decreases with the
distance from it, like the volume of a loudspeaker.  I'm not sure
whether you would define these from use data, or something more a
priori.  The former doesn't require as much contortion.  Some have
proposed the same tactic for culture division and nation borders,
incidentally, as one 'compromization' tactic; that is, to 'fuzzily'
classify regions, and languages likewise.  It would make Python, say
10% appropriate for scripting, and 90% object-oriented, just as 10% of
'our' police comes from, and 10% of our taxes goes to Sweden.
However, you've never heard of a 70% Catholic, and further, the
logistics on that formulation don't align quite rightly: it would be
more like, 'we' pay takes to Sweden at 10% of the tax rate at its
capital and anywhere else that only Sweden influences.  Some places of
commerce even accept the kroner too.  That still might not make you be
70% Catholic, but definitely ten people definitely definitely can.

Come to think of it, the percentages don't have to add up to 1, it's
more like Python is 50% appropriate for scripting, and object-oriented
in the high nineties.  I guess I was just trying to be politically
correct, impartial, or otherwise unbiased, though consequently rigid
in the meantime.  Sadly so.

Linguists define distinctions between natural languages by measure of
mutual interpretability.  The two most similar languages out there may
be practically the same; the two most different may not have even any
syntax in common; and the one most different from the one closest to
it may well be Python.  Fancy that!

In this participant's humble opinion, no.  There are better scripting
languages out there.

In his arrogant opinion, no.  It 'sets the curve'.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Exotic Logics

[Aaron Brady]

On Jun 17, 10:23 am, Mensanator  wrote:
> On Jun 17, 11:59 am, Aaron Brady  wrote:
>
>
>
> > On Jun 17, 1:44 am, Steven D'Aprano
>
> >  wrote:
> > > On Tue, 16 Jun 2009 22:46:14 -0700, William Clifford wrote:
> > > > I was staring at a logic table the other day, and I asked myself, "what
> > > > if one wanted to play with exotic logics; how might one do it?"
>
> > > This might be useful for you, and if not useful, at least it might blow
> > > your mind like it did mine.
>
> > > (This is not original to me -- I didn't create it. However, I can't find
> > > the original source.)
>
> > > Imagine for a moment that there are no boolean values.
> > > There are no numbers.  They were never invented.
> > > There are no classes.
> > > There are no objects.
> > > There are only functions.
>
> > > Could you define functions that act like boolean values? And could you
> > > define other functions to operate on them?
>
> > snip
>
> > I think high and low /voltages/, though continuous and approximate,
> > might satisfy this.
>
> > There are no such things as electrons,
>
> I've got a Tesla coil if you'd like to meet some electrons personally.

The Wall Street Journal ran an article about Asian pleasure markets;
they provide a-- quote-- "perfectly reasonable professional option".
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Exotic Logics

[Aaron Brady]

On Jun 17, 10:05 am, pdpi  wrote:
> On Jun 17, 5:37 pm, Lie Ryan  wrote:
>
>
>
> > Steven D'Aprano wrote:
> > > On Tue, 16 Jun 2009 22:46:14 -0700, William Clifford wrote:
>
> > >> I was staring at a logic table the other day, and I asked myself, "what
> > >> if one wanted to play with exotic logics; how might one do it?"
>
> > > This might be useful for you, and if not useful, at least it might blow
> > > your mind like it did mine.
>
> > > (This is not original to me -- I didn't create it. However, I can't find
> > > the original source.)
>
> > > Imagine for a moment that there are no boolean values.
> > > There are no numbers.  They were never invented.
> > > There are no classes.
> > > There are no objects.
> > > There are only functions.
>
> > > Could you define functions that act like boolean values? And could you
> > > define other functions to operate on them?
>
> > > def true(x, y):
> > >     return x
>
> > > def false(x, y):
> > >     return y
>
> > > def print_bool(b):
> > >     print b("true", "false")
>
> > String isn't considered object?
>
> > Also, b/true()/false() is a function object, isn't it? Unless function
> > is first-class, you can't pass them around like that, since you need a
> > function pointer (a.k.a number); but if function is first-class then
> > there it is an object.
>
> What Steven was doing was implementing some of the more basic stuff
> from Lambda calculus in python. If you're implementing a different
> system in an existing language, you'll need to use _some_ facilities
> of the original language to interface with the outside world.

Sir!  Entropy levels are approaching dangerously low levels.  We don't
even have enough entropy to fi
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Exotic Logics

[Aaron Brady]

On Jun 17, 10:04 am, Aaron Brady  wrote:
snip
> You (OP) may be interested in the definitions of the fuzzy operators:
>
> and( x, y ) := min( x, y )
> or( x, y ) := max( x, y )
> not( x ) := 1 (one)- x
> nand( x, y ) := not( and( x, y ) ) = 1- min( x, y )
>
> Defining 'xor' as '( x or y ) and ( not( x and y ) )', we have:
>
> xor( x, y ) := min( max( x, y ), 1- min( x, y ) )
>
> However, defining 'xor' as '( x and not y ) or ( y and not x )', we
> don't have:
>
> xor( x, y ) := max( min( x, 1- y ), min( y, 1-x ) )

Corollary:

xor1( x, y ) === xor2( x, y ).

Non-exhaustive demonstration, excerpt:

>>> def xor1( x, y ):
... return min( max( x, y ), 1- min( x, y ) )
...
>>> def xor2( x, y ):
... return max( min( x, 1- y ), min( y, 1- x ) )
...
>>> for i in range( 0, 11, 2 ):
... for j in range( 0, 11, 2 ):
... print i, j, xor2( x, y )*10, '  ',
... print

0 0 0.00 2 2.00 4 4.00 6 6.00 8 8.0
2 0 2.02 2 2.02 4 4.02 6 6.02 8 8.0
4 0 4.04 2 4.04 4 4.04 6 6.04 8 6.0
6 0 6.06 2 6.06 4 6.06 6 4.06 8 4.0
8 0 8.08 2 8.08 4 6.08 6 4.08 8 2.0
10 0 10.0  10 2 8.0   10 4 6.0   10 6 4.0   10 8 2.0

They appear to be equal.

I forgot to mention, fuzzy values take on values from the continuous
open or closed range 0 to 1.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Exotic Logics

[Aaron Brady]

On Jun 17, 1:28 am, Steven D'Aprano
 wrote:
> On Tue, 16 Jun 2009 22:46:14 -0700, William Clifford wrote:
> > I was staring at a logic table the other day, and I asked myself, "what
> > if one wanted to play with exotic logics; how might one do it?"
>
> First question: what's an exotic logics?
>
> Do you mean things like three-value logic, fuzzy logic, probabilistic
> reasoning, etc?

You (OP) may be interested in the definitions of the fuzzy operators:

and( x, y ) := min( x, y )
or( x, y ) := max( x, y )
not( x ) := 1 (one)- x
nand( x, y ) := not( and( x, y ) ) = 1- min( x, y )

Defining 'xor' as '( x or y ) and ( not( x and y ) )', we have:

xor( x, y ) := min( max( x, y ), 1- min( x, y ) )

However, defining 'xor' as '( x and not y ) or ( y and not x )', we
don't have:

xor( x, y ) := max( min( x, 1- y ), min( y, 1-x ) )

Good question.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Exotic Logics

[Aaron Brady]

On Jun 17, 1:44 am, Steven D'Aprano
 wrote:
> On Tue, 16 Jun 2009 22:46:14 -0700, William Clifford wrote:
> > I was staring at a logic table the other day, and I asked myself, "what
> > if one wanted to play with exotic logics; how might one do it?"
>
> This might be useful for you, and if not useful, at least it might blow
> your mind like it did mine.
>
> (This is not original to me -- I didn't create it. However, I can't find
> the original source.)
>
> Imagine for a moment that there are no boolean values.
> There are no numbers.  They were never invented.
> There are no classes.
> There are no objects.
> There are only functions.
>
> Could you define functions that act like boolean values? And could you
> define other functions to operate on them?
snip

I think high and low /voltages/, though continuous and approximate,
might satisfy this.

There are no such things as electrons, only variations in density of
the luminiferous ether.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Executing a python script while it is running

[Aaron Brady]

On Jun 16, 3:48 pm, Zach Hobesh  wrote:
> > A lot more information would be useful.  What version of Python, and what
> > operating system environment?  Exactly what would you like to happen when
> > the batch file is invoked a second time?
>
> I'm running Python 2.6.2 on Windows.  I'm passing filenames to the
> batch files and I need all filenames to be processed.  I can't have
> any fails.  I'm working on logging any fails I do have so that I can
> maybe batch process at the end of the day.
>
> >  2) let them both run as separate processes
>
> This sounds like a good option, but I'm not totally sure on how to go
> about this?
>
> >  4) queue something to be processed when the first run finishes
>
> I had the same idea, but I believe it would involve having another
> python script run all day long, which wouldn't necessarily be a bad
> thing, but I'd like to explore other options as well.

This sort of falls under both categories, 2 & 4, and it will probably
be judged 'poor practice' by history.  We're all historians now, I
guess.

Windows has what's called a 'named mutex' for interprocess
synchro'tion.  Start your new process, acquire their shared mutex by
name, and block on it.  You will have one process for each file, but
only one will run at once.

You won't even need to build a shared library; 'ctypes' will suffice.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about None

[Aaron Brady]

On Jun 17, 5:47 am, Bruno Desthuilliers  wrote:
> John Yeung a écrit :
> > But mathematically speaking, it's intuitive that "nothing" would match
> > any type.
>
> IOW, what's the OP is after is not the None type, but some yet
> unexisting "Anything" type !-)

The behaviors of the 'anything' object are a subset of those of any
other object.

I don't believe that 'subset' is a proper characterization of the
relationship between the methods of a subclass and the methods of its
superclass.  But 'superset' may be.

Should 'object' inherit from None?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 16, 10:09 am, Mike Kazantsev  wrote:
> On Tue, 16 Jun 2009 06:57:13 -0700 (PDT)
>
> Aaron Brady  wrote:
> > Making the charitable interpretation that this was the extent of c-l-
> > py's support and enthusiasm for my idea, I will now go into mourning.
> > Death occurred at oh-eight-hundred.  Rest in peace, support &
> > enthusiasm.
>
> I've read this thread from the beginning, being tempted to insert
> remarks about shelve module or ORMs like SQLAlchemy, but that'd be
> meaningless without the problem description, which I haven't seen
> anywhere. Is it some trick idea like "let's walk on our heads"?

More like bronze them, or hang them on a tackboard.  You haven't
convinced me that it's not a problem, or that it's an easy one.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 15, 4:56 pm, Aaron Brady  wrote:
> On Jun 15, 11:10 am, Paul Rubin <http://phr...@nospam.invalid> wrote:
>
> > Aaron Brady  writes:
> > > > A real-world application of persistent data structures can be found 
> > > > here:
> > > >http://stevekrenzel.com/persistent-list
>
> > > Jaime, thanks for the link.  I contacted its author.
>
> > You might also look atwww.couchdb.org.
>
> I'm not much for the interface.  But the back end might match what I'm
> doing.

Making the charitable interpretation that this was the extent of c-l-
py's support and enthusiasm for my idea, I will now go into mourning.
Death occurred at oh-eight-hundred.  Rest in peace, support &
enthusiasm.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 15, 11:10 am, Paul Rubin <http://phr...@nospam.invalid> wrote:
> Aaron Brady  writes:
> > > A real-world application of persistent data structures can be found here:
> > >http://stevekrenzel.com/persistent-list
>
> > Jaime, thanks for the link.  I contacted its author.
>
> You might also look atwww.couchdb.org.

I'm not much for the interface.  But the back end might match what I'm
doing.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 15, 8:37 am, a...@pythoncraft.com (Aahz) wrote:
> In article <79mtt7f1r480...@mid.uni-berlin.de>,
> Diez B. Roggisch  wrote:
>
> >Aaron Brady wrote:
>
> >> Some time ago, I recommended a pursuit of keeping 'persistent
> >> composite' types on disk, to be read and updated at other times by
> >> other processes.  Databases provide this functionality, with the
> >> exception that field types in any given table are required to be
> >> uniform.  Python has no such restriction.
>
> >> I tried out an implementation of composite collections, specifically
> >> lists, sets, and dicts, using 'sqlite3' as a persistence back-end.
> >> It's significantly slower, but we might argue that attempting to do it
> >> by hand classifies as a premature optimization; it is easy to optimize
> >> debugged code.
>
> >Sounds like you are re-inventing the ZODB.
>
> ...or SQLAlchemy or pickles in a SQL BLOB or...

I recognize your aversion to my claim of making a new approach on
something.  I suppose an austere review of literature would compare
with the existing alternatives.

My approach does not define tables, so it is not SQL Alchemy; it is
not mere sugar for SQL.  It defines a 'set' class and a 'tuple' class,
so some of the functionality may (and probably should be expected to)
overlap.

http://www.sqlalchemy.org/docs/05/ormtutorial.html#define-and-create-a-table

My approach does not pickle objects, so it is not a mere BLOB pickle.
If a user writes, listA[ 50 ].append( "abcde" ), one addition is made
to an existing structure, and the remaining contents of 'listA' are
unread and unchanged.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about None

[Aaron Brady]

On Jun 14, 9:50 pm, Steven D'Aprano
 wrote:
> On Sun, 14 Jun 2009 19:14:10 -0400, Terry Reedy wrote:
> > Steven D'Aprano wrote:
>
> >> So-called "vacuous truth". It's often useful to have all([]) return
> >> true, but it's not *always* useful -- there are reasonable cases where
> >> the opposite behaviour would be useful:
> [...]
> > It seems to me that the absurd conclusion implied by the theorem
> > invalidates the theorem rather than supporting your point.
>
> I wouldn't go so far as to say the vacuous truth theorem ("empty
> statements are true") is invalidated. The Wikipedia article discusses
> various reasons why it's more correct (or at least more useful) to treat
> vacuous statements as true:
>
> http://en.wikipedia.org/wiki/Vacuous_truth
>
> But it's not without difficulties -- however those difficulties are
> smaller than those if you take vacuous statements as false in general.
snip

Those difficulties are pretty gaping.

I would start by dividing the natural language 'use cases' of 'if'
statements into imperative and declarative.

Declarative:

If it's raining, it's cloudy.

In this case, the assertion is meant to convey a general, non-
concrete, observation trend across space and time.  Its content is a
claim of 100% correlation between two statuses of the real world.

Imperative:

If you're out of bread, go buy some.

Here, the speaker is in a position of authority over the audience, who
will be following his/er commands, and acting under the speaker's
authority.  The speaker is meaning to convey conditional instructions,
for a possible circumstance.  There is no component of observation or
assertion.

We see this distinction in programming too.  Is the user merely
asserting a relation, or defining a procedure?

Implies( Raining( x ), Cloudy( x ) )

or

if OutOfBread( x ):
BuyBread( )

The 'if' statement is counterfactual in its native environment.  As
such, natural speakers never use it in vacuous cases, and it's not
naturally defined.

In a mathematical (ideal) environment, its semantics are artificially
constructed like any other math predicate, and proofs involving it
will define its vacuous case, or fail.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 14, 10:18 am, Jaime Fernandez del Rio 
wrote:
> On Sun, Jun 14, 2009 at 4:27 PM, Aaron Brady wrote:
>
> > Before I go and flesh out the entire interfaces for the provided
> > types, does anyone have a use for it?
>
> A real-world application of persistent data structures can be found here:
>
> http://stevekrenzel.com/persistent-list
>
> Jaime

Jaime, thanks for the link.  I contacted its author.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 15, 5:45 am, "Diez B. Roggisch"  wrote:
> Aaron Brady wrote:
> > Hi, please forgive the multi-posting on this general topic.
>
> > Some time ago, I recommended a pursuit of keeping 'persistent
> > composite' types on disk, to be read and updated at other times by
> > other processes.  Databases provide this functionality, with the
> > exception that field types in any given table are required to be
> > uniform.  Python has no such restriction.
>
> > I tried out an implementation of composite collections, specifically
> > lists, sets, and dicts, using 'sqlite3' as a persistence back-end.
> > It's significantly slower, but we might argue that attempting to do it
> > by hand classifies as a premature optimization; it is easy to optimize
> > debugged code.
>
> 
>
> Sounds like you are re-inventing the ZODB.
>
> Diez

Alright, Diez.  Here is some private consulting for free.

''Section 2.6.1:
The most common idiom that isn't caught by the ZODB is mutating a list
or dictionary''

My approach performs this for free.

The docs also don't mention interprocess communication, which is one
of the two primary functions that I satisfy in my approach.

The syntax is different, non-trivially so, and mine is more Pythonic.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about None

[Aaron Brady]

On Jun 14, 12:37 pm, Paul Rubin  wrote:
> Andre Engels  writes:
snip
> > type "thingy". A car is a single car. Nothing is zero cars, which is
> > not a car, just like two cars is not a car.
>
> That seems to confuse values with collections of them.  A car is not
> the same as a one-element list of cars.  Nothing is not the same as a
> zero-element list of cars.

Collections are a mathematical (ideal, cognitive) construct.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about None

[Aaron Brady]

On Jun 14, 10:02 am, Arnaud Delobelle  wrote:
snip
> guilt, it doesn't mean they will be convicted.  There needs to be enough
> evidence to convince the jury.  So it would be something like:
>
> if sum(guilt_weight(e) for e in evidence) > GUILT_THRESHOLD:
>    the defendant is guilty
>    ...
>
> --
> Arnaud

You all are only two months late.

http://groups.google.com/group/comp.lang.python/msg/7b63d0d11246ecad

Can't argue with results.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: persistent composites

[Aaron Brady]

On Jun 14, 8:24 am, Steven D'Aprano
 wrote:
> Aaron Brady wrote:
> > Some time ago, I recommended a pursuit of keeping 'persistent
> > composite' types on disk, to be read and updated at other times by
> > other processes.  Databases provide this functionality, with the
> > exception that field types in any given table are required to be
> > uniform.  Python has no such restriction.
> ...
> > I will be sure to kill any interest you might have by now, by
> > "revealing" a snippet of code.
>
> How about telling us what problem you're trying to solve? Giving us your
> implementation isn't useful if we don't even know what it's for. Persistent
> data on disk, okay, I get that -- but how is it different from (say) XML,
> or INI files, or pickles, or similar? Is the idea to have *live* data
> stored on disk, updated by multiple processes simultaneously? Don't assume
> that people will remember your recommendation from "some time ago".
>
> --
> Steven

It was no time at all in the Usenet world, I suppose.  There are lots
of strategies for storing data on a disk, and lots of strategies for
storing data in memory.  I was trying on disk what Python uses on
memory.

If you want to argue that Python isn't useful, you're in the wrong
place.  If you want to argue that Python's strategy is useful on
disks, you're talking to the right person.

I'll draw an analogy.  static data structures : sql & co. :: Python
data structures : this undertaking.  It has all the advantages over
SQL & XML structures that Python has over C structures: sort of a
'worst of neither' combination.  I haven't written a textbook on DSs,
though, so I don't quite know where to begin.  In my eyes, you're
practically asking, "What's your use case for Python?"  Either that,
or I don't know enough other languages.

I have nothing to use it for, but someone might, and it might be just
plumb interesting to fellow Pythoneers.  If I did, I'd have somewhere
to start.
-- 
http://mail.python.org/mailman/listinfo/python-list

persistent composites

[Aaron Brady]

Hi, please forgive the multi-posting on this general topic.

Some time ago, I recommended a pursuit of keeping 'persistent
composite' types on disk, to be read and updated at other times by
other processes.  Databases provide this functionality, with the
exception that field types in any given table are required to be
uniform.  Python has no such restriction.

I tried out an implementation of composite collections, specifically
lists, sets, and dicts, using 'sqlite3' as a persistence back-end.
It's significantly slower, but we might argue that attempting to do it
by hand classifies as a premature optimization; it is easy to optimize
debugged code.

The essentials of the implementation are:
  - each 'object' gets its own table.
= this includes immutable types
  - a reference count table
= when an object's ref. count reaches zero, its table is dropped
  - a type-map table
= maps object's table ID to a string of its type
  - a single 'entry point' table, with the table ID of the entry-point
object
= the entry point is the only data structure available to new
connections.  (I imagine it will usually be a list or dict.)

I will be sure to kill any interest you might have by now, by
"revealing" a snippet of code.

The object creation procedure:

def new_table( self, type ):
  ''' 'type' is a string, the name of the class the object is an
instance of '''
  cur= self.conn.cursor( )
  recs= cur.execute( '''SELECT max( tableid ) FROM refcounts''' )
  rec= cur.fetchone( )
  if rec[ 0 ] is None:
obid= 0
  else:
obid= rec[ 0 ]+ 1
  cur.execute( '''INSERT INTO types VALUES( ?, ? )''', ( obid,
type ) )
  cur.execute( '''INSERT INTO refcounts VALUES( ?, ? )''', ( obid,
1 ) )

The increment ref. count procedure:

def incref( self, obid ):
  cur= self.conn.cursor( )
  recs= cur.execute( '''SELECT count FROM refcounts WHERE tableid
= ?''', ( obid, ) )
  rec= recs.fetchone( )
  newct= rec[ 0 ]+ 1
  cur.execute( '''UPDATE refcounts SET count = ? WHERE tableid = ?''',
( newct, obid ) )

The top-level structure contains these two procedures, as well as
'setentry', 'getentry', and 'revive' procedures.

Most notably, user-defined types are possible.  The dict is merely a
persistent dict.  'revive' checks the global namespace by name for the
original type, subject to the same restrictions that we all know and
love that 'pickle' has.

As usual, deadlocks and cyclic garbage pose the usual problems.  The
approach I used last time was to maintain a graph of acquired locks,
and merely check for cycles to avert deadlocks, which would go in a
separate table.  For garbage, I can't find a better solution than
Python already uses.

>From the 3.0 docs:
gc.garbage

A list of objects which the collector found to be unreachable but
could not be freed (uncollectable objects).
...
Python doesn’t collect such [garbage] cycles automatically because, in
general, it isn’t possible for Python to guess a safe order in which
to run the __del__() methods. If you know a safe order, you can force
the issue by examining the garbage list, and explicitly breaking
cycles due to your objects within the list.

Before I go and flesh out the entire interfaces for the provided
types, does anyone have a use for it?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Different types of dicts with letter before the curly braces.

[Aaron Brady]

On Jun 14, 6:30 am, kindly  wrote:
> On Jun 14, 1:59 pm, Steven D'Aprano
snip
> I am glad the ordered dict will be in 2.7 and 3.1. I
> was just imagining what would be the next step in definition of
> structures. New languages like clojure have adopted the dict as top
> level.  I imagine immutable/thread safe/transactional dicts to be
> around soon in other languages to help with concurrency.  It would be
> nice if python was ahead of the game in this.

Just not ahead of its /time/.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Different types of dicts with letter before the curly braces.

[Aaron Brady]

On Jun 14, 4:02 am, kindly  wrote:
> I am sure people have thought of this before, but I cant find where.
> I think that python should adapt a way of defining different types of
> mapping functions by proceeding a letter before the curly brackets.
> i.e   ordered = o{},  multidict = m{}  (like paste multidict).  So you
> could define an ordered dict by newordered = o{"llvm" : "ptyhon",
> "parrot" : "perl"} .  (they should also probably have there own
> comprehensions as well o{foo for bar in foobar}).

That kind of stuff is highly explosive, unfortunately.  o{ }, m{ }, d
[ ], and q[ ] are just a few.  But 'collections' is kind of sparse.  I
expect you would also want users to be able to define their own
prefixes, no?  As is, the syntax is not atrocious:

oA= OrderedDict( [ ( pair1 ), ( pair2 ) ]
oA= o{ pair1, pair2 }

At least you can still include literals, and are not restricted to per-
line additions as in C.

snip
> Most packages that I have seen re-
> implement these different container types at one point anyway. It
> seems a shame they are not brought up to the top level, with
> potentially new, cleverer ones that have not been thought of yet.
snip

Do you merely want to populate the 'collections' module, or are the
prefixes essential to your idea?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can it be shorter?

[Aaron Brady]

On Jun 8, 9:50 am, Jean-Michel Pichavant 
wrote:
> Aaron Brady wrote:
> > Shorter is always better.
>
> > url+= { '/': '' }.get( url[ -1 ], '/' )
>
> Why bother with spaces or 3 letter-wide token, check this  :o) :
> x+={'/':''}.get(x[-1],'/')
>
> Apart from joking, the following proposed solution is by **far** the one
> I prefer
>
> > if not url.endswith('/'):
> >    url += '/'
>
> Maybe not the shorter, but the most concise and clear to me.

Why won't Python permit:

url.endswith( '/' ) or url.append( '/' )

?  Should it?  Do we find it just as concise and clear?  Does it
outweigh the priority of the immutability of strings?  It works on
lists, for example.  A sole mutating operation could create a highly
and finely tempered compromise with immutability.  Would it be
'append'?

I like Scott's and MRAB's idea for slicing, not indexing, the last
character.

The most literal translation of the original natural language is:

>>> #ensure that the url ends with a '/'
>>> ensure( url, string.endswith, '/' )

(Is it not?)  But the parameters aren't sufficient to define 'ensure'
generally, and it won't be able to mutate 'url' regardless.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can it be shorter?

[Aaron Brady]

On Jun 7, 6:13 pm, Paul Rubin <http://phr...@nospam.invalid> wrote:
> Aaron Brady  writes:
> > url+= { '/': '' }.get( url[ -1 ], '/' )
>
> > Shorter is always better.
>
> url = url.rstrip('/') + '/'

I was joking.  Sheesh.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can it be shorter?

[Aaron Brady]

On Jun 6, 8:07 am, "tsangpo"  wrote:
> I want to ensure that the url ends with a '/', now I have to do thisa like
> below.
> url = url + '' if url[-1] == '/' else '/'
>
> Is there a better way?

url+= { '/': '' }.get( url[ -1 ], '/' )

Shorter is always better.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fastest way to test file for string?

[Aaron Brady]

On Jun 5, 5:50 am, kj  wrote:
> Hi.  I need to implement, within a Python script, the same
> functionality as that of Unix's
>
>    grep -rl some_string some_directory
>
> I.e. find all the files under some_directory that contain the string
> "some_string".
snip

The 'mmap.mmap' class has a 'find' method.  Not what you asked for,
put possibly an alternative.  No regex 'search' method either.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: hash and __eq__

[Aaron Brady]

On May 30, 12:11 pm, Dennis Lee Bieber  wrote:
> On Sat, 30 May 2009 11:20:47 -0700 (PDT), Aaron Brady
>  declaimed the following in
> gmane.comp.python.general:
>
> > P.S.  I always feel like my posts should start like, "A mapping object
> > am writing I."  Not too many verbs you can do that with in English.
> > What if I did it?
>
>         You turn into a short fuzzy large eared critter in a robe with a
> light-saber?

A fuzzy large eared critter turn I into.

"Ending sentences in prepositions is nonsense up with which I will not
put." -Churchill

I'm mimicking the dict type with an SQL table (something to which I
alluded some time ago).  I'm going to 'select' on the hash code, then
(*THEN*) revive the keys and compare on equality.  The values don't
need to be revived.

P.S.  def revive!
-- 
http://mail.python.org/mailman/listinfo/python-list

hash and __eq__

[Aaron Brady]

I am writing a mapping object, and I want to ask about the details of
__hash__ and __eq__.  IIUC if I understand correctly, the Python
dict's keys' hash codes are looked up first in O( 1 ), then all the
matching hash entries are compared on equality in O( n ).  That is,
the hash code just really narrows down the search.  Am I correct?

P.S.  I always feel like my posts should start like, "A mapping object
am writing I."  Not too many verbs you can do that with in English.
What if I did it?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: try except inside exec

[Aaron Brady]

On May 29, 9:55 am, Michele Petrazzo
 wrote:
> Aaron Brady wrote:
> > STR = """
> > class Globals:
> >    err = 0
> > def a_funct():
> >    try:
> >       1/0
> >    except ZeroDivisionError:
> >       import traceback
> >       Globals.err = traceback.format_exc()
> > """
> > exec STR in env
> > try:
> >   env["a_funct"]()
> > except ZeroDivisionError:
> >   import traceback
> >   err = traceback.format_exc()
>
> > Then the formatted exception will be available in 'err'.  Do you want
> > the exception-handling to be part of the function you are entering
> > into 'env'?
>
> My goal is to execute a function received from a third-part, so I cannot
> modify as you made in your first piece of code.
> I want a "clean" exception with the real line code/tb so I can show a
> "real" error message. This means that the try/execpt has to include the
> passed function and catch every possible exception.
>
> Do you have any ideas on how can I figure this out?
>
> Thanks,
> Michele

Exceptions are only caught when raised in a 'try' statement.  If you
don't raise the exception in a try statement, it won't be caught.

The function you posted will raise an exception.  If you are making
the call yourself, that is, if only the definition is foreign, then
simply catch exceptions when so.

Otherwise, my only idea so far is to use the 'ast' module to inject an
exception handler into the third-party function.  That is quite
advanced and may be error-prone, though I think it would look
something like this:

for every 'def' statement in STR:
   def_statement.body= try_statement + def_statement.body

I expect it would be more reliable than hand-parsing the text and
concatenating a string exception statement.

Any interest in this, or do you make the call yourself?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: try except inside exec

[Aaron Brady]

On May 29, 8:21 am, Michele Petrazzo
 wrote:
> Hi all,
> I want to execute a python code inside a string and so I use the exec
> statement. The strange thing is that the try/except couple don't catch
> the exception and so it return to the main code.
> Is there a solution to convert or make this code work?
>
> Thanks,
> Michele
>
> My code:
>
> STR = """
> err = 0
> try:
>    def a_funct():
>      1/0
> except:
>    import traceback
>    err = traceback.format_exc()
> """
>
> env = {}
> exec STR in env
> env["a_funct"]()
> print env["err"]
>
> My error:
>    File "tmp/test_exec.py", line 14, in 
>      env["a_funct"]()
>    File "", line 5, in a_funct
> ZeroDivisionError: integer division or modulo by zero

The code you posted defines a function, and if the definition of it
raises an exception, it will enter your except clause.  By the way,
you generally should avoid 'bare' except clauses, and specify the type
of error you want to catch, in this case, 'ZeroDivisionError', as you
see (just good practice stuff).

It's not clear from your attempt what your exact goal was.  You
probably mean:

STR = """
class Globals:
   err = 0
def a_funct():
   try:
  1/0
   except ZeroDivisionError:
  import traceback
  Globals.err = traceback.format_exc()
"""

Note that the formatted exception will be available in 'env
["Globals"].err'.  But you could also mean:

exec STR in env
try:
  env["a_funct"]()
except ZeroDivisionError:
  import traceback
  err = traceback.format_exc()

Then the formatted exception will be available in 'err'.  Do you want
the exception-handling to be part of the function you are entering
into 'env'?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Adding a Par construct to Python?

[Aaron Brady]

On May 27, 11:07 pm, Steven D'Aprano  wrote:
> On Wed, 27 May 2009 12:58:02 +, Albert van der Horst wrote:
>
> >>And how is reduce() supposed to know whether or not some arbitrary
> >>function is commutative?
>
> > Why would it or need it? A Python that understands the ``par'' keyword
> > is supposed to know it can play some tricks with optimizing reduce() if
> > the specific function is commutative.
>
> Fine. Move the smarts out of reduce() into the compiler. How is the
> compiler supposed to know if an arbitrary function is commutative?

Unit testing.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Adding a Par construct to Python?

[Aaron Brady]

On May 19, 11:20 pm, Paul Rubin  wrote:
> Steven D'Aprano  writes:
> > (4) the caller is responsible for making sure he never shares data while
> > looping over it.
>
> > I don't think I've missed any possibilities. You have to pick one of
> > those four.
>
> I wonder if the compiler can check that a given function doesn't
> change any data.  Then:
>
> @pure
> def f(x):
>    return x*sqrt(x) + 3      # does not mutate any data
>
> @pure
> def g(x): ...                # likewise
>
> s = parallel_dot_product(parallel_map(f, vec), parallel_map(g,vec))

You can do the basics of this using the 'ast' module.  Just check that
no nodes in the ast tree are Assign nodes, including augmented
assign.  Then 'f' is defined as:

f= pure( '''
   return x*sqrt(x) + 3  # does not mutate any data
''' )

Untested.  However, you do need to check that the subchildren don't
make mutations.  This adds hooks to the AST, since names can change at
run-time.  Also, it's only defined for functions that call functions
that are all pure Python and defined using 'pure' as well, without
inspecting the bytecode.  At this point one might look into
metaprogramming and script autogeneration.

The 'pure' wrapper doesn't fundamentally change the semantics of the
function; it is more like a pre- and post-condition check.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Adding a Par construct to Python?

[Aaron Brady]

On May 17, 7:05 am, jer...@martinfamily.freeserve.co.uk wrote:
> From a user point of view I think that adding a 'par' construct to
> Python for parallel loops would add a lot of power and simplicity,
> e.g.
>
> par i in list:
>     updatePartition(i)
>
> There would be no locking and it would be the programmer's
> responsibility to ensure that the loop was truly parallel and correct.
>
> The intention of this would be to speed up Python execution on multi-
> core platforms. Within a few years we will see 100+ core processors as
> standard and we need to be ready for that.
>
> There could also be parallel versions of map, filter and reduce
> provided.
>
> BUT...none of this would be possible with the current implementation
> of Python with its Global Interpreter Lock, which effectively rules
> out true parallel processing.
>
> See:http://jessenoller.com/2009/02/01/python-threads-and-the-global-inter...
>
> What do others think?
>
> Jeremy Martin

Hi, joining late.

My first guess is that you haven't considered the existing syntactic
variants.

for i in list:
fireandforget( updatePartition )( i )

for i in list:
with fireandforget( ) as x:
x.call( updatePartition )( i )

@fireandforget
def fun( i ):
updatePartition( i )
for i in list:
fun( i )

Are you suggesting only syntactic sugar, or a true semantic change to
make parallel programming easier?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why there is a parameter named "self" for classmethod function?

[Aaron Brady]

On May 8, 7:52 pm, Steven D'Aprano  wrote:
> On Thu, 07 May 2009 04:27:15 -0700, Aaron Brady wrote:
> >> Can be, but if there's reason enough to keep it with a class, there's
> >> no reason not to.
>
> > That's a bit of hyperbole; the usual reasons such as code bloat,
> > namespace bloat, maintainability etc. all weigh against it.  It's just
> > that the benefits weigh heavier.
>
> I'm not sure I understand. Whether the function is in the class or
> extracted out into the module, it is exactly the same function. (Apart
> from an extra level of indentation, and either an unused "self" parameter
> or a @staticmethod line.) Same amount of code, same difficulty of
> maintainability. Namespace bloat is also the same, the only difference
> being which namespace is bloated: the class or the method.
>
> --
> Steven

My post was a little harsh.  I meant to say that I think I got your
idea, but the form it came in was a tautology.  Its form was X if not
not X.  Tautologies come across as particularly low bandwidth in
writing, though they do have some content in context.  I observe
people say them a lot too: 'it is what it is', 'if I want to I want
to', 'two bucks is two bucks', etc.  Some of the context comes from
the partial conventional asymmetry of sentences in English between
subject and predicate (nominative and accusative), though there is
none in logic.  Not to mention, 'hyperbole' isn't the word for what I
was thinking either... ahem.

The best place for such a function between those two choices can
depend on many things, some of which are matters of taste.  If the
definition occurs in the class, it may be more convenient to read or
edit.  That difference is pretty negligible.  There is semantic
difference to inherited classes: whether you have to import the class
or the class /and/ the function.  Then when it is called, the form
will be either 'class.methodA' or just 'methodA', which makes a
difference in readability (obviousness) elsewhere that it is called.

> being which namespace is bloated: the class or the method.

I think you meant 'class or module'.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why there is a parameter named "self" for classmethod function?

[Aaron Brady]

On May 7, 1:29 am, Steven D'Aprano
 wrote:
> On Thu, 07 May 2009 00:39:28 -0400, Terry Reedy wrote:
> > Functions that refer to neither the class nor an instance thereof can
> > usually be moved outside the class altogether.  Python is not Java.  I
> > believe staticmethod() was mainly added because it is needed for
> > .__new__(), at least in some appearances.
>
> Can be, but if there's reason enough to keep it with a class, there's no
> reason not to. Sometimes I have a class with a few methods that share
> common code, but that code happens to not include self (or cls). Since
> the common code is only of interest to that class, even though it doesn't
> need the instance, I factor the common code out into a method.
>
> This would be a good candidate for a staticmethod, only due to laziness I
> don't usually bother :)
>
> --
> Steven

It's also useful if you want to access the function using the syntax
of attributes, as in 'self.amethod' where 'amethod' is a static
method.  If I understand correctly, no bound method is created if the
function is static, and as such, the decorator can increase
performance.

> Can be, but if there's reason enough to keep it with a class, there's no
> reason not to.

That's a bit of hyperbole; the usual reasons such as code bloat,
namespace bloat, maintainability etc. all weigh against it.  It's just
that the benefits weigh heavier.

There is something that is 'correct' about it IMO, that is, provides
syntax to what is valid semantics in an OO context.  That may just be
another way of stating the thesis though.
--
http://mail.python.org/mailman/listinfo/python-list

Re: object query assigned variable name?

[Aaron Brady]

On May 6, 12:56 am, John O'Hagan  wrote:
> On Tue, 5 May 2009, Sion Arrowsmith wrote:
> > John O'Hagan   wrote:
> > >I can see that it's tantalizing, though, because _somebody_ must know
> > > about the assignment; after all, we just executed it!
>
> > Except we haven't, if we're talking about reporting from the
>
> > object's __init__:
> > >>> class Brian:
>
> > ...     def __init__(self):
> > ...         print "I'm Brian!"
> > ...
>
> > >>> l = []
> > >>> l[1] = Brian()
>
> > I'm Brian!
> > Traceback (most recent call last):
> >   File "", line 1, in 
> > IndexError: list assignment index out of range
>
> > (Yeah, I know that's a setitem call not an assignment. Point stands.
> > It also demonstrates why the whole idea of "what name is a newly-
> > created object assigned to" is broken.)
>
> I guess what I meant was that if I type:
>
> brian = Brian()
>
> in the python shell and then hit return, it seems to me that _somewhere_ (in
> the interpreter? I have no idea how it's done) it must be written that the
> new Brian object will later be assigned the name "brian", even as the process
> of creating the instance begins. As you've just demonstrated, the actual
> assignment occurs afterwards.
>
> But even if this is true I'm guessing it'd be black magic to get to it.
>
> Regards,
>
> John

In principle, you can't get all the names an object is bound to, since
it might be a temporary value or member of a container at some time.
If it was in a dictionary, would you want its key?  Or sequence and
index?

Furthermore, we don't have a data structure for two-way lookup.
Sometimes you would want to lookup name by object, sometimes object by
name.

Lastly, it would make code significantly less readable .
--
http://mail.python.org/mailman/listinfo/python-list

Re: Self function

[Aaron Brady]

On May 6, 6:49 am, Lie Ryan  wrote:
> Luis Zarrabeitia wrote:
> > Btw, is there any way to inject a name into a function's namespace? 
> > Following
> > the python tradition, maybe we should try to create a more general solution!
>
> How about a mechanism to pass arguments that are optional to accept?
>
> So (in the general case) the caller can call a function with certain
> arguments, but unless the function explicitly accept it, the argument
> will be discarded.
>
> With this, the injecting a local would simply be a case of adding an "I
> accept the argument" statement.
snip

You could write a decorator to do that with the inspect module, or you
could just accept a keyword dict for extra as-desired args.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Self function

[Aaron Brady]

On May 6, 2:23 am, Arnaud Delobelle  wrote:
> On May 5, 10:20 pm, Aaron Brady  wrote:
>
> > def auto( f ):
> >     def _inner( *ar, **kw ):
> >         return f( _inner, *ar, **kw )
> >     return _inner
>
> Quoting myself near the start of this thread:
>
> Here's an idea:
>
> >>> def bindfunc(f):
>
> ...     def boundf(*args, **kwargs):
> ...         return f(boundf, *args, **kwargs)
> ...     return boundf
>
> --
> Arnaud

Yeah, but that was two days ago.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Self function

[Aaron Brady]

On May 5, 3:54 pm, bearophileh...@lycos.com wrote:
> Aaron Brady:
>
> > >>> def auto( f ):
>
> > ...     def _inner( *ar, **kw ):
> > ...             return f( g, *ar, **kw )
> > ...     g= _inner
> > ...     return g
>
> Looks nice, I'll try to the following variant to see if it's usable:
>
> def thisfunc(fun):
>     """Decorator to inject a default name of a
>     function inside a function"""
>     def _inner(*args, **kwds):
>         return fun(g, *args, **kwds)
>     g = _inner
>     g.__name__ = fun.__name__
>     g.__dict__.update(fun.__dict__)
>     g.__doc__ = fun.__doc__
>     g.__module__ = fun.__module__
>     return g
>
> @thisfunc
> def SOMEVERYUGLYNAME(this, n):
>     return 1 if n <= 1 else n * this(n - 1)
>
> assert SOMEVERYUGLYNAME(6) == 2*3*4*5*6
>
> Bye,
> bearophile

/Actually/, the 'g' variable was spurious.  FWIW.

def auto( f ):
def _inner( *ar, **kw ):
return f( _inner, *ar, **kw )
return _inner

--
http://mail.python.org/mailman/listinfo/python-list

Re: call function of class instance with no assigned name?

[Aaron Brady]

On May 5, 2:17 pm, George Oliver  wrote:
> On May 5, 11:59 am, Dave Angel  wrote:
>
> > 1) forget about getattr() unless you have hundreds of methods in your
> > map.  The real question is why you need two maps. What good is the
> > "command string" doing you?   Why not just map the keyvalues directly
> > into function objects?
>
> Thanks for the reply Dave. I understand your example and it's what I
> originally used. Here is more detail on what I'm doing now, I hope
> this will explain my question better.
>
> In the game I'm writing the player, monsters, items and so on are
> instances of class Thing, like:
>
> class Thing(object):
>     def __init__(self, x, y, name):
>         self.x, self.y = x, y
>         self.name = name
>         self.brain = None
>
> Some Things have an instance of class Brain attached. The Brain
> instance has a list of handlers, like:
>
> class Brain(object):
>     def __init__(self):
>         self.handlers = []
>
> A handler class defines some functionality for the Brain. Each Brain
> has an update method like this:
>
> def update(self, arguments):
>     for handler in self.handlers:
>         handler.update(arguments)
>
> So on each pass through the main game loop, it calls the update method
> of all the brains in the game, which run their handler update methods
> to change the state of the game.
>
> A handler would be something like a key input handler. The key input
> handler is defined separately from the command handler in the case I
> want to use a different method of input, for example a mouse or
> joystick.
>
> In the dictionary of key inputs I could map each input directly to a
> function. However there is no instance name I can call the function
> on, as I create a thing, add a brain, and add handlers to the brain
> like this:
>
> player = Thing(26, 16, 'player')
> player.brain = Brain()
> player.brain.add_handlers(
>                             commandHandler(player.brain, player),
>                             keyboardHandler(player.brain),
>                             fovHandler(player.brain))
>
> So what I'm wondering is how to reference the instance in each brain's
> list of handlers when I want to map something like a key input to a
> command, or what a better way might be to structure the code.

I'm not entirely sure I follow, but you may want to try keeping a set
(or list) of handlers that you pass in to the 'add_handlers' function,
and trying the 'getattr( handlerX, keyY )' on each of them, possible
only until you have a success.  You could also cache these results to
avoid performing the same search subsequently.

Otherwise, query each of the passed-in handlers when they are passed
in for a list of events they can handle; this may be by a brute
'getattr( x ) for x in ALL_EVENTS', or by maintaining a list of
handlers on a per-object basis, and merely combining the dictionaries;
then pick one or all of the handlers you've accumulated for an event
when the event occurs.



If you're really trying to be clever, you might use a metaclass or
class decorator to accumulate what events are handled in a class; or
just try the '__dict__' member of the /class/.

Will any of your handlers be handling overlapping or duplicate
events?  Do they all get a crack at it, or do you follow a pre-
deterimend order of precedence, or follow the order in which they were
passed in?

I'd probably try to discourage you from circularly linking the object
and handlers, not only for practical reasons.

There is also the possibility you could use dynamic mix-ins to create
a specialized once-in-a-lifetime brain class:

def create_brain( handlers ):
class new_brain( Brain ):
pass
for x in handlers:
setattr( new_brain, x.__name__, x )
return new_brain

brain= create_brain( handlers )( creation_args )

Forgive, just Thing-king aloud.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Self function

[Aaron Brady]

On May 5, 2:50 pm, Steve Howell  wrote:
> On May 4, 11:08 pm, Steven D'Aprano
>
>  wrote:
>
> > I propose a small piece of sugar. When a function is entered, Python
> > creates an ordinary local name in the function's local namespace, and
> > binds the function itself to that name. Two possibilities for the name
> > are `this` or `__this__`, analogous to `self` in methods and `__name__`
> > in modules.
>
> > If there is any support for this, I propose to send the following (long)
> > post to python-ideas. Feedback, corrections and suggestions welcome.
>
> I totally support this proposal.   I've definitely wasted time in the
> past trying to invent my own workarounds for the use cases you
> describe.
>
> Obviously, there will be some bikeshed debates on __this__ vs.
> __name__ vs. __func__, etc.  I don't have an opinion there, just want
> *something* handy for introspection, DRYness, etc.
>
> A more interesting question is whether __this__ would just always be
> there (that's my vote), or if you should have to apply a special
> decorator to get it (which I oppose, but I can see some merits).

Here is how to get the function into the function as an argument, and
still permit recursive calls onto it.

>>> def auto( f ):
... def _inner( *ar, **kw ):
... return f( g, *ar, **kw )
... g= _inner
... return g
...
>>> @auto
... def f( self, n ):
... print( self, n )
... return 1 if n== 1 else n* self( n- 1 )
...
>>> f(7)
 7
 6
 5
 4
 3
 2
 1
5040

The function that is passed into the function is the decorated
function, not the original.  The decorator function is also returned
from the decorator calls, so both the external caller and the internal
caller are calling the same function, '_inner' in the sample, which
then back-calls the original.

Please stay tuned for the post-graduate lecture.  There will be time
for questions at the cookies-and-creme seminar afterward.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing Pool and functions with many arguments

[Aaron Brady]

On Apr 29, 1:01 pm, "psaff...@googlemail.com"
 wrote:
> I'm trying to get to grips with the multiprocessing module, having
> only used ParallelPython before.
>
> based on this example:
>
> http://docs.python.org/library/multiprocessing.html#using-a-pool-of-w...
>
> what happens if I want my "f" to take more than one argument? I want
> to have a list of tuples of arguments and have these correspond the
> arguments in f, but it keeps complaining that I only have one argument
> (the tuple). Do I have to pass in a tuple and break it up inside f? I
> can't use multiple input lists, as I would with regular map.
>
> Thanks,
>
> Peter

The basic constructor should allow it.

http://docs.python.org/library/multiprocessing.html#the-process-class

p = Process(target=f, args=('bob',))

I think you are imagining:

result = pool.apply_async(f, [ *(1,2,3), *(4,5,6) ] )

But I don't know of a way.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why bool( object )?

[Aaron Brady]

On May 1, 4:30 am, Steven D'Aprano  wrote:
> On Fri, 01 May 2009 16:30:19 +1200, Lawrence D'Oliveiro wrote:
> > I have never written anything so unbelievable in my life. And I hope I
> > never will.
>
> I didn't say you did. If anyone thought I was quoting Lawrence's code,
> I'd be surprised. It was not my intention to put words into your mouth.
>
> But seeing as you have replied, perhaps you could tell us something.
> Given so much you despise using non-bools in truth contexts, how would
> you re-write my example to avoid "a or b or c"?
>
> for x in a or b or c:
>     do_something_with(x)

I think Hendrik's is the closest so far, but still doesn't iterate
over x:

for t in [a,b,c]:
if t:
for x in t:
do_something_with(x)
break

This is still not right, since Steven's code will raise an exception
if 'a' and 'b' test False, and 'c' is non-iterable.

>>> for x in 0 or 0 or 0:
... print( x )
...
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'int' object is not iterable
>>> for x in 0 or 0 or []:
... print( x )
...
>>> for x in 0 or 0 or 1:
... print( x )
...
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'int' object is not iterable
>>> for x in ['abc'] or 0 or 0:
... print( x )
...
abc

To mimic it exactly, you'd have to actually convert the first one,
execute 'or' on the other two, since they may have side-effects or
return other values than themselves, and try iterating over the last
one.  I think you are looking at an 'ireduce' function, which doesn't
exist in 'itertools' yet.

I don't think it would be very common to write Steven's construction
for arbitrary values of 'a', 'b', and 'c'.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing.Queue - I want to end.

[Aaron Brady]

On Apr 30, 3:49 pm, Luis Zarrabeitia  wrote:
> Hi. I'm building a script that closely follows a producer-consumer model. In
> this case, the producer is disk-bound and the consumer is cpu-bound, so I'm
> using the multiprocessing module (python2.5 with the multiprocessing backport
> from google.code) to speed up the processing (two consumers, one per core,
> and one producer). The consumers are two multiprocessing.Process instances,
> the producer is the main script, and the data is sent using a
> multiprocessing.Queue instance (with bounded capacity).
>
> The problem: when there is no more data to process, how can I signal the
> consumers to consume until the queue is empty and then stop consuming? I need
> them to do some clean-up work after they finish (and then I need the main
> script to summarize the results)
snip
>     for data in iter(queue.get, None):
>         process_data(data, outfile) # stores the result in the outfile
snip
>     queue.put(None); queue.put(None)
snip
> As you can see, I'm sending one 'None' per consumer, and hoping that no
> consumer will read more than one None. While this particular implementation
> ensures that, it is very fragile. Is there any way to signal the consumers?
> (or better yet, the queue itself, as it is shared by all consumers?)
> Should "close" work for this? (raise the exception when the queue is
> exhausted, not when it is closed by the producer).

You may have to write the consumer loop by hand, rather than using
'for'.  In the same-process case, you can do this.

producer:
sentinel= object( )

consumer:
while True:
  item= queue.get( )
  if item is sentinel:
break
  etc.

Then, each consumer is guaranteed to consume no more than one
sentinel, and thus producing one sentinel per consumer will halt them
all.

However, with multiple processes, the comparison to 'sentinel' will
fail, since each subprocess gets a copy, not the original, of the
sentinel.  A sample program which sent the same object multiple times
produced this output:




Theoretically, you could send a shared object, which would satisfy the
identity test in the subprocess.  That failed with this exception:

  File "c:\programs\python30\lib\multiprocessing\queues.py", line 51,
in __getstate__
assert_spawning(self)
...
RuntimeError: Queue objects should only be shared between processes th
rough inheritance

As a result, your options are more complicated.  I think the best
option is to send a tuple with the data.  Instead of sending 'item',
send '( True, item )'.  Then when the producer is finished, send
'( False,  )'.  The consumer will break when it encounters a
'False' first value.

An alternative is to spawn a watchman thread in each subprocess, which
merely blocks for a shared Event object, then sets a per-process
variable, then adds a dummy object to the queue.  The dummy is
guaranteed to be added after the last of the data.  Each process is
guaranteed to consume no more than one dummy, so they will all wake
up.

If you don't like those, you could just use a time-out, which checks
the contents of a shared variable, like a one-element array, then
checks the queue to be empty.  If the shared variable is True, and the
queue is empty, there is no more data.

I'm curious how these work and what you decide.
--
http://mail.python.org/mailman/listinfo/python-list

Re: subprocess & shared environments

[Aaron Brady]

On May 1, 12:09 am, Robert Dailey  wrote:
> I'm currently calling subprocess.call() on a batch file (in Windows)
> that sets a few environment variables that are needed by further
> processes started via subprocess.call(). How can I persist the
> environment modifications by the first call() function? I've done my
> own searching on this and I came up with nothing.
>
> Help is appreciated.

I don't know if this will work, but you can try spawning a batch file
that calls the first program to set the variables, then executes the
second program.  You might need to create that batch file by hand, as
well as the command line for the second process.

You can also try setting the env. variables in the parent process, and
restoring them after.
--
http://mail.python.org/mailman/listinfo/python-list

Re: suggestion on a complicated inter-process communication

[Aaron Brady]

Um, that's the limit of what I'm familiar with, I'm afraid.  I'd have
to experiment.

On Apr 28, 10:44 am, Way  wrote:
> Thanks a lot for the reply. I am not familiar with multi-process in
> Python. I am now using something like:
snip
> However, in this case, Process5's stdout cannot be passed to
> MainProcess for Process1, since it has not finished (waiting Process.
> 1/3 finish).
>
> I am now using Fifos (an external file) to inter-communicate Process1
> and 5. But at first run, it compliants "not file found".
>
> Is there any possible solution to make it work? Thank you very much!
snip
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why bool( object )?

[Aaron Brady]

On Apr 28, 9:54 pm, Steven D'Aprano
 wrote:
> On Tue, 28 Apr 2009 11:59:18 -0700, Aaron Brady wrote:
> >> To steal an idiom from Laura: Python has a float-shaped Nothing 0.0, a
> >> list-shaped Nothing [], a dict-shaped Nothing {}, an int-shaped Nothing
> >> 0, a singleton Nothing None, and so forth.
>
> > The sound of that metaphor is rather pleasing ('sweet nothings'), but
> > I'm not so sure that metaphors belong in computer science and
> > programming.
>
> Programming models the world, it isn't the world. Every model is a
> metaphor. You can't get away from metaphors in programming, you can only
> hide them and pretend they aren't there.

I was not aware that the intended use for 'True' was 'Something'.  I
thought it merely signified membership in a predicate or relation.
'GreaterThan( 3, 2 )', so '3> 2== True'.  You might argue that numbers
are metaphors for the world, since they are external to causality,
have no temperature, etc., and likewise with logical predicates.  But
just because True is one metaphor, doesn't mean it is every metaphor.
Is casting to a boolean a metaphor for a metaphor?

Mathematically, 'bool' injects mathematical objects into the set
{ True, False }.  I would say it's a stretch to say that the empty
list maps to the True value, and non-empty lists map to the False
value.  Mathematically, there's no reason to choose that mapping as
the natural meaning and definition over any other, say 'list( x ) iff
len( list( x ) )% 2'.  (Some shells, I understand, suppose that there
are multiple ways to fail, while others suppose that there are
multiple ways to succeed.)  In programming practice, there is a
frequent division between empty lists and non-empty lists, so the
author chose to map one case to one value, and the other case to the
other.

> > Nothing can't have many shapes.
>
> In computing, you can't have a literal "nothing", because EVERYTHING
> needs to be stored as a pattern of bits.

Correct... until memory gates can self-destruct.  

> In practice, for most high-level
> languages, those bits are interpreted as a type as well as a value, so
> the same bit pattern can mean different things according to what type it
> is expected to be. Alternatively, the same information can be represented
> in different bit patterns depending on how you interpret those bit
> patterns.
>
> So in the context of computer programming, of course you can have nothing
> with many "shapes": you have the number 0 represented as a single byte, a
> 16-bit integer, a 32-bit integer, a long-int or BigNum, a pointer with
> address 0, fixed point decimals of various sizes, Binary Coded Decimal, a
> rational, a single float 0.0, a double float 0.0, a string of "tally
> marks" without any tally, and so on, all of them in big-endian and little-
> endian formats.
>
> > Having no onions is the same as having no carrots.
>
> "If you have one onion and I take it away, how many onions do you have
> left?"
>
> If you answered "no carrots", I'd consider that a bizarre and unhelpful
> answer, and wonder what you possibly thought carrots had to do with my
> question.
>
> "Nothing" is not an absolute. "There is no wind" is not the same thing as
> "there is no money left in my bank account".
>
> > If the different shapes of nothing don't
> > compare equal to each other, which they don't, then they aren't all the
> > same thing in different shapes.
>
> Who said they were the same thing?

The fact that bool( [] ) == bool( {} ).  Merely that 'f( x )== f( y )'
does not imply that 'x== y'.  I am not arguing that '[]== {}', and I'm
pretty sure you aren't either.  However, x and y do form an
equivalence class under 'f', IIRC.

If you ask of (blank), 'Is there anything?', which is what you state
the 'bool' function means to do, there are only two answers: 'Yes,
there is something', and 'No, there is nothing'.

> > Furthermore, it is awfully presumptuous to give objects a default
> > 'nothing-ness' of 'something'.  If anything, they should be nothing by
> > default.  
>
> That's a stupid idea.
[argument] snip

Fine, withdrawn.  They shouldn't have one.

> > Conversion to other primitives is very picky; why should
> > Boolean be so tolerant?
>
> Because it is useful and meaningful to ask whether an arbitrary object
> represents something or nothing, but it's not useful or meaningful to ask
> what integer is equivalent to an arbitrary object.

It's more 

Re: Why bool( object )?

[Aaron Brady]

On Apr 28, 2:39 am, Steven D'Aprano
 wrote:
> On Mon, 27 Apr 2009 23:11:11 -0700, Aaron Brady wrote:
> > What is the rationale for considering all instances true of a user-
> > defined type?  Is it strictly a practical stipulation, or is there
> > something conceptually true about objects?
>
> Seven years ago, in an attempt to convince Guido *not* to include
> booleans in Python, Laura Creighton wrote a long, detailed post
> explaining her opposition.
>
> At the heart of her argument is the observation that Python didn't need
> booleans, at least not the ints-in-fancy-hats booleans that we've ended
> up with, because Python already made a far more useful and fundamental
> distinction: between Something and Nothing.
>
> http://groups.google.com/group/comp.lang.python/msg/2de5e1c8384c0360?...
>
> All objects are either Something or Nothing. The instances of some
> classes are always Something, just as the instances of some classes are
> always Nothing. By default, instances are Something, unless __nonzero__
> returns False, or __len__ returns 0, then they are Nothing.
>
> In a boolean (or truth) context, Something and Nothing behave like True
> and False in languages with real booleans:
>
> if obj:
>     print "I am Something"
> else:
>     print "I am Nothing"
>
> To steal an idiom from Laura: Python has a float-shaped Nothing 0.0, a
> list-shaped Nothing [], a dict-shaped Nothing {}, an int-shaped Nothing
> 0, a singleton Nothing None, and so forth.

The sound of that metaphor is rather pleasing ('sweet nothings'), but
I'm not so sure that metaphors belong in computer science and
programming.  Nothing can't have many shapes.  Having no onions is the
same as having no carrots.  If the different shapes of nothing don't
compare equal to each other, which they don't, then they aren't all
the same thing in different shapes.

Furthermore, it is awfully presumptuous to give objects a default
'nothing-ness' of 'something'.  If anything, they should be nothing by
default.  Conversion to other primitives is very picky; why should
Boolean be so tolerant?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why bool( object )?

[Aaron Brady]

On Apr 28, 1:35 am, Lie Ryan  wrote:
> Aaron Brady wrote:
> > What is the rationale for considering all instances true of a user-
> > defined type?  
>
> User-defined objects (or type) can override .__len__() [usually
> container types] or .__nonzero__() to make bool() returns False.
>
> > Is it strictly a practical stipulation, or is there
> > something conceptually true about objects?
>
> Objects are true unless they define themself as false. The practical
> implication is we can do this:
>
> def foo(args = None):
>      if args:
>          ...
>
> In python all objects are true except: None, False, 0/0L/0.0/0j, empty
> sequence or container, and on objects that defines .__len__() or
> .__nonzero__() that returns 0 or False.
>
>
>
> > '''
> > object.__bool__(self)
> > If a class defines neither __len__() nor __bool__(), all its instances
> > are considered true.
> > '''
>
> > This makes it so all objects except False, None, 0, and empty
> > containers are true by default.  I am not convinced that 'if  > generic object>' should have any meaning; it should probably throw an
> > exception.  Is it part of Python's look and feel or its mentality?  Is
> > it part of the Zen?  Certainly other ideal types can't be cast from
> > generic objects, so why booleans?  Is it an ineffable component of the
> > author's vision for the language?  I think that giving arbitrary
> > syntactic constructs meaning is just space-filler.  It's worse than
> > syntactic sugar, it's semantic sugar.  Why not assign meanings willy-
> > nilly to other random juxtapositions of tokens?
>
> It's part of the design decision. In almost all cases (in any language),
> a so-called "Design Decision" is rather random and prone to subjective
> judgment, just as the decision to make bool(int) returns False only on
> 0, -1, or for all negative values; whether to make bool(100) and
> exception or True; or round() rounds down or up or even-odd; or the use
> of brackets vs. indentation; or whether to treat empty list as True or
> False.

Whether it's a good feature and whether it's a popular feature are two
different things.  (It's not always clear that the former is even real
anyway.)

I'm actually not sure that my question was /entirely/ scientifically
motivated.  Perhaps I'm expressing admiration (or maybe envy).  Maybe
I want to design a popular language too.

If I was really being scientific, I would have been wondering, what is
the balance of pros and cons of the decision?  pros - cons = ?  Is
that an objective expression?  Are there only measurable quantities in
both operands?  If it's a close call, I might not agree, or might not
have followed the same experimental method.

Trivially the answer is, 'the decision maker found sum( pros ) > sum
( cons )'.  Many arts and disciplines do guesswork to a degree in
evaluating their 'sum( pros )', so perhaps I should expect a degree of
variation in what the experts all say that quantity is.

On a more idealist's level, perhaps I am asking the outside world
what /my/ evaluation of 'sum( pros )' is.  What's yours?

--
http://mail.python.org/mailman/listinfo/python-list

Why bool( object )?

[Aaron Brady]

What is the rationale for considering all instances true of a user-
defined type?  Is it strictly a practical stipulation, or is there
something conceptually true about objects?

'''
object.__bool__(self)
If a class defines neither __len__() nor __bool__(), all its instances
are considered true.
'''

This makes it so all objects except False, None, 0, and empty
containers are true by default.  I am not convinced that 'if ' should have any meaning; it should probably throw an
exception.  Is it part of Python's look and feel or its mentality?  Is
it part of the Zen?  Certainly other ideal types can't be cast from
generic objects, so why booleans?  Is it an ineffable component of the
author's vision for the language?  I think that giving arbitrary
syntactic constructs meaning is just space-filler.  It's worse than
syntactic sugar, it's semantic sugar.  Why not assign meanings willy-
nilly to other random juxtapositions of tokens?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Light (general) Inter-Process Mutex/Wait/Notify Synchronization?

[Aaron Brady]

On Apr 28, 12:20 am, Gunter Henriksen 
wrote:
> > If you don't want to use a 3rd party module you could
> > use the multiprocessing module
>
> That is definitely good for when I have a tree of
> processes which are all Python applications.  I use
> it for that.  But I am looking for something where
> the Python application can interact conveniently
> with an arbitrary external application.  Using a
> socket/pipe and shared memory is convenient, but
> not feasible when system call overhead matters.

You could write your own '.pyd' or '.so' extension that merely brokers
the OS synchronization calls.  The 'ctypes' module may be enough.

You will need a way to get the owner's pid and synch. object handle to
the remote process of course.
--
http://mail.python.org/mailman/listinfo/python-list

Re: suggestion on a complicated inter-process communication

[Aaron Brady]

On Apr 27, 10:59 pm, Way  wrote:
> Hello friends,
>
> I have a little messy situation on IPC. Please if you can, give me
> some suggestion on how to implement. Thanks a lot!
>
> -> denotes create
>
> MainProcess -> Process1 -> Process3 (from os.system)
>                    |
>                     -> Process2 (from os.system) -> Process4 (from
> os.system) ->Process5
>
> I would like to make the communication between Process1 and Process5.
> Process1 needs Process5's output to provide argument to generate
> Process3, and in turn Process5 needs to wait Process3 finished.
>
> Thank you very much if you can give a hint.

The 'mmap' module can help with getting the data from 5 to 1.  It
requires creating a file.  If it's a small amount of data, any old
file will do.  You may need a socket in order to wait for Process3 to
join, or write a small '.pyd' or '.so' file that gives you access to
your system's synchronization object.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Lisp mentality vs. Python mentality

[Aaron Brady]

On Apr 26, 10:52 am, Scott David Daniels 
wrote:
> Travis wrote:
> > ... I've noticed that every one of you is wrong about programming.
> > Since I can't say it effectively, here's someone who can:
> >http://www.youtube.com/watch?v=XHosLhPEN3k
>
> > That's the answer.
>
> That is a wonderful link.  Thanks for sharing.
>
> --Scott David Daniels
> scott.dani...@acm.org

import mentality?  help( mentality ).
--
http://mail.python.org/mailman/listinfo/python-list

Re: relation class

[Aaron Brady]

On Apr 24, 1:18 am, Aaron Brady  wrote:
> On Apr 22, 11:34 pm, Aaron Brady  wrote:
>
> > On Apr 22, 11:52 am, Aaron Brady  wrote:
>
> > > On Apr 22, 12:09 am, Chris Rebert  wrote:
>
> > > > On Tue, Apr 21, 2009 at 5:51 PM, Aaron Brady  
> > > > wrote:
> > > > > Hi all,
>
> > > > > I think Python should have a relation class in the standard library.
> > > > > Fat chance.
>
> > > > Perhaps I'm not understanding "relation" correctly, but are you not
> > > > aware ofhttp://docs.python.org/library/sqlite3.html?
>
> > > > Cheers,
> > > > Chris
> > > > --
> > > > I have a blog:http://blog.rebertia.com
> > snip
> > > It only supports numbers and strings.
>
> > snip
>
> > My point is that in undirected relations, it's redundant to maintain
> > reciprocal membership.
> snip
> > Here is another sample:
>
> > Tree= Relation( ( "parent", "child", "direction" ) )
> > nodes= [ object( ) for _ in range( 10 ) ]
> > Tree( ( nodes[ 0 ], nodes[ 1 ], "left" ) )
> > Tree( ( nodes[ 0 ], nodes[ 2 ], "right" ) )
> snip
> > 'select' would need the context
> > of its caller, which isn't available.
>
> snip
>
> Or, pass 'locals()'.
snip
>
> > I think some kind of markers would have to replace any identifiers it
> > would use:
>
> > recordset= Tree.select( [ "child" ],
> >     "parent is %var and direction=='left'", nodes[0] )
>
> snip
snip
> class TreeNode:
>     relation= Tree
>     left= Relation.getter(
>         "child", "parent is ? and direction=='left'" )
snip
> It only allows one '?' in the
> definition, since getter functions only take one argument: self!

This isn't true.  The query could have multiple parameter arguments.
Since 'self' isn't defined when calling the property creation
function, the question mark will need a special flag, or an argument
will have to be a sentinel object.

left= Tree.getter(
"child", "parent is ?self and direction==?", "left" )

Or:

left= Tree.getter(
"child", "parent is ? and direction==?", self_mark, "left" )

On doing my homework, I find parameters can be named as well:

left= Tree.getter(
"child", "parent is :self and direction==:dir",
{ 'dir': "left" } )

'self' would be reserved.

snip
> You might want 'getter', 'getiter', and 'getone' methods, which return
> a set of, an iterator over, and just the field of an arbitrary one of
> the matching records respectively.
snip

> > > > I want to return an
> > > > iterator from the following function calls.

It isn't clear that a bare iterator is the proper tool.  Taking
'columns' to be fields, and 'rows' to be records, we have four cases
of the return from a select statement.

- One row, one column.  'getter' and 'setter' can take a single-object
value.  'deller' merely deletes the record.
- One row, multiple columns.  'getter' and 'setter' return and accept
dictionaries which specify the values of the columns in the given
record.  'deller' deletes the record.  It's possible to create nested
properties which correspond to the unspecified fields, but you may
lose identifier fidelity.  Is a dictionary prohibitively inconvenient?
- Multiple rows, one column.  The return from 'SELECT' should have
iterator semantics, as well as set operations.  'getter' returns the
object, but no 'setter' or 'deller' descriptors are meaningful.  The
'add' method on the iterator/set should insert a new record that
fulfills the rest of the statement along with the value added; there
is one degree of freedom in the query.  'discard', 'remove', 'pop',
etc., should all do analogous things.  The methods may have to be
implemented from scratch, due to the fact that they are short-hand for
operations on the relation with larger tuples, as well as
corresponding entries in the hashes and trees.
- Multiple rows, multiple columns.  The return from 'SELECT' should be
an iterator/set.  Arguments to the set operations should be
dictionaries as above.

The 'children' descriptor of the 'Tree' relation corresponds to the
last case:

class TreeNode:
children= Tree.getter(
"child", "parent is ?self" )

A call to 'list( nodeA.c

Re: relation class

[Aaron Brady]

On Apr 22, 11:34 pm, Aaron Brady  wrote:
> On Apr 22, 11:52 am, Aaron Brady  wrote:
>
> > On Apr 22, 12:09 am, Chris Rebert  wrote:
>
> > > On Tue, Apr 21, 2009 at 5:51 PM, Aaron Brady  wrote:
> > > > Hi all,
>
> > > > I think Python should have a relation class in the standard library.
> > > > Fat chance.
>
> > > Perhaps I'm not understanding "relation" correctly, but are you not
> > > aware ofhttp://docs.python.org/library/sqlite3.html?
>
> > > Cheers,
> > > Chris
> > > --
> > > I have a blog:http://blog.rebertia.com
> snip
> > It only supports numbers and strings.
>
> snip
>
> My point is that in undirected relations, it's redundant to maintain
> reciprocal membership.
snip
> Here is another sample:
>
> Tree= Relation( ( "parent", "child", "direction" ) )
> nodes= [ object( ) for _ in range( 10 ) ]
> Tree( ( nodes[ 0 ], nodes[ 1 ], "left" ) )
> Tree( ( nodes[ 0 ], nodes[ 2 ], "right" ) )
snip
> 'select' would need the context
> of its caller, which isn't available.
snip

Or, pass 'locals()'.  Actually, I discovered an interesting catch to
'locals()'.  When you don't use a variable that's in an outer scope in
a function, it doesn't appear in 'locals()'.  However, if you just use
it in a blank statement, it magically appears.

>>> def f( ):
... x= []
... def g( ):
... print( locals( ) )
... g( )
...
>>> f( )
{}
>>> def f( ):
... x= []
... def g( ):
... x # empty use of 'x'
... print( locals( ) )
... g( )
...
>>> f( ) # changes the contents of 'locals()'
{'x': []}

Here is what the docs have to say about *that*:

"Free variables are returned by locals() when it is called in a
function block."

Since 'x' doesn't appear in the function, 'g' in this case, it isn't
included in 'g's free variables.  'eval' actually therefore doesn't
return exactly the result of evaluating a string where it's used.

>>> def f( ):
... x= []
... def g( ):
... print( eval( 'x' ) )
... g( )
...
>>> f( )
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 5, in f
  File "", line 4, in g
  File "", line 1, in 
NameError: name 'x' is not defined

The 'eval' function docs do assert otherwise though:

"...the expression is executed in the environment where eval() is
called."

'eval' doesn't obviously keep its contract as shown.

> I think some kind of markers would have to replace any identifiers it
> would use:
>
> recordset= Tree.select( [ "child" ],
>     "parent is %var and direction=='left'", nodes[0] )
snip

The 'sqlite3' module just uses a question mark.  Despite the fact that
variables are separated from exactly where they are used, it has the
advantage, as S. D'Aprano pointed out nearby, of being able to use
queries independently of them, as well as of being legal Python.

If anyone is still reading, there is a further nifty-cool construct
that the 'Relation' class can offer.  Given the earlier definition of
'Tree' as an instance of 'Relation', a new class can combine property
descriptors with the select statement.

class TreeNode:
relation= Tree
left= Relation.getter(
"child", "parent is ? and direction=='left'" )

nodes= [ TreeNode( ) for _ in range( 10 ) ]
record= nodes[ 0 ].left

Getting the 'left' member of an instance of this class is just
syntactic sugar for a long-hand query.  It only returns one field of
an arbitrary element of the recordset.  It only allows one '?' in the
definition, since getter functions only take one argument: self!
Here's the definition of 'getter':

@staticmethod
def getter( field, where ):
def _getter( self ):
return getattr( next(
self.relation.select( '*', where, self ) ), field )
return property( fget= _getter )

It could be an attribute of the module, not specifically of the
Relation class... which might even be advisable.  You could even
combine the 'relation' member into the call to 'getter', so 'getter'
wouldn't have to retrieve it as 'self.relation'.  That has the
disadvantage of increasing repetition of the relation in the class
statement, but enables a class to participate in multiple relations.
Come to think of it, you could write it as 'Tree.getter', and
compromise.  Then 'getter' becomes:

def getter( self, field, where ):
def _getter( ob ):
return getattr( next(
self.select( '*', where, ob ) ), field )
return property( fget= _getter )

and 'TreeNode' becomes:

class TreeNode:
left= Tree.getter(
"child", "parent is ? and direction=='left'" )

You might want 'getter', 'getiter', and 'getone' methods, which return
a set of, an iterator over, and just the field of an arbitrary one of
the matching records respectively.

Proof-of-concept isn't even complete; but nonetheless, pretty cool,
eh?
--
http://mail.python.org/mailman/listinfo/python-list

Re: relation class

[Aaron Brady]

On Apr 22, 11:52 am, Aaron Brady  wrote:
> On Apr 22, 12:09 am, Chris Rebert  wrote:
>
> > On Tue, Apr 21, 2009 at 5:51 PM, Aaron Brady  wrote:
> > > Hi all,
>
> > > I think Python should have a relation class in the standard library.
> > > Fat chance.
>
> > Perhaps I'm not understanding "relation" correctly, but are you not
> > aware ofhttp://docs.python.org/library/sqlite3.html?
>
> > Cheers,
> > Chris
> > --
> > I have a blog:http://blog.rebertia.com
snip
> It only supports numbers and strings.
snip

My point is that in undirected relations, it's redundant to maintain
reciprocal membership.  Each participant is a member of the other.
'child.parent.child is child'.  It's not a difficult redundancy to
maintain, but it isn't true to the concept.

Here is another sample:

Tree= Relation( ( "parent", "child", "direction" ) )
nodes= [ object( ) for _ in range( 10 ) ]
Tree( ( nodes[ 0 ], nodes[ 1 ], "left" ) )
Tree( ( nodes[ 0 ], nodes[ 2 ], "right" ) )

Here is the query (*non-functional)*:

recordset= Tree.select( [ "child" ],
"parent is nodes[0] and direction=='left'" )

Here's an alternative*:

recordset= Tree.select( [ "child" ],
lambda record: record.parent is nodes[0] and
record.direction=='left' )

I want some more concise ways of retrieving the same data.  Those two
don't even work as is.*

recordset= Tree.select( operator.and_(
operator.is_( 'parent', nodes[0] ),
operator.eq( 'direction', 'left' ) ) )

This is evaluated too soon.  The comparisons need to be
'functools.partial' functions*.

recordset= Tree.select(
partial(
operator.and,
partial(
operator.is_,
'parent',
nodes[0]
),
partial(
operator.eq,
'direction',
'left'
)
)
)

This is close, but the partial functions just compare strings.  They
would have to be replaced with specialized functions that can
recognize designated strings as field names, and replace them with
actual fields.

Furthermore, it doesn't address the awkwardness of prefix notation,
which is required for function calls that aren't native operators.

I think the first is the closest.  'ast' makes it particularly
convenient, though maybe not trivial, to parse the selection.  Where
it sees an 'ast.Name' node, it can replace it with an actual field
name.  Here it is again*:

recordset= Tree.select( [ "child" ],
"parent is nodes[0] and direction=='left'" )

The only problem is passing variables in to the expression.  By the
time 'nodes[0]' would get evaluated, it would be too late to know what
it is, without dirty frame stackery.  'select' would need the context
of its caller, which isn't available.

I think some kind of markers would have to replace any identifiers it
would use:

recordset= Tree.select( [ "child" ],
"parent is %var and direction=='left'", nodes[0] )

It's counterintuitive, but it's better than entirely prefix notation.
This way the function gets all the actual objects it needs, and can
still take shortcuts in its search.

I don't really see the argument for only returning a subset of fields,
instead of the entire record where matched, although I may not have
experience enough to know the downside, such as in the case of large
joins.  It could be optional, defaulting to 'select *'.

recordset= Tree.select(
"parent is %var and direction=='left'", nodes[0] )

Since select returns an iterator, it's probably wise to include a
'selectone' function, like the 'Cursor.fetchone' function in
'sqllite3'.  Of course, one could just add a '.next()' call to the
front of the query.

record= next( Tree.select(
"parent is %var and direction=='left'", nodes[0] ) )

or

record= Tree.selectone(
"parent is %var and direction=='left'", nodes[0] )

Does anyone see a better way?  Would anyone have use for it?  Do you
agree that it is more conceptually true to data structures like
trees?  Is the optimization worth a specialized syntax?  Even if
selection used a linear-time search, would encapsulating relations be
an improvement over redundant and cyclic attributes?  Are the
equivalents in existing packages just as light-weight and functional?
That is, is it competitive?

Furthermore, is anyone developing something which uses a quoted
expression in a custom evaluation?  I'd be interested in seeing those
results too.  I understand that languages like LISP and Logix (
http://www.livelogix.com/logix/ ) are suited for it, but I want
something that is just a single class.  Is it outside the scope of
Python's capabilities to do?
--
http://mail.python.org/mailman/listinfo/python-list

Re: relation class

[Aaron Brady]

On Apr 22, 12:09 am, Chris Rebert  wrote:
> On Tue, Apr 21, 2009 at 5:51 PM, Aaron Brady  wrote:
> > Hi all,
>
> > I think Python should have a relation class in the standard library.
> > Fat chance.
>
> Perhaps I'm not understanding "relation" correctly, but are you not
> aware ofhttp://docs.python.org/library/sqlite3.html?
>
> Cheers,
> Chris
> --
> I have a blog:http://blog.rebertia.com

Yes, I am aware of it.  I should have mentioned.  It is much the same
as what I am picturing, especially with the special ':memory:' name
for creating in RAM.

It only supports numbers and strings, which makes it appropriate for
persistence but not in-memory objects.  It is also very advanced
compared to what I am picturing.

Its strategy for the syntax of the select statement is to use a
string, followed by the arguments for interpolation.  How could you
use it to map, say, tree nodes back and forth to their parents?

There are many data structures which are in undirected relations to
others.
--
http://mail.python.org/mailman/listinfo/python-list

Re: problem with PyMapping_SetItemString()

[Aaron Brady]

On Apr 21, 2:25 am, rahul  wrote:
> i have a c extension
snip
>           dict=PyEval_GetLocals();
snip
>           PyMapping_SetItemString(dict,varname,newVar_pyvalue);
snip

> than first two test cases runs correctly and gives result for var1
> "value changed"  but 3rd test case not gives correct result and value
> of var1 remains "abcd"
>
> why this happen and how i correct it ??

There's no good way to change the variable space programmatically.
Python has more flexibility than most other languages as it is.
Anyway, your best bet is to use an independent namespace:

class Globals: pass
setattr( Globals, 'itemX', 'value1' )
setattr( Globals, 'itemX', 'value2' )

Your 'updateCheck' function would then have to take the namespace as
an additional parameter.
--
http://mail.python.org/mailman/listinfo/python-list

relation class

[Aaron Brady]

Hi all,

I think Python should have a relation class in the standard library.
Fat chance.  I want to write a recipe for it, but I don't know how.  I
want your advice on some of the trade-offs, what it should look like,
what the pitfalls are, different strengths and weaknesses, etc.

Fundamentally, a relation is a set of tuples.  A simple generator
expression or itertools filter can fulfill the requirements of a
query.  But the fun stops there unless you want faster-than-linear
lookup times.  I do.

For matching equality, you need per-column hash tables
(dictionaries).  If I have a "WHERE lastname= 'Newton'" clause, I can
get all the matching records in constant time.  For matching
inequality, you need a per-column sorted list (balanced tree).  If I
have a "WHERE sales> 400" clause, I can get one matching record in log-
n time, and all of them in (log-n)+k time.

That's not the fun part either.  What is the fun part, is the exact
nuance of query structure in a select statement.  I want to return an
iterator from the following function calls.

recordset= Parts.select( [ "part" ], model== 'foopad' )
recordset= Sales.select( [ "model" ], sales>400 and sales<600 )
recordset= (Parts+Sales).select( [ "part" ], sales>400 and sales<600 )

The third of these is a join statement.  It selects every part which
was in at least one model between 400 and 600 of which were sold.  It
might need something more explicit, especially for the different types
of joins: 'Parts.join( Sales ).select' and 'Parts.innerjoin
( Sales ).select', or 'relation.innerjoin( Parts, Sales ).select'.

Unfortunately, even in this form, it won't work.  While the statements
are valid expressions, the second argument is evaluated too soon.  If
I put a lambda in front of it, I lose the ability to beat linear-time
lookups, and I might as well just use 'ifilter'.  I want live Python
objects in the tuples and queries, so I can't just convert everything
to a string.  What are my options?  This is Python, so they can't all
be bad.

P.S.  Recurring topic!
--
http://mail.python.org/mailman/listinfo/python-list

Re: generating random tuples in python

[Aaron Brady]

On Apr 20, 11:04 pm, per  wrote:
> On Apr 20, 11:08 pm, Steven D'Aprano
>
>
>
>  wrote:
> > On Mon, 20 Apr 2009 11:39:35 -0700, per wrote:
> > > hi all,
>
> > > i am generating a list of random tuples of numbers between 0 and 1 using
> > > the rand() function, as follows:
>
> > > for i in range(0, n):
> > >   rand_tuple = (rand(), rand(), rand()) mylist.append(rand_tuple)
>
> > > when i generate this list, some of the random tuples might be very close
> > > to each other, numerically. for example, i might get:
> > [...]
> > > how can i maximize the amount of "numeric distance" between the elements
> > > of
> > > this list, but still make sure that all the tuples have numbers strictly
> > > between 0 and 1 (inclusive)?
>
> > Well, the only way to *maximise* the distance between the elements is to
> > set them to (0.0, 0.5, 1.0).
>
> > > in other words i want the list of random numbers to be arbitrarily
> > > different (which is why i am using rand()) but as different from other
> > > tuples in the list as possible.
>
> > That means that the numbers you are generating will no longer be
> > uniformly distributed, they will be biased. That's okay, but you need to
> > describe *how* you want them biased. What precisely do you mean by
> > "maximizing the distance"?
>
> > For example, here's one strategy: you need three random numbers, so
> > divide the complete range 0-1 into three: generate three random numbers
> > between 0 and 1/3.0, called x, y, z, and return [x, 1/3.0 + y, 2/3.0 + z].
>
> > You might even decide to shuffle the list before returning them.
>
> > But note that you might still happen to get (say) [0.332, 0.334, 0.668]
> > or similar. That's the thing with randomness.
>
> > --
> > Steven
>
> i realize my example in the original post was misleading. i dont want
> to maximize the difference between individual members of a single
> tuple -- i want to maximize the difference between distinct tuples. in
> other words, it's ok to have (.332, .334, .38), as long as the other
> tuple is, say, (.52, .6, .9) which is very difference from (.332, .
> 334, .38).  i want the member of a given tuple to be arbitrary, e.g.
> something like (rand(), rand(), rand()) but that the tuples be very
> different from each other.
>
> to be more formal by very different, i would be happy if they were
> maximally distant in ordinary euclidean space... so if you just plot
> the 3-tuples on x, y, z i want them to all be very different from each
> other.  i realize this is obviously biased and that the tuples are not
> uniformly distributed -- that's exactly what i want...
>
> any ideas on how to go about this?
>
> thank you.

Two ideas.  One, start with a square grid and jitter the individual
points by a small random amount.  Two, start with one point, and move
from it by a random large distance: a2= a1+ .5+ rand( ), then %1.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a programming language that is combination of Python andBasic?

[Aaron Brady]

On Apr 19, 3:05 am, "Hendrik van Rooyen"  wrote:
>  "Aaron Brady"  wrote:
>
> On Apr 18, 4:44 am, "Hendrik van Rooyen"  wrote:
>
> >> to untangle some spaghetti code. He did not mention if
> >> the spaghetti was actually doing it's job, bug free, which
> >> IMO is the only rational test for the quality of a piece
>
> >I don't use 'rational' in the same way.  Do you mean objective?  Do
> >readability, brevity, simplicity, purity, etc. contribute to quality?
> >Is program quality equivalent (or identical) to code quality?
>
> This paragraph illustrates the problem, I think:
>
> Is there a significant difference between "rational" and "objective"?
> Define "readability, brevity, simplicity, purity, etc" as applied
> to the quality of a programme - it leads, inevitably, to a floundering
> around in a wash of words.

It is an old philosopher's problem, that is, a problem of old
philosophers: do all rational wo/men behave the same given the same
situations, or more specifically, analogously given related
situations?

Rational and objective are related, I think.  An objective test of
code would be a process anyone can follow and derive the same result,
such as how many identifiers it contains.  A rational test involves
some sort of abstraction of humans, such as what the ultimately
rational human likes most, etc.  Objective partial credit on an exam,
for example, must come from a black-and-white rubric with exactly no
room for interpretation.  Rational partial credit could involve
inferences, guesses, and context, esp. acquaintance with the taker.

I agree that many of those traits are matters of degree and multi-
dimensional.  I am looking for an objective metric of code: code X has
traits P, Q, and R.  Period.  It would probably be simpler to start
describing code objectively, rather than trying to square-peg-round-
hole existing words.  However, there may be no objective test of
especially large sizes of code, and may fall to some sort of
interpretation or assessment of the most rational wo/man.

If I disagree with your measuring of a code with objective criteria,
we merely walk through the measurement, and resolve our disagreement
from the criteria upon discovering it.  If I disagree with your
measuring of it with rational criteria, we have to take into account
all sorts of trade-offs, such as long-short term, one-many readers,
different consequences, etc.  I'm speculating a little bit in defining
those words.

I advance that two equifunctional programs (same input-> same output)
can be compared w.r.t. brevity, to take an example, up to isomorphism
of identifier names, control structures, and statement order.  That
is, we can determine how brief code X is, disregarding identifier
names, and statement order where irrelevant.  Control structure
brevity might be something we're trying to measure:

while 1:
  if cond(): break

while not cond():
  ...

Are these equally brief, assuming they accomplish the same thing?  If
we don't have unanimous agreement on the answer to that question, we
must conclude that even brevity is a composite trait and we'll have to
start simpler.  A theory of traits would have to make concrete
predictions about *something*.  If some code has trait P, it will Q.

> However, to stop playing Devil's Advocate, there is such a thing
> as code quality, but it cannot be defined, just like quality in general
> terms cannot be defined - as soon as you try, it turns to dross in
> your hands.  Read Robert Pfirsig's "Zen and the art of motorcycle
> maintenance" for a full explanation of this effect.

Are you saying that quality reduces to how pleasant a particular
reader, or composite of multiple readers, finds the code?

"In the 1960s, programmers gave goto statements an average rating of 6
out of 10, 10 being most pleasing.  In the 70s, that rating dropped to
5 out of 10.  Subroutines were initially disliked, but came to be
favored, advancing from 4/10 in the 60s to 7/10 in the 80s and on.
Possible confounding variables include the increase in processor speed
and thus the increased forgivingness of running time of gosubs; the
prevalence of languages capable of generating subroutine calls; and
the efficiency of the particular calls generated, surveys of which
were not attempted."  --Fabrication

> >> I do not agree with the reasoning that effectively says:
> >> "If it is difficult to comprehend, it must be wrong"
>
> >Wrong no, but impractical, possibly or probably or almost certainly,
> >notwithstanding the subject-dependence of ease of comprehension.
> >Simple code is more future-resilient than that which is difficult to
> >comprehend, even holding the language (version) constant.  It is a
>
> I think tha

Re: Overriding methods per-object

[Aaron Brady]

On Apr 17, 9:41 pm, Steven D'Aprano  wrote:
> On Fri, 17 Apr 2009 18:22:49 -0700, Pavel Panchekha wrote:
> > I've got an object which has a method, __nonzero__ The problem is, that
> > method is attached to that object not that class
>
> >> a = GeneralTypeOfObject()
> >> a.__nonzero__ = lambda: False
> >> a.__nonzero__()
> > False
>
> > But:
>
> >> bool(a)
> > True
>
> > What to do?
>
> (1) Don't do that.
>
> (2) If you *really* have to do that, you can tell the class to look at
> the instance:
>
> class GeneralTypeOfObject(object):
>     def __nonzero__(self):
>         try:
>             return self.__dict__['__nonzero__']
>         except KeyError:
>             return something
snip

I think you need to call the return, unlike you would in
'__getattr__'.
 return self.__dict__['__nonzero__']( )

You're free to use a different name for the method too.
 return self.custom_bool( )

And you don't even have to implement an ABC.  Er... /have/ to, that
is.
--
http://mail.python.org/mailman/listinfo/python-list

Re: python alternatives to C structs??

[Aaron Brady]

On Apr 18, 2:25 pm, KoolD  wrote:
> Hey all,
> I need to convert a C code to python please help me figure out how to
> do
> it.
> Suppose the C program's like:
>
> 
> typedef struct _str
> {
> int a;
> char *b;
> int c;}str;
>
> int main()
> {
> str mbr;
> fd=open("/dev/sda",O_RDONLY);
> read(fd,&mbr,sizeof(str));}
>
> 
>
> Is there a way to code it in python.
>
> Thanks
> Sourya

There is the 'ctypes' module, but you might also need 'mmap', for the
'read( &mbr )' operation.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a programming language that is combination of Python andBasic?

[Aaron Brady]

On Apr 18, 4:44 am, "Hendrik van Rooyen"  wrote:
> "baykus"  wrote:
> > I guess I did not articulate myself well enough. I was just looking
> > for a toy to play around. I never suggested that Python+Basic would be
> > better than Python and everyone should use it. Python is Python and
> > Basic is Basic. I am not comparing them at all. I understand the
> > merits of Python but that does not mean I can play with ideas?
>
> Apparently this is not allowed by the CS thought police.
>
> The reasoning is based on an Argument from Authority,
> namely the Dijkstra link.

Boo, Appeal to Authority!

snip
> to untangle some spaghetti code.  He did not mention if
> the spaghetti was actually doing it's job, bug free, which
> IMO is the only rational test for the quality of a piece

I don't use 'rational' in the same way.  Do you mean objective?  Do
readability, brevity, simplicity, purity, etc. contribute to quality?
Is program quality equivalent (or identical) to code quality?

> of code, because it is the reason for its existence.  
> The aesthetics are, like all aesthetics, a matter of opinion.
>
> I do not agree with the reasoning that effectively says:
> "If it is difficult to comprehend, it must be wrong"

Wrong no, but impractical, possibly or probably or almost certainly,
notwithstanding the subject-dependence of ease of comprehension.
Simple code is more future-resilient than that which is difficult to
comprehend, even holding the language (version) constant.  It is a
matter of priorities, which have no objective right.  The amount of
people that can comprehend a code structure is a competing value to
that of exploration, pioneering, research, and development.  However,
even in simplest terms, some structures e.g. recursion, may be
difficult to comprehend, but that doesn't mean they would be better
more complicated.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Something weird about re.finditer()

[Aaron Brady]

On Apr 17, 9:37 pm, Steven D'Aprano  wrote:
> On Sat, 18 Apr 2009 12:37:09 +1200, Lawrence D'Oliveiro wrote:
> > In message ,
> > Steven D'Aprano wrote:
>
> >> BTW, testing for None with == is not recommended, because one day
> >> somebody might pass your function some strange object that compares
> >> equal to None.
>
> > Presumably if it compares equal to None, that is by design, precisely so
> > it would work in this way.
>
> In context, no. We're not talking about somebody creating an object which
> is equivalent to None when treated as a value, but using None as a
> sentinel. Sentinels are markers, and it is important that nothing else
> can be mistaken for that marker or breakage will occur.
>
> Of course, if the caller knows how the sentinel is used, then he might
> choose to duplicate that usage but pass some other object. But that would
> be stupid and should be discouraged. I mean, what would be the point? I
> can think of use-cases for creating something that returns equal to None
> -- the Null object pattern comes to mind. But what would be the point of
> creating an object that was not None but would fool a function into
> treating it as the same sentinel as None?

In that case, they could use separate sentinels, that are instances of
a class or classes that have defined behavior for comparing to each
other.

It might get as bad as setting a flag on the class or sentinel, though
you'd have to be careful about concurrency and especially nested
calls.

You'd have to rely on the user function to use equality instead of
identity testing, since 'sentinel is None' won't return true no matter
what you do to it.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Overriding methods per-object

[Aaron Brady]

On Apr 17, 9:28 pm, Pavel Panchekha  wrote:
> > The docs don't say you can do that:
>
> Thanks, hadn't noticed that.
>
> > Should you be able to?
>
> I'd say so. In my case, I need a class that can encapsulate any
> object, add a few methods to it, and spit something back that works
> just like the object, but also has those extra methods. I can't just
> add the methods, because it has to work on e.g. lists. So I'll have to
> end up defining all the possible methods on that class (and that's
> still not best because I can't use hasattr to test if, for example,
> addition is allowed on that object).
>
> On the other hand, I see how this severely restricts the possibly
> optimizations that can be made in the interpreter.

Can you dynamically subclass it:

def subclasser( obj ):
  class newclass( obj.__class__ ):
def __nonzero__.
  obj.__class__= newclass

FYI, the __class__ member is assignable, though I've personally never
done it in practice.
--
http://mail.python.org/mailman/listinfo/python-list

Re: send() to a generator in a "for" loop with continue(val)??

[Aaron Brady]

On Apr 17, 3:59 pm, Dale Roberts  wrote:
> I've started using generators for some "real" work (love them!), and I
> need to use send() to send values back into the yield inside the
> generator. When I want to use the generator, though, I have to
> essentially duplicate the machinery of a "for" loop, because the "for"
> loop does not have a mechanism to send into the generator. Here is a
> toy example:
>
> def TestGen1():
>     for i in xrange(3):
>         sendval = yield i
>         print "   got %s in TestGen()" % sendval
>
> g = TestGen1()
> sendval = None
> try:
>     while True:
>         val = g.send(sendval)
>         print 'val in "while" loop %d' % val
>         sendval = val * 10
> except StopIteration: pass
>
> I have to explicitly create the generator with an assignment, send an
> initial None to the generator on the first go, then have to catch the
> StopIteration exception. In other words, replicate the "for"
> mechanism, but use send() instead of next().
>
> It would be nice if I could just do this instead:
>
> for val in TestGen1():
>     print 'val in "for" loop %d' % val
>     continue(val*10)
>
> ...or something similar. Is this an old idea? Has it been shot down in
> the past already? Or is it worth pursuing? I Googled around and saw
> one hit 
> here:http://mail.python.org/pipermail/python-ideas/2009-February/003111.html,
> but not much follow-up.
>
> I wonder if people want to keep the idea of an "iterator" style
> generator (where send() is not used) separate from the idea of a "co-
> routine" style generator (where send() is used). Maybe combining the
> two idioms in this way would cause confusion?
>
> What do folks think?
>
> dale

You can do it with a wrapping generator.  I'm not sure if it
interferes with your needs.  It calls 'next' the first time, then just
calls 'send' on the parameter with the value you send it.

>>> def genf( ):
... x= True
... for _ in range( 10 ):
... i= yield( x )
... if i is None:
... i= not x
... x= i
...
>>> def for_send( gen ):
... x= next( gen )
... while 1:
... x= yield( x )
... x= gen.send( x )
...
>>> a= for_send( genf( ) )
>>> i= 0
>>> for x in a:
... print( i, x )
... i+= 1
... if not i% 3:
... _= a.send( False )
...
0 True
1 False
2 True
3 True
4 False
5 True
6 True
7 False

As you can see, it skips a beat every third iteration as directed.
--
http://mail.python.org/mailman/listinfo/python-list

Re: large db question about no joins

[Aaron Brady]

On Apr 17, 3:16 pm, "Martin P. Hellwig" 
wrote:
> Daniel Fetchinson wrote:
>
> 
>
>
>
> > In an relational database setting you would have a table for artists,
> > a table for cd's and a table for songs and a table for comments where
> > people can comment on songs. All of this with obvious foreign keys.
> > Now you want to display on your website the total number of cd's, the
> > total number of songs and the total number of comments because that
> > looks very cool on top of every page: 1242342 cd's and 134242342342
> > songs and 284284728347284728 comments!
>
> > Okay, so you need to issue a query with joins in your relational db.
> > Or in another query, you want to select all comments of an artist,
> > meaning all comments that were comments on a song belonging to cd's of
> > a given artist. This would also involve joins.
>
> > How would I do these queries if I can't have joins or in other words
> > how would I lay out my data storage?
>
> > Thanks by the way for your help and replies,
> > Daniel
>
> What actually is happening here is that feature x which is usually in an
> abstraction layer (like a RDBM) is not available to you. Which means you
> have to write code which does the same but on your side of the code.
>
> I played a bit around with the GAE some time ago and designed the
> storage of all data in an sort of EAV model
> (http://en.wikipedia.org/wiki/Entity-Attribute-Value_model)
> The abstraction layer had all my meta code, counting stuff up was a
> matter of creating a list of id's from one section and with that list
> selecting the appropriate records on the other section, then count up
> the number of results or use the result again, it is very likely you
> have to do quite a lot of queries before you get the result.
>
> So in other words, just lay out the data which makes the most sense to
> you, the pain of recreating SQL like logic is there no matter what
> layout you choose.
>
> --
> mph

Instead of creating a new table, just create an iterator that returns
successive entries in it.  Just don't change the size of either table
during iteration.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Overriding methods per-object

[Aaron Brady]

On Apr 17, 8:22 pm, Pavel Panchekha  wrote:
> I've got an object which has a method, __nonzero__
> The problem is, that method is attached to that object not that class
>
> > a = GeneralTypeOfObject()
> > a.__nonzero__ = lambda: False
> > a.__nonzero__()
>
> False
>
> But:
>
> > bool(a)
>
> True
>
> What to do?

The docs don't say you can do that:

Special method names
A class can implement certain operations that are invoked by special
syntax (such as arithmetic operations or subscripting and slicing) by
defining methods with special names.

Should you be able to?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Man Bites Python

[Aaron Brady]

On Apr 17, 7:03 pm, "AD."  wrote:
> On Apr 17, 11:11 pm, Aaron Brady  wrote:
>
> > Man bites python.
> > Python bites dog.
> > Dog bites man.
>
> or just:
>
> man,python bites python,man
>
> No need for the temporary value in Python.

Is Python a mutable type?
Just don't pass it by reference.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Help improve program for parsing simple rules

[Aaron Brady]

On Apr 17, 2:01 pm, Paul McGuire  wrote:
> On Apr 17, 1:26 pm, Aaron Brady  wrote:
>
> > Hi, not to offend; I don't know your background.  
>
> Courtesy on Usenet!!!  I'm going to go buy a lottery ticket!
>
> Not to worry, I'm a big boy.  People have even called my baby ugly,
> and I manage to keep my blood pressure under control.
>
>
>
> > One thing I like
> > about Python is it and the docs are careful about short-circuiting
> > conditions.  ISTR that C left some of those details up to the compiler
> > at one point.
>
> > >>> def f():
>
> > ...     print( 'in f' )
> > ...     return 10
> > ...>>> 0
> > in f
> > True>>> 0
> > in f
> > in f
> > True
>
> > Therefore, if op{n} has side effects, 'op1 operator1 op2 AND op2
> > operator2 op3' is not equivalent to 'op1 optor1 op2 optor2 op3'.
>
> Interesting point, but I don't remember that "A < B < C" is valid C
> syntax, are you perhaps thinking of a different language?
>
> By luck, my implementation of EvalComparisonOp.eval does in fact
> capture the post-eval value of op2, so that if its evaluation caused
> any side effects, they would not be repeated.
>
> -- Paul

It was very hazy in my memory and is easily solved by a web search.
Perhaps I was thinking of the short-circuiting of 'and', and mixing it
up with the warning about parentheses in macros.

-Macros?  NO, Batman.
-I'm afraid so Robin.  Macros.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Lambda alternative?

[Aaron Brady]

On Apr 17, 1:43 pm, "J. Cliff Dyer"  wrote:
> On Thu, 2009-04-16 at 13:33 +0200, Hrvoje Niksic wrote:
> > mousemeat  writes:
>
> > > Correct me if i am wrong, but i can pickle an object that contains a
> > > bound method (it's own bound method).
>
> > No, you can't:
>
> > >>> import cPickle as p
> > >>> p.dumps([])
> > '(l.'
> > >>> p.dumps([].append)
> > Traceback (most recent call last):
> >   File "", line 1, in 
> > TypeError: expected string or Unicode object, NoneType found
>
> Yes he can.  mousemeat stated that he could pickle an object that
> *contains* a bound method, not that he could pickle the method itself.
>
> That said, you can make an instance method out of a lambda, just as well
> as any named function, and you can pickle that object, too:
snip

'Contains' here is ambiguous.  If the object contains a bound method,
that is, if a bound method is in its dictionary, you can't.

>>> import pickle as p
>>> class A: pass
...
>>> a= A()
>>> class A:
... def f( self ): print( 'f' )
...
>>> a= A()
>>> class B: pass
...
>>> b= B()
>>> b.f= a.f
>>> b.f()
f
>>> p.dumps( b )
Traceback (most recent call last):
  File "", line 1, in 
  File "C:\Programs\Python30\lib\pickle.py", line 1329, in dumps
Pickler(f, protocol).dump(obj)
_pickle.PicklingError: Can't pickle : attribute
lookup
builtins.method failed

In this example, 'b' contains a bound method, 'a.f'.  However, for
other definitions of 'contains', such as if 'b' is an instance of a
class that contains methods, you can.  But in that case, the method is
not in 'b.__dict__'.

>>> b.__dict__
{'f': >}

--
http://mail.python.org/mailman/listinfo/python-list

Re: Help improve program for parsing simple rules

[Aaron Brady]

On Apr 17, 12:15 pm, Paul McGuire  wrote:
> On Apr 17, 10:43 am, John Machin  wrote:
snip
> not only does this handle
> "0.00 LE A LE 4.00", but it could also evaluate "0.00 LE A LE 4.00 LE
> E > D".  (I see that I should actually do some short-circuiting here -
> if ret is false after calling fn(val1,val2), I should just break out
> at that point.  I'll have that fixed in the online version shortly.)

Hi, not to offend; I don't know your background.  One thing I like
about Python is it and the docs are careful about short-circuiting
conditions.  ISTR that C left some of those details up to the compiler
at one point.

>>> def f():
... print( 'in f' )
... return 10
...
>>> 0>> 0http://mail.python.org/mailman/listinfo/python-list

Re: Help improve program for parsing simple rules

[Aaron Brady]

On Apr 17, 7:37 am, prueba...@latinmail.com wrote:
> On Apr 16, 3:59 pm, Aaron Brady  wrote:
>
>
>
> > On Apr 16, 10:57 am, prueba...@latinmail.com wrote:
>
> > > Another interesting task for those that are looking for some
> > > interesting problem:
> > > I inherited some rule system that checks for programmers program
> > > outputs that to be ported: given some simple rules and the values it
> > > has to determine if the program is still working correctly and give
> > > the details of what the values are. If you have a better idea of how
> > > to do this kind of parsing please chime in. I am using tokenize but
> > > that might be more complex than it needs to be. This is what I have
> > > come up so far:
>
snip
> > > def main():
> > >     for cur_rule in rules[20:26]:
> > >         tokens=get_tokens(cur_rule)
> > >         normal=replace_comps(tokens,COMP_REPLACERS)
> > >         subst=replace_names(normal,vars_)
> > >         groups=split_seccions(subst,COMP_REPLACERS.values())
> > >         rep=all_seccions(groups)
> > >         rep_out=''.join(x[0]+x[1] for x in rep)
> > >         calc=calc_seccions(rep)
> > >         calc_out=''.join(x[0]+x[1] for x in calc)
> > >         deltas=calc_deltas(calc)
> > >         result=eval(calc_out,{},{})
>
> > snip
>
snip
> > >>> a= '-1000.00 < A < 0.00'
> > >>> eval( a, { 'A': -100 } )
snip
> > >>> a= '-1000.00 LE A LE 0.00'
> > >>> b= a.replace( ' LE ', ' <= ' )
snip
>
> I know about evals implication of safety. Rules are defined by the
> programmers so I don't worry too much about it at this point. They
> should know better than messing up their application server. Unless
> there is some easier way to do it I am willing to take the risk.
> Precedence is standard math, we can always add some extra parenthesis
> to the rules, I don't thing the old system would mind.
>
> I thought about using eval with a locals dictionary, but they want
> output of the intermediate values. I want to avoid the situation where
> the intermediate output does not match what eval is doing.

I take you to need the operands to the individual comparison
operators; that is, each side of each comparison.  There might be a
way using abstract syntax trees.  Or, a basic 're' split can simplify
it.

>>> import re
>>> a= '-1000.00 LE A LE 0.00'
>>> b= re.split( r'\s(LT|GT|LE|GE|=|<|>|<=|>=)\s', a )
>>> b
['-1000.00', 'LE', 'A', 'LE', '0.00']
>>> COMP_REPLACERS={'LT':'<', 'GT':'>', 'LE':'<=', 'GE':'>=', '=':'=='}
>>> c= [ COMP_REPLACERS.get( x, x ) for x in b ]
>>> c
['-1000.00', '<=', 'A', '<=', '0.00']
>>> vars_={'A': 0, 'B': 1.1, 'C': 2.2, 'D': 3.3, 'E': 4.4, 'F': 5.5, 'G':
... 6.6, 'H':7.7, 'I':8.8, 'J':9.9}
>>> d= [ str( vars_.get( x, x ) ) for x in c ]
>>> d
['-1000.00', '<=', '0', '<=', '0.00']
>>> eval( ''.join( d ) )
True

The 'dict.get( x, x )' expression returns the value of the entry for
x, or x itself if it is not present.

No promises.  I didn't think it all the way through.
--
http://mail.python.org/mailman/listinfo/python-list