Re: Grouping pairs - suggested tools
* Arnaud Delobelle, on 21.09.2010 11:13:
On Sep 21, 7:19 am, "Alf P. Steinbach /Usenet" wrote:
* Alf P. Steinbach /Usenet, on 21.09.2010 01:09:
* Astley Le Jasper, on 20.09.2010 23:42:
I have a list of tuples that indicate a relationship, ie a is related
to b, b is related to c etc etc. What I want to do is cluster these
relationships into groups. An item will only be associated with a
single cluster.
Before I started, I wondered if there was any particular tool within
Python I should be looking at. I don't expect anyone to code this for
me, just say ... "you need to look at using x". I was going to use
populate a dictionary and
Sorry for being so vague.
Example Data:
[(a,b)
(a,c)
(a,d)
(b,c)
(b,d)
(c,d)
(e,f)
(e,g)
(f,g)
(h,i)]
Output (grouping id, item ref)
(1,a),
(1,b),
(1,c),
(1,d),
(2,e),
(2,f),
(2,g),
(3,h),
(3,i)
It seems to be the same problem as "equivalence sets".
This problem was solved early on because e.g. Fortran compilers had to construct
such sets (equivalence partitions of a set).
I though I'd just say "Google it!", because I know there's a standard algorithm
but I can't recall the name. However, googling it I found no mention of that
algorithm. Not even in the Wikipedia articles on equivalence sets.
A number of approaches spring to mind:
Approach 1:
Multi-pass. Originally you assign a distinct set number to each symbol.
In each pass through the symbols you replace one number with another as
per one of the equivalence specs. Stop when no replacements are done.
Approach 2:
Merging. For each new equivalence A=B you check whether A has been assigned
to a set, if not you create a new one, call that S1, and ditto for B, S2.
Merge S1 and S2, e.g. move all elements of S2 to S1.
Approach 3:
In a first pass convert the data to more explicit form, linking each symbol
to the symbols it's directly equivalent to. Then in second pass simply drag
out each equivalence group (recurse on each symbol's list of equivalences).
Approaches 1 and 2 seem to be pretty inefficient for a large number of symbols,
but I think approach 1 may be a practical option for a small number of symbols.
Uhm, thinking about it (it must have been my unconscious mind doing the work, it
just popped into my head), if you first sort each individual equivalence
relation so that you never have e.g. C=A but only A=C and so on, and then sort
the list of equivalences, then it should reduce to walking the list and just
starting a new set whenever a symbol is encountered that isn't yet in a set.
class Attributes: pass
sym = Attributes()
for name in ("a", "b", "c", "d", "e", "f", "g", "h", "i"):
setattr( sym, name, name )
eq_specs = [
(sym.a, sym.d),
(sym.b, sym.a),
(sym.b, sym.c),
(sym.b, sym.d),
(sym.c, sym.d),
(sym.c, sym.a),
(sym.e, sym.f),
(sym.e, sym.g),
(sym.f, sym.g),
(sym.h, sym.i),
]
equalities = []
for eq in eq_specs:
sorted_eq = eq if eq[0]<= eq[1] else (eq[1], eq[0])
equalities.append( sorted_eq )
equalities.sort()
eq_sets = {}
eq_set_ids = {}
current_eq_set_id = 0
for eq in equalities:
This would make the body of the loop easier to read:
for x, y in equalities:
if eq_set_ids.get( eq[0] ) is None:
Why not simply:
if eq[0] in eq_set_ids:
current_eq_set_id += 1
eq_set_ids[eq[0]] = current_eq_set_id
eq_sets[current_eq_set_id] = set( eq[0] )
eq_set_id = eq_set_ids[eq[0]]
eq_set_ids[eq[1]] = eq_set_id
eq_sets[eq_set_id].add( eq[1] )
for eq_set_id in eq_sets.keys():
for sym_name in eq_sets[eq_set_id]:
print( "{}, {}".format( eq_set_id, sym_name ) )
1, a
1, c
1, b
1, d
2, e
2, g
2, f
3, i
3, h
Disclaimer: for me it's pretty late in day/night.
Cheers& hth.,
- Alf
--
blog athttp://alfps.wordpress.com>
I think this won't work with the following graph:
eq_specs = [('a', 'c'), ('b', 'd'), ('c', 'd')]
Note that it is already sorted according to your sorting method. I
don't have a Python machine to check this but I believe it will
output:
1, a
1, c
1, d
2, b
The flaw is that you fail to consider that the two vertices in the
current pair may already be part of a (partial) connected component.
Yeah, thanks.
I think the three general methods I listed will nicely work, though.
Moral: don't post insights from dream-world... ;-)
Cheers,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Grouping pairs - suggested tools
* Alf P. Steinbach /Usenet, on 21.09.2010 01:09:
* Astley Le Jasper, on 20.09.2010 23:42:
I have a list of tuples that indicate a relationship, ie a is related
to b, b is related to c etc etc. What I want to do is cluster these
relationships into groups. An item will only be associated with a
single cluster.
Before I started, I wondered if there was any particular tool within
Python I should be looking at. I don't expect anyone to code this for
me, just say ... "you need to look at using x". I was going to use
populate a dictionary and
Sorry for being so vague.
Example Data:
[(a,b)
(a,c)
(a,d)
(b,c)
(b,d)
(c,d)
(e,f)
(e,g)
(f,g)
(h,i)]
Output (grouping id, item ref)
(1,a),
(1,b),
(1,c),
(1,d),
(2,e),
(2,f),
(2,g),
(3,h),
(3,i)
It seems to be the same problem as "equivalence sets".
This problem was solved early on because e.g. Fortran compilers had to construct
such sets (equivalence partitions of a set).
I though I'd just say "Google it!", because I know there's a standard algorithm
but I can't recall the name. However, googling it I found no mention of that
algorithm. Not even in the Wikipedia articles on equivalence sets.
A number of approaches spring to mind:
Approach 1:
Multi-pass. Originally you assign a distinct set number to each symbol.
In each pass through the symbols you replace one number with another as
per one of the equivalence specs. Stop when no replacements are done.
Approach 2:
Merging. For each new equivalence A=B you check whether A has been assigned
to a set, if not you create a new one, call that S1, and ditto for B, S2.
Merge S1 and S2, e.g. move all elements of S2 to S1.
Approach 3:
In a first pass convert the data to more explicit form, linking each symbol
to the symbols it's directly equivalent to. Then in second pass simply drag
out each equivalence group (recurse on each symbol's list of equivalences).
Approaches 1 and 2 seem to be pretty inefficient for a large number of symbols,
but I think approach 1 may be a practical option for a small number of symbols.
Uhm, thinking about it (it must have been my unconscious mind doing the work, it
just popped into my head), if you first sort each individual equivalence
relation so that you never have e.g. C=A but only A=C and so on, and then sort
the list of equivalences, then it should reduce to walking the list and just
starting a new set whenever a symbol is encountered that isn't yet in a set.
class Attributes: pass
sym = Attributes()
for name in ("a", "b", "c", "d", "e", "f", "g", "h", "i"):
setattr( sym, name, name )
eq_specs = [
(sym.a, sym.d),
(sym.b, sym.a),
(sym.b, sym.c),
(sym.b, sym.d),
(sym.c, sym.d),
(sym.c, sym.a),
(sym.e, sym.f),
(sym.e, sym.g),
(sym.f, sym.g),
(sym.h, sym.i),
]
equalities = []
for eq in eq_specs:
sorted_eq = eq if eq[0] <= eq[1] else (eq[1], eq[0])
equalities.append( sorted_eq )
equalities.sort()
eq_sets = {}
eq_set_ids = {}
current_eq_set_id = 0
for eq in equalities:
if eq_set_ids.get( eq[0] ) is None:
current_eq_set_id += 1
eq_set_ids[eq[0]] = current_eq_set_id
eq_sets[current_eq_set_id] = set( eq[0] )
eq_set_id = eq_set_ids[eq[0]]
eq_set_ids[eq[1]] = eq_set_id
eq_sets[eq_set_id].add( eq[1] )
for eq_set_id in eq_sets.keys():
for sym_name in eq_sets[eq_set_id]:
print( "{}, {}".format( eq_set_id, sym_name ) )
1, a
1, c
1, b
1, d
2, e
2, g
2, f
3, i
3, h
Disclaimer: for me it's pretty late in day/night.
Cheers & hth.,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Grouping pairs - suggested tools
* Astley Le Jasper, on 20.09.2010 23:42: I have a list of tuples that indicate a relationship, ie a is related to b, b is related to c etc etc. What I want to do is cluster these relationships into groups. An item will only be associated with a single cluster. Before I started, I wondered if there was any particular tool within Python I should be looking at. I don't expect anyone to code this for me, just say ... "you need to look at using x". I was going to use populate a dictionary and Sorry for being so vague. Example Data: [(a,b) (a,c) (a,d) (b,c) (b,d) (c,d) (e,f) (e,g) (f,g) (h,i)] Output (grouping id, item ref) (1,a), (1,b), (1,c), (1,d), (2,e), (2,f), (2,g), (3,h), (3,i) It seems to be the same problem as "equivalence sets". This problem was solved early on because e.g. Fortran compilers had to construct such sets (equivalence partitions of a set). I though I'd just say "Google it!", because I know there's a standard algorithm but I can't recall the name. However, googling it I found no mention of that algorithm. Not even in the Wikipedia articles on equivalence sets. A number of approaches spring to mind: Approach 1: Multi-pass. Originally you assign a distinct set number to each symbol. In each pass through the symbols you replace one number with another as per one of the equivalence specs. Stop when no replacements are done. Approach 2: Merging. For each new equivalence A=B you check whether A has been assigned to a set, if not you create a new one, call that S1, and ditto for B, S2. Merge S1 and S2, e.g. move all elements of S2 to S1. Approach 3: In a first pass convert the data to more explicit form, linking each symbol to the symbols it's directly equivalent to. Then in second pass simply drag out each equivalence group (recurse on each symbol's list of equivalences). Approaches 1 and 2 seem to be pretty inefficient for a large number of symbols, but I think approach 1 may be a practical option for a small number of symbols. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Static typing, Python, D, DbC
* Paul Rubin, on 13.09.2010 04:50: Ed Keith writes: I think DbC as envisioned by the Eiffel guy... the term is that it's a static verification technique, Eiffel throws an exception when a contract is violated. That is run time behavior, not static verification. The runtime checks are for when static analysis hasn't been supplied (that is usually a partly manual process). DBC is always intended to be statically verified as I understand it. Doing it at runtime is just a hackish fallback. DBC can't in generally be statically checked. E.g. a precondition of a routine might be that its argument is a sorted array. So regarding the nature of the checks it's not hopelessly incompatible with Python. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: [Q] How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
* Standish P, on 16.08.2010 09:20: [garble garble] Nonsense article "We look for an exogenous stack" cross-posted to [comp.lang.c], [comp.lang.c++], [comp.theory], [comp.lang.python], [comp.lang.forth]. Please refrain from following up on Standish' article. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Different python versions confusion under Windows Vista x64
* Edward Diener, on 19.07.2010 14:53: In Windows Vista x64 I have installed python 2.6 64-bit version and python 3.1 64-bit version to separate folders. Within the command interpreter I add python 2.6 to the PATH. In the command interpreter, When I type python somescript.py with an import sys print (sys.version) in the script, it shows: 3.1.2 (r312:79149, Mar 20 2010, 22:55:39) [MSC v.1500 64 bit (AMD64)] In the command interpreter if I type 'python' I see: Python 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD64)] on win32 Does anybody have any ideas why this is happening ? At a guess your description of what's happening is not entirely accurate. Although it could be, since Windows moves in mysterious ways. Please try the following commands in sequence, with no other commands: python -V echo %path% ftype python.file python somescript.py Then right-click the command interpreter's title bar to get edit menu. /Mark/ the text of your commands and results. Then /copy/ it to the clipboard (note: you can't use [Ctrl C] here, use the edit menu or just press Enter). Then post the commands and results here, /paste/ them into your message (e.g. [Ctrl V]). And then, if you haven't already figured it out, somebody probably will. :-) Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
CPython 3.1.1 docs error in noddy3 example?
"Extending and Embedding the Python Interpreter" §2.1.2, the noddy3 extension module example, uses "S" as format character for string arguments in its call to PyArg_ParseTupleAndKeywords. This causes Noddy to only accept bytes as arguments, instead of strings (format "U"). I suspect this is a leftover from Python 2.x, where strings were bytes, and that this is a bug? Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing: member type deduction for member pointers (Alf's device?)
* Vladimir Jovic, on 19.07.2010 09:41:
Alf P. Steinbach /Usenet wrote:
#include // PyWeakPtr, PyPtr, PyModule,
PyClass
using namespace progrock;
namespace {
using namespace cppy;
struct Noddy
{
PyPtr first;
PyPtr last;
int number;
Noddy( PyWeakPtr pySelf, PyPtr args, PyPtr kwArgs )
: number( 0 )
{
devsupport::suppressUnusedWarning( pySelf );
PyWeakPtr pFirstName = 0;
PyWeakPtr pLastName = 0;
static char*kwlist[] = { "first", "last", "number", 0 };
::PyArg_ParseTupleAndKeywords(
args.rawPtr(), kwArgs.rawPtr(), "|OOi", kwlist,
pointerTo( pFirstName ), pointerTo( pLastName ), &number
)
>> Accept< IsNonZero >()
|| throwX( "Invalid args" );
if( pFirstName != 0 ) { first = pFirstName; }
if( pLastName != 0 ){ last = pLastName; }
Why not initiallize all member variables in the initializer list?
'PyPtr' is a smart pointer with default constructor, so 'first' and 'last' are
initialized to zero implicitly.
My goal was to emulate the Python 3.1.1 docs' Noddy2 example as closely as
possible. And that code's assignment of zero to 'first' and 'last' here
corresponds to default initialization to zero. :-)
Anyway, parsing the Python arguments in the constructor initializer list is
possible, but would be inefficient to do directly since it would then involve
parsing the Python arguments three times (once for each data member).
There is at least one way around that problem, namely outfitting a derived class
with knowledge of the Noddy class' Python-side __init__ arguments.
But I haven't implemented such support, and I don't know if it would do any
good. As it is Noddy above is self-contained, except for exporting names to the
Python side. With a derived class doing the argument parsing it would be less
self-contained, and the machinery for that would add complexity, I think.
So, I strive to make things simple and explicit (KISS: Keep It Simple, Stupid!).
And I strive to not go into orgies of smart templates doing things implicitly...
}
PyPtr name()
{
(first != 0)
|| throwX( ::PyExc_AttributeError, "first" );
(last != 0)
|| throwX( ::PyExc_AttributeError, "last" );
Nice trick. I would still find this more readable :
if ( first != 0 )
{
throwX( ::PyExc_AttributeError, "first" );
}
Ah, well. :-)
You can read about the rationale here: http://alfps.wordpress.com/2010/07/18/cppx-b-true-or-b-thrown-using-the-throwing-pattern/>
-- posted yesterday.
Of course the only reason that those member variables /can/ be 0 is because I'm
following the doc's progression of examples. The next version in the docs avoid
the 0 possibility, as I recall. And the simple way of achieving that in cppy
would be to declare 'first' and 'last' as PyString instead of generic PyPtr...
return (PyString( first ) + L" " + PyString( last )).pyPtr();
}
};
struct NoddyPyClass: PyClass< Noddy >
{
NoddyPyClass( PyModule& m, PyString const& name, PyString
const& doc )
: PyClass< Noddy >( m, name, doc, Exposition()
.method(
L"name",CPPY_GLUE( name ),
L"Return the name, combining the first and last name"
)
.attribute(
L"first", CPPY_GLUE( first ), L"first name"
)
.attribute(
L"last",CPPY_GLUE( last ), L"last name"
)
.attribute(
L"number", CPPY_GLUE( number ),L"noddy number"
)
)
{}
};
class NoddyModule: public PyModule
{
private:
NoddyPyClassnoddyPyClass_;
public:
NoddyModule()
: PyModule(
L"noddy2", L"Example module that creates an extension
type." )
, noddyPyClass_( *this,
L"Noddy", L"A Noddy object has a name and a noddy
number" )
hmmm what is L ?
It's standard C and C++ notation for a wide character literal, where each
character is of type 'wchar_t' instead of plain 'char'.
Essentially the intent is to hold Unicode UTF16 or UTF32 code points, which is
how it is in practice, although for political correctness the C and C++
standards allow for other wide ch
Re: why is this group being spammed?
* be.krul, on 18.07.2010 07:01:
why is this group being spammed?
It depends a little on what you're asking, e.g. technical versus motivation.
But I'll answer about something you probably didn't mean to ask, namely what
human trait enables and almost forces that kind of behavior.
And I believe it is the same trait that let hypnotists do funny TV-shows with
quite normal folks behaving in ridiculous ways, the same trait that causes wars,
the same trait that causes religion, the same trait that holds back science so
that "paradigm shifts" only occur when new generations take over, a trait that
/makes/ your feet start moving to the rhythm when you hear the right kind of
music, and so on. Namely the instinctual drive to fit in in a social group by
sharing that group's behavior and values. This instinct is possibly stronger
than the sex drive, e.g., wars are seldom fought over sex ("surrender, and give
us sex!", no, that's not it, it's "surrender, and become part of us!").
Consider, there would be almost no spam if spamming didn't pay.
And the only way that spam can pay is if there are at least some suckers with
good money.
And the only way that there can be a reasonable number of suckers of such
monumental stupidity, who also have good money, is, as I see it, if intelligence
and good sense is not a requirement for succeeding in society.
But what is required, then, for succeeding in society?
Some of it is no doubt blind chance, but I believe that the main requirement is
simply one of fitting in, conformance. Thus, if my hypothesis is right, almost
any idiot can rise to any level of responsibility and social standing simply by
fitting in, conforming. This idiot then has good money and might be one of the
suckers responding to spam.
Cheers,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Sharing: member type deduction for member pointers (Alf's device?)
* Alf P. Steinbach /Usenet, on 17.07.2010 11:50:
[Cross-posted comp.lang.c++ and comp.lang.python]
[snip]
this occurred to me:
#define CPPY_GETSET_FORWARDERS( name ) \
::progrock::cppy::forwardersGetSet( \
&CppClass::name \
).themForwarders< &CppClass::name >()
Here forwardersGetSet is a templated function that via argument type
deduction produces an instance of a templated struct, thereby "knowing"
the member type, which struct in turn has a member function templated on
the member pointer, which the macro supplies *twice*, once as run-time
arg and once as compile-time.
That trick allowed uniform treatment of data and function member pointers. :-)
So, with cppy the complete noddy2 example from the docs (I'm sort of working my
way through the Python doc examples) now looks as shown below.
The "typelessness" of the 'first' and 'last' members is in order to recreate as
close as possible the noddy2 functionality, or, lack of it. Also, I haven't got
around to sort of backporting the 'Exposition' scheme to modules. So, exposure
of module functionality looks a little different from same for types, for now.
// The Python 3.1.1 docs' Noddy 2 example rewritten using cppy.
// Docs: "Extending and Embedding the Python Interpreter" §2.1.1
#include // PyWeakPtr, PyPtr, PyModule, PyClass
using namespace progrock;
namespace {
using namespace cppy;
struct Noddy
{
PyPtr first;
PyPtr last;
int number;
Noddy( PyWeakPtr pySelf, PyPtr args, PyPtr kwArgs )
: number( 0 )
{
devsupport::suppressUnusedWarning( pySelf );
PyWeakPtr pFirstName = 0;
PyWeakPtr pLastName = 0;
static char*kwlist[] = { "first", "last", "number", 0 };
::PyArg_ParseTupleAndKeywords(
args.rawPtr(), kwArgs.rawPtr(), "|OOi", kwlist,
pointerTo( pFirstName ), pointerTo( pLastName ), &number
)
>> Accept< IsNonZero >()
|| throwX( "Invalid args" );
if( pFirstName != 0 ) { first = pFirstName; }
if( pLastName != 0 ){ last = pLastName; }
}
PyPtr name()
{
(first != 0)
|| throwX( ::PyExc_AttributeError, "first" );
(last != 0)
|| throwX( ::PyExc_AttributeError, "last" );
return (PyString( first ) + L" " + PyString( last )).pyPtr();
}
};
struct NoddyPyClass: PyClass< Noddy >
{
NoddyPyClass( PyModule& m, PyString const& name, PyString const& doc )
: PyClass< Noddy >( m, name, doc, Exposition()
.method(
L"name",CPPY_GLUE( name ),
L"Return the name, combining the first and last name"
)
.attribute(
L"first", CPPY_GLUE( first ), L"first name"
)
.attribute(
L"last",CPPY_GLUE( last ), L"last name"
)
.attribute(
L"number", CPPY_GLUE( number ),L"noddy number"
)
)
{}
};
class NoddyModule: public PyModule
{
private:
NoddyPyClassnoddyPyClass_;
public:
NoddyModule()
: PyModule(
L"noddy2", L"Example module that creates an extension type." )
, noddyPyClass_( *this,
L"Noddy", L"A Noddy object has a name and a noddy number" )
{}
};
}// namespace
PyMODINIT_FUNC
PyInit_noddy2()
{
return cppy::safeInit< NoddyModule >();
}
I wonder if this is readable / self-documenting, or not?
Cheers,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Sharing: member type deduction for member pointers (Alf's device?)
[Cross-posted comp.lang.c++ and comp.lang.python]
Consider the following code, from an example usage of some C++ support for
Python I'm working on, "cppy":
struct Noddy
{
PyPtr first;
PyPtr last;
int number;
Noddy( PyWeakPtr pySelf, PyPtr args, PyPtr kwArgs )
: number( 0 )
{
// ... some initialization here
}
PyPtr name()
{
return (PyString( first ) + L" " + PyString( last )).pyPtr();
}
};
struct NoddyPyClass: PyClass< Noddy >
{
typedef PyClass< Noddy >Base;
NoddyPyClass( PyModule& m, PyString const& name, PyString const& doc )
: Base( m, name, doc,
Exposition()
.method(
L"name",CPPY_METHOD_FORWARDER( name ),
L"Return the name, combining the first and last name"
)
.attribute(
L"first", CPPY_GETSET_FORWARDERS( first ),
L"first name"
)
.attribute(
L"last",CPPY_GETSET_FORWARDERS( last ),
L"last name"
)
.attribute(
L"number", CPPY_GETSET_FORWARDERS( number ),
L"noddy number"
)
)
{}
};
Originally e.g.
CPPY_GETSET_FORWARDERS( number ),
had to be written as
CPPY_GETSET_FORWARDERS( int, number ),
because in order to use a compile time member pointer as template actual
argument, one needs to supply the member type as a separate template argument.
E.g. the template might look like
template< class Class, class MemberType, MemberType Class::*pMember >
struct ForwardersForGetAndSet
{
// Some definitions here, hardwiring that compile time member pointer!
};
Apparently there was no way around the user repeating explicitly the member type
that the compiler already knew about... It seemed akin to deducing the return
type of a function. Difficult in C++98 (although Boost does a fair job).
But then it seemed that I'm not totally senile yet, for this occurred to me:
#define CPPY_GETSET_FORWARDERS( name ) \
::progrock::cppy::forwardersGetSet( \
&CppClass::name \
).themForwarders< &CppClass::name >()
Here forwardersGetSet is a templated function that via argument type deduction
produces an instance of a templated struct, thereby "knowing" the member type,
which struct in turn has a member function templated on the member pointer,
which the macro supplies *twice*, once as run-time arg and once as compile-time.
He he.
Perhaps this trick is well-known already, but it was new to me, so! :-)
Cheers,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Nested loop not working
* Johann Spies, on 16.07.2010 16:34:
I am overlooking something stupid.
I have two files: one with keywords and another with data (one record per line).
I want to determine for each keyword which lines in the second file
contains that keyword.
The following code is not working. It loops through the second file
but only uses the first keyword in the first file.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
keywords = open("sleutelwoorde",'r')
data = open("sarua_marine_sleutelwoorde.csv",'r')
remove_quotes = re.compile('"')
for sw in keywords:
for r in data:
swc = remove_quotes('',sw)[:-1]
if swc in r.lower():
print swc + ' ---> ' + r
print swc
What am I missing?
For the inner loop, 'data' is an object that represents a file and keeps track
of a current read position of the file. The first execution of the loop moves
that read position all the way to the End Of the File, EOF. The second time this
loop is attempted, which would be for the second keyword, the 'data' object's
read position is already at end of file, and thus nothing's done.
One way to just make it work is to open and close the data file within the outer
loop. Actually with CPython it's automatically closed, as far as I can recall,
so you only need to reopen it, but this (if true) is less than completely
documented. This way is inefficient for small data set, but works.
In order to get a better handle on the general problem -- not the Python
technicalitities -- google up "KWIC", KeyWord In Context. It's a common exercise
problem given to first or second-year students. So I think there should be an
abundance of answers and discussion, although I haven't googled.
Cheers & hth.,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Cpp + Python: static data dynamic initialization in *nix shared lib?
* Hrvoje Niksic, on 14.07.2010 10:17:
"Alf P. Steinbach /Usenet" writes:
Also, things like the 'owned' option is just asking for trouble.
Isn't owned=true (or equivalent) a necessity when initializing from a
PyObject* returned by a function declared to return a "new reference"?
No, not quite.
Consider this signature (PyCXX):
String( PyObject* pyob, bool owned = false )
versus this (cppy):
PyString( PyPtr object )
With the first signature, every time you construct a String you have to remember
to (explicitly or implicitly) set the bool flag correctly depending on where the
actual argument value comes from.
With the second signature the /type/ of the actual argument decides,
automatically, so there's much less room for messing things up.
With the second signature, if the actual argument is a raw PyObject* pointer
then it's not owned, and the PyPtr formal argument doesn't increment the
reference count but just takes ownership. If the actual argument, on the other
hand, is a PyPtr, then the object is owned by the collection of PyPtr instances
that refer to the object, and the new formal argument PyPtr instance then
increments the reference count. In passing, if it should happen that the Python
community uses the word 'owned' in the opposite sense of what's conventional in
C++, then I guess & hope you'll figure out what I mean from this description.
PyPtr currently looks like this (complete code):
// progrock.cppy -- "C++ plus Python"
// A simple C++ framework for writing Python 3.x extensions.
//
// Copyright (c) Alf P. Steinbach, 2010.
#ifndef CPPY_PYPTR_H
#define CPPY_PYPTR_H
#include
//- Dependencies:
#include
#include
#include
//- Interface:
namespace progrock{ namespace cppy {
using namespace cppx;
enum DoAddRef {};
class PyPtr
{
private:
PyObject* p_;
public:
typedef cppy::DoAddRef DoAddRef;
PyPtr( PyObject* p = 0 ): p_( p ) {}
PyPtr( PyObject* p, DoAddRef ): p_( p )
{
assert( p != 0 ); Py_INCREF( p_ );
}
PyPtr( PyPtr const& other ): p_( other.p_ ) { Py_XINCREF( p_ ); }
~PyPtr() { Py_XDECREF( p_ ); }
void swapWith( PyPtr& other ) { std::swap( p_, other.p_ ); }
PyPtr& operator=( PyPtr other ) { swapWith( other ); return *this; }
PyObject* get() const { return p_; }
PyObject* release()
{
PyObject* const result = p_;
p_ = 0;
return result;
}
};
inline PyObject* withAddedRef( PyObject* p )
{
Py_INCREF( p );
return p;
}
inline PyObject* pyNoneRef()
{
return withAddedRef( Py_None );
}
} } // namespace progrock::cppy
#endif
As you can see at the end there, there is 'withAddedRef' and 'pyNoneRef', and
for that matter the 'DoAddRef' and the 'release()' method, for the cases where
PyPtr default handling doesn't quite cut it... :-)
How does your API deal with the distinction between new and borrowed
references?
See above.
There are some cases where it must be dealt with, but the main idea is to encode
that into /types/, instead of as context-dependent figure-it-out.
That's also the main idea of C++ as compared to C, to encode much more into
types, which helps both for compile time error detection and for automating
things (like above), which is Very Nice, but which also can make for enormous
waste of time trying to make the darned language do what you want it to do! :-)
Cheers & hth.,
- Alf
Disclaimer: it's late in the day/night for me, so the above explanation may have
big logic holes, mispelings and so on, but I hope it gives the idea.
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: floatref
* Steven D'Aprano, on 14.07.2010 06:31: Gary did the right thing by pointing out that the simple-sounding term "points to" is anything but simple, it depends on what you mean by pointing and pointers. Possibly you have a point here. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: floatref
* Gary Herron, on 14.07.2010 01:26:
On 07/13/2010 03:02 PM, Roald de Vries wrote:
Hi Gary,
On Jul 13, 2010, at 8:54 PM, Gary Herron wrote:
On 07/13/2010 10:26 AM, Roald de Vries wrote:
Hi all,
I have two objects that should both be able to alter a shared float.
So i need something like a mutable float object, or a float reference
object. Does anybody know if something like that exists? I know it's
not hard to build, but I have a feeling that there should be a
standard solution to it.
Roald
Huh? I must be missing something here. Isn't this what you use a
variable for:
Maybe I didn't explain well:
>>> shared_var = 1.0
>>> x.var = shared_var
>>> y.var = shared_var
>>> x.var = 2.0
>>> y.var
1.0
I wanted y.var and x.var to point to the same value, so that always
x.var == y.var. So that the last line becomes:
>>> y.var
2.0
Cheers, Roald
Please keep responses and further discussions on
list.python-l...@python.org
instead of using private emails.
Seconded. I didn't see that posting.
Python does not have pointers, so if I take your wording"y.var and x.var
to point to the same value" literally, then the answer is NO Python does
not do that.
This is just a terminological issue. Saying "Python does not have pointers" is
highly misleading in response to the OP's statement. It's easy enough to
understand what he means. E.g., in the Java language specification "pointer" has
a suitable meaning. And in general, the term "pointer" encompasses far more than
the concept of a C or Pascal pointer, e.g., in C++ it includes offset-like
things called member pointers. You can't represent or usefully think of a C++
member pointer as a C or Pascal pointer. It isn't useful on its own. So,
considering C++ and Java, the general pointer notion is something that refers,
however obliquely. You chose a specific meaning of "pointer" where the OP's
statement does not make sense, but presumably the OP is explaining his needs in
terms of a more general meaning of "pointer"; assuming anything else is silly.
However, Python does have references all over the place, so you can
achieve something similar in many ways.
If x and y in your example code are instances of a class, than look into
using a property for x.var and y.var. A property is a thing that looks
like an attribute, (that would be var in x.var and y.var), but which
really executes getter/setter code when accessed. That getter/setter
code would then access/set the value in shared_var:
shared_var = 123
class Thing(object):
def get_var(self):
return shared_var
def set_var(self, v):
global shared_var
shared_var = v
var = property(get_var, set_var)
x = Thing()
y = Thing()
print x.var, y.var # prints: 123 123
x.var = 99
print x.var, y.var # prints: 99 99
This code goes through hoops to /hide/ the fact of the sharing.
Rather, I'd make that as explicit as possible.
Like,
x = {"sharedVar": 123}
y = x
The one won't be surprised when changing x["sharedVar"] also changes
y["sharedVar"].
Cheers & hth.,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Cpp + Python: static data dynamic initialization in *nix shared lib?
* sturlamolden, on 13.07.2010 22:06: On 13 Jul, 21:39, "Alf P. Steinbach /Usenet" wrote: Thanks! It seems that SCXX does those things that I've been planning to do but haven't got around to (wrapping standard Python types), while what it doesn't do (abstracting away all those tables etc. and mapping Python calls to C++ calls) is what I've been working on. Which could be a Very Nice combination except that I'm assuming Py3, while SCXX seems to be Py2 only. :-( I'd suggest PyCXX instead. http://cxx.sourceforge.net SCXX is a tiny wrapper mostly used in scipy.weave to inline C++ in Python. Thanks. I looked up your URL, and PyCXX design goals seem to be much like what I'm doing, but different in some crucial ways. It's probably great, but it's not to my taste. E.g. the PyCXX String class interface: explicitString( PyObject *pyob, bool owned = false ) String( const Object &ob ) String() String( const char *latin1 ) String( const char *latin1, Py_ssize_t size ) String( const std::string &latin1 ) String( const std::string &v, const char *encoding, const char *error=NULL ) String( const char *s, const char *encoding, const char *error=NULL ) String( const char *s, Py_ssize_t len, const char *encoding, const char *error=NULL ) String &operator=( const Object &o ) String &operator=( PyObject *p ) String &operator=( const unicodestring &v ) size_type size() const size_type capacity() const unicodestring as_unicodestring() const std::string operator std::string() const String encode( const char *encoding, const char *error="strict" ) std::string as_std_string( const char *encoding=NULL, const char *error="strict" ) const In C++ the only way to portably specify a string literal with national characters, is as a wide string literal. Otherwise the result depends on the source code encoding. Yet PyCXX's String does not support wchar_t. Also, things like the 'owned' option is just asking for trouble. I chose this example because a Python string wrapper is the only object wrapper (apart from a general PyPtr) that I've implemented so far. The PyCXX string wrapper fails the design criterions of general usability (including in particular lack of wide char support) and safety (in particular the 'owned' option). And it's underdocumented, like, what encoding does the operator std::string() produce? The details don't matter though. I'm sure that PyCXX is Very Nice for its purpose, as is e.g. Boost.Python (which is Very Very Nice for its purpose). :-) Cheers, & thanks for the link, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Cpp + Python: static data dynamic initialization in *nix shared lib?
* sturlamolden, on 13.07.2010 22:03: On 9 Jul, 17:52, "Alf P. Steinbach /Usenet" wrote: For an extension module it seems that Python requires each routine to be defined as 'extern "C"'. That is strange. PyMethodDef is just a jump table. So why should 'extern "C"' matter? Formally because they're incompatible function pointer types. C++98 standard §7.5/1: "Two function types with different language linkages are distinct types even if they are otherwise identical". Add to that §7.5/4 "A linkage-specification shall occur only in namespace scope". And add to that §14-4 "A template, a template explicit specialization, or a class-template partial specialization shall not have C linkage." This means that formally correct code that generates callbacks by templating, is ruled out. In practice, 'extern "C"' matters for the jump tables because for those few compilers if any where it really matters (not just the compiler emitting a warning like reportedly Sun CC does), different linkage can imply different machine code level calling convention. For example, who's responsible for cleaning up the stack, the order in which arguments are pushed or which registers they're passed in, and so forth. Ignoring such matters your code gets into la-la land pretty fast, but, it's a different matter when one /understands/ this and places a requirement on the compiler. Good luck on re-inventing the wheel (you've probably heared about Swig, SIP, Boost.Python, PyCXX, scipy.weave and Cython...) Yes, I know Boost.Python in more detail and I've heard of all the rest except SIP, but then regarding SIP I really don't like QT (QT makes eminent sense in the context of Python, they're both essentially dynamically typed, but that means QT is not very nice as C++, plus there is the ugly preprocessor). And as you'd guess if you were not in silly ignoramus assertion-mode, I'm not reinventing the wheel. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Cpp + Python: static data dynamic initialization in *nix shared lib?
* Jonathan Lee, on 13.07.2010 16:41:
Problem (C) is outside the realm of the C++ standard, since the C++ standard
doesn't support shared libraries, and I've never actually used *nix shared
libraries so I don't /know/...
Is such dynamic initialization guaranteed?
Not guaranteed, though I think there's a combination of dlopen options
and gcc command line parameters that invoke this behavior. See the
second page of
http://www.linuxjournal.com/article/3687
about auto-registration.
Personally, though, it never worked for me :/
Ah, well. :-( Thanks for the info! OK, I'll just have to replace the
auto-registration with some C++ magic. For which I think I'll simply /require/
that the compiler supports mixing of C and C++ linkage, that is, that ...
#include
extern "C"
{
typedef int (*Callback)( int );
}
void foo( Callback f ) { std::cout << "foo!" << f( 42 ) << std::endl; }
int a( int ) { return 1; }
extern "C" int b( int ) { return 2; }
int main()
{
foo( a ); // Unholy Mix of C++ and C linkage, formally not OK.
foo( b ); // Should be OK with any compiler.
}
... compiles, and works.
Cheers, & thanks,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Cpp + Python: static data dynamic initialization in *nix shared lib?
* Robert Kern, on 13.07.2010 17:16: On 7/13/10 2:34 AM, Alf P. Steinbach /Usenet wrote: PS: You (the reader) may be wondering, why why why Yet Another Python/C++ binding? Well, because I had this great name for it, "pyni", unfortunately already in use. But cppy is very different from Boost: Boost is large, cppy is tiny; Boost has as main goal to expose arbitrary C++ code to Python, automating argument conversion etc., while with cppy your Python design is exposed to C++ with no enforced arg conversions and such; Boost relies on canned magic, difficult to subvert when it doesn't do what you want, while with cppy you are (or, so far, I am) in control; and I suspect that the Boost Python binding, relying on dynamic registries and stuff, is not all that efficient, while cppy is as efficient as using the Python C API to create an extension. And besides, cppy supports national characters in doc strings etc. And I'm Norwegian. So. :-) Note that Boost is not the only C++ binding out there. You may want to take a look at the old SCXX library, which appears to be similar in intent: http://davidf.sjsoft.com/mirrors/mcmillan-inc/scxx.html matplotlib uses it heavily, and their included copy may include some more recent bugfixes and enhancements: http://matplotlib.sourceforge.net/ Thanks! It seems that SCXX does those things that I've been planning to do but haven't got around to (wrapping standard Python types), while what it doesn't do (abstracting away all those tables etc. and mapping Python calls to C++ calls) is what I've been working on. Which could be a Very Nice combination except that I'm assuming Py3, while SCXX seems to be Py2 only. :-( Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Cpp + Python: static data dynamic initialization in *nix shared lib?
* geremy condra, on 09.07.2010 23:43: On Fri, Jul 9, 2010 at 5:22 PM, Ian Collins wrote: On 07/10/10 03:52 AM, Alf P. Steinbach /Usenet wrote: [Cross-posted comp.lang.python and comp.lang.c++] I lack experience with shared libraries in *nix and so I need to ask... This is about "cppy", some support for writing Python extensions in C++ that I just started on (some days ago almost known as "pynis" (not funny after all)). For an extension module it seems that Python requires each routine to be defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern "C"' and C++ linkage, using a routine declared as 'static' in a class as a C callback, formally they're two different kinds, and I seem to recall that /some/ C++ compiler balks at that kind of mixing unless specially instructed to allow it. Perhaps it was the Sun compiler? Yes, it will (correctly) issue a warning. As the is a bit OT, contact me directly and we can work through it. I have had similar fun and games adding PHP modules! I'd appreciate it if you'd either leave this on-list or cc me in on this, as I'm working through a similar issue. Well, we got no further, but I know of three solutions: A) Punting: just say that the compiler has to support C++/C function type mingling. -> Perhaps the practical solution, but formally unsafe. B) On the script side of things, delegate all calls to single Mother Of All C func downcaller that supplies as extra arg an id of the C++ function. -> Micro-level inefficient but easy to use and formally correct. C) Let the user define the C linkage function wrappers via macros. -> Efficient and formally correct but exposes ugly macro names. I chose (C). I believe Boost's Python binding uses (A), or perhaps (B). Cheers, - Alf PS: You (the reader) may be wondering, why why why Yet Another Python/C++ binding? Well, because I had this great name for it, "pyni", unfortunately already in use. But cppy is very different from Boost: Boost is large, cppy is tiny; Boost has as main goal to expose arbitrary C++ code to Python, automating argument conversion etc., while with cppy your Python design is exposed to C++ with no enforced arg conversions and such; Boost relies on canned magic, difficult to subvert when it doesn't do what you want, while with cppy you are (or, so far, I am) in control; and I suspect that the Boost Python binding, relying on dynamic registries and stuff, is not all that efficient, while cppy is as efficient as using the Python C API to create an extension. And besides, cppy supports national characters in doc strings etc. And I'm Norwegian. So. :-) -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* Steven D'Aprano, on 13.07.2010 01:34: On Mon, 12 Jul 2010 20:28:49 +0200, Alf P. Steinbach /Usenet wrote: As I see it it doesn't matter whether the implementation is CPython call frame slots or that mechanism called something else or a different mechanism called the same or a different mechanism called something different; what IMO matters is same semantics, that any assignment to a variable within a routine serves as a compile time declaration, creating that local variable in advance, unless, with Python 3.x., that name has been declared as a 'global' or 'nonlocal'. So, this is a possible point of disagreement. I say the semantics of local variable creation are part of the language definition, but I get the /impression/ that maybe you think it's CPython-specific, that e.g. def foo(): x x = 0 might not raise an unassigned variable exception with some conforming Python implementation, i.e. different effect for same code with different implementations, that this is at least /unspecified behavior/ in Python? Almost. I believe that "any assignment to a variable within a routine serves as a compile time declaration" is a promise of the language, but what an implementation does in response to that declaration is unspecified. So long as foo() raises NameError, or a subclass of it, it will be a conforming Python implementation. That's what Python 1.5 does, and I think if some competing implementation targeted 1.5 we'd still be happy to call it Python. (Although we might wonder about the author's sanity...) Implementations are free to subclass NameError, as CPython does with UnboundLocalError, but mustn't raise a completely unrelated error, or no error at all. E.g. I don't think it would be acceptable to implicitly create new names and bind them to some arbitrary default value. E.g. an implementation might do something like this: * when parsing the function, prior to compiling the byte-code, tag every name with a sigil representing whether it is local or non-local; * compile a single byte-code for name lookup; * when executing that instruction, if the name is tagged as a local, search only the local namespace, otherwise search the nonlocal, global and builtin namespaces; * when displaying names to the user (say, in tracebacks) suppress the sigil. Under this implementation, no variable actually exists until it is assigned to. This is equivalent to shifting the decision to use LOAD_FAST or LOAD_GLOBAL to runtime rather than compile time, so it would probably hurt performance rather than increase it, but it would still be a conforming implementation. But of course I'm not Guido, and he has the final word on what counts as acceptable behaviour. The 3.1.1 docs explicitly require UnboundLocalError (I just checked). So, at least for 3.x it is not an implementation detail. Anyway, your phrase "actually exist" presumably refers to storage allocation. That is an implementation detail. As a similar implementation scheme, a compiler can in certain cases detect that a variable is only assigned once, and substitute the value whereever that variable is used, not allocating any storage for it. This is the case in Python, in C++ and in almost any language. We don't start doubting the existence of variables in general (as some in the Python community do) on such grounds. More to the point of this sub-thread, it would be impossible to reason about things if one had to take into account the particular implementation's details at all times. Doing that renders most terms, including "exist", pretty meaningless and useless, since you would have to check whether each particular variable was optimized away or not: for the purpose of discussing existence in Python, what matters is portable effect. Summing up, how CPython implements the required semantics, is irrelevant. :-) Cheers from Norway, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* Steven D'Aprano, on 13.07.2010 01:50: On Mon, 12 Jul 2010 22:57:10 +0200, Alf P. Steinbach /Usenet wrote: Existence of a variable means, among other things, that * You can use the value, with guaranteed effect (either unassigned exception or you get a proper value): in particular, you won't be accessing a global if you're using the name of a local declared by a later assignment. That is too strong. Given the global code: x (where x doesn't exist in the global namespace, and therefore does not exist, as you agreed earlier) Python promises to raise NameError. By the above definition, this counts as "variable x exists". But surely that is undesirable -- that implies that *all* variables exist. Even $...@*@( is a variable that exists, as that is guaranteed to raise SyntaxError. Hm, I already answered someone else here committing that logic error. In one case an exception is generated by removing a variable. In the other case an exception is generated by adding a variable. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* Rami Chowdhury, on 13.07.2010 00:14: Perhaps I'm misunderstanding, but ... On Jul 12, 2010, at 13:57 , Alf P. Steinbach /Usenet wrote: Existence of a variable means, among other things, that * You can use the value, with guaranteed effect (either unassigned exception or you get a proper value) Surely by that definition any variable in any Python program "exists" -- you are guaranteed to get one of NameError, UnboundLocalError, or a value. That seems to argue away the meaning of the word entirely, and renders it not particularly useful. No, you're conflating non-existence (NameError) with accessing the value of an existing but unassigned variable (UnboundLocalError). It is like arguing that vacuum is the same as cement because sticking your head into it for ten minutes or so yields an effect -- you're dead -- that in many ways is just about the same. However, the tangible existence of cement means that one can pour it over your head, /adding/ the cement, while it's difficult to pour vacuum over your head; rather, for the latter effect one would need to /remove/ the air around your head. OK, the analogy halts a little. But I think you'll get it. Similarly, you can add a variable and thereby cause am UnboundLocalError, and you can remove a variable and thereby cause a NameError. How the Python implementation implements that is an implementation detail. In short, how CPython does things is completely irrelevant to the language's semantics, so you're conflating things here. As I'd understood the previous discussion, it is the CPython implementation that reserves local names and produces UnboundLocalErrors. The language semantics don't call for it, and another implementation might choose to handle function locals the same way as globals, through a namespace dictionary -- in which case the variable *wouldn't* exist in any way, shape, or form until it was assigned to. What am I getting wrong here? The bit about the language semantics not specifying the effect. From the 3.1.1 language reference §4.1: "When a name is not found at all, a NameError exception is raised. If the name refers to a local variable that has not been bound, a UnboundLocalError exception is raised. UnboundLocalError is a subclass of NameError." And it goes on to elaborate on that, a little later: "If a name binding operation occurs anywhere within a code block, all uses of the name within the block are treated as references to the current block. This can lead to errors when a name is used within a block before it is bound. This rule is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations." In short, Stephen D'Aprano's remarks were technically on the spot, while Rhodri James's follow up, that it seems influenced your response, was meaningless mumbo-jumbo with strong emphasis added repeatedly to denials of reality. This is the usual reaction of the religious when exposed to reality. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Standard distutils package problems with MSVC / lacking functionality?
I let the setup.py script talk: # 03_1__noddy from distutils.core import setup, Extension import distutils.ccompiler compilerName = distutils.ccompiler.get_default_compiler() options = [] if compilerName == "msvc": # * distutils sets warning level 3: # Overriding with warning level 4 generates command line warning D9025... # There's no good away around that, it needs fix/extension of distutils # or the config file(s) that distutil uses (perhaps that's a makefile?). options.append( "/W4" ) # Must be done in this script. # * distutils forgets to enable exception handling: options.append( "/EHsc" ) # Could be done via CL env. var. # * distutils forgets to enable RTTI: options.append( "/GR" ) # Could be done via CL env. var. # * distutils forgets to enable standard 'for' loop and 'wchar_t' type: options.append( "/Zc:forScope,wchar_t" )# Could be done via CL env. var. module1 = Extension( name = "noddy", sources = [ "noddy.cpp" ], extra_compile_args = options ) setup( name= "noddy", version = '1.0', description = 'This is a demo package', ext_modules = [module1] ) Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* Rhodri James, on 12.07.2010 22:19:
On Mon, 12 Jul 2010 13:56:38 +0100, bart.c wrote:
"Steven D'Aprano" wrote in
message news:4c3aedd5$0$28647$c3e8...@news.astraweb.com...
On Mon, 12 Jul 2010 09:48:04 +0100, bart.c wrote:
That's interesting. So in Python, you can't tell what local variables a
function has just by looking at it's code:
def foo(day):
if day=="Tuesday":
x=0
print ("Locals:",locals())
#foo("Monday")
Does foo() have 1 or 2 locals?
That's easy for CPython: it prepares two slots for variables, but only
creates one:
foo("Monday")
('Locals:', {'day': 'Monday'})
foo.func_code.co_varnames
('day', 'x')
foo.func_code.co_nlocals
2
So, the question is, is x a local variable or not? It's not in locals,
but the function clearly knows that it could be.
So Alf P.S. could be right; x exists, but Python pretends it doesn't
until it's assigned to.
CPython, not Python. And as Steven said, x *doesn't* exist. Allowance is
made by that specific implementation of the interpreter because x
*might* exist, but in this particular case it doesn't and a more dynamic
implementation might choose not to reserve a slot just in case. x is
created until it's actually used.
You are conflating existence with space allocation.
It's up to the implementation whether to allocate memory for the variable's
reference in any particular case where that memory isn't strictly required. This
is known as "optimization". Optimization depends on the implementation.
Existence of a variable means, among other things, that
* You can use the value, with guaranteed effect (either unassigned exception
or you get a proper value): in particular, you won't be accessing a global
if you're using the name of a local declared by a later assignment.
* You can assign to it.
How the Python implementation implements that is an implementation detail.
In short, how CPython does things is completely irrelevant to the language's
semantics, so you're conflating things here.
Cheers & hth.,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* Steven D'Aprano, on 12.07.2010 04:39: On Mon, 12 Jul 2010 03:12:10 +0200, Alf P. Steinbach /Usenet wrote: * MRAB, on 12.07.2010 00:37: [...] In Java a variable is declared and exists even before the first assignment to it. In Python a 'variable' isn't declared and won't exist until the first 'assignment' to it. That is a misconception. In Python a variable is declared by having an assignment to it, which for a local variable may be anywhere within a routine. Oh, I'm going to regret being sucked into this... In *CPython*, but not necessarily other implementations, variables which are local to a function are not kept in a dictionary-based namespace, but in slots in the code object (not to be confused with __slots__ used for classes). Python has STORE_FAST and LOAD_FAST byte-codes for accessing locals. This is intended as a speed, and possibly memory, optimization. I don't believe this is a requirement though, so implementations may not do this. It is true that the slot is created at compile time, and in *that sense*, local variables exist before they are bound. I'm not entirely convinced that this is the only sense that matters, but never mind. The error message given exposes this to the user: def f(): ... print x ... x = 1 ... f() Traceback (most recent call last): File "", line 1, in File "", line 2, in f UnboundLocalError: local variable 'x' referenced before assignment If you try this with a global, you get this: def f(): ... global x ... print x ... f() Traceback (most recent call last): File "", line 1, in File "", line 3, in f NameError: global name 'x' is not defined In this case, there's no doubt that global variable "x" doesn't exist at all -- there is no key "x" in the global namespace. Yes. What I disproved was the statement that every Python variable is created by the execution of an assignment. Some (the global ones) are. :-) It seems to me that "a slot to hold the variable is created for local variables" is an implementation detail, not a language feature. Yes. However, any Python implementation has to implement the same user level semantics in some way (note: use level semantics do not include "space reserved" for an unassigned variable, or even for all assigned variables since with single assignment a sufficiently smart compiler can optimize away the space, but user level semantics do include existence and resultant effect). As I see it it doesn't matter whether the implementation is CPython call frame slots or that mechanism called something else or a different mechanism called the same or a different mechanism called something different; what IMO matters is same semantics, that any assignment to a variable within a routine serves as a compile time declaration, creating that local variable in advance, unless, with Python 3.x., that name has been declared as a 'global' or 'nonlocal'. So, this is a possible point of disagreement. I say the semantics of local variable creation are part of the language definition, but I get the /impression/ that maybe you think it's CPython-specific, that e.g. def foo(): x x = 0 might not raise an unassigned variable exception with some conforming Python implementation, i.e. different effect for same code with different implementations, that this is at least /unspecified behavior/ in Python? CPython could easily hide the difference by changing the exception from UnboundLocalError to: NameError: local name 'x' does not exist and nobody would be any wiser. (Well, perhaps people who catch UnboundLocalError, but why would you do that?) I also note that UnboundLocalError is a subclass of NameError, so "variable exists but is not bound" is considered to be a special case of "variable doesn't exist" rather than a completely independent case. In that sense, I think I'm on solid ground to say that in Python variables don't exist until they are bound to a value, and leave it to pedants like you and I to mention that for CPython local variables have space reserved for them by the compiler before they are bound. He he. I wouldn't say "space reserved". That is an implementation detail. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* sturlamolden, on 12.07.2010 16:59: On 12 Jul, 07:51, "Alf P. Steinbach /Usenet" wrote: We're talking about defining a 'swap' routine that works on variables. I did not miss the point. One cannot make a swap function that rebinds its arguments in the calling stack frame. But a swap function can swap values, given that the type is not immutable: def swap(a,b): a[0],b[0] = b[0],a[0] a,b = [1],[2] swap(a,b) print a,b [2] [1] OK, that's missing the point. I thought you were joking. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Design questions for C++ support for Python extensions (cppy)
Hi.
With the current cppy code the Python 3.1.1 doc's spam example extension module
looks like this (actual working code):
#include
#include
using namespace progrock;
namespace {
class Spam: public cppy::Module
{
public:
Spam(): cppy::Module( L"spam", L"blåbærsyltetøy er blått" )
{}
PyObject* system( PyObject* args )
{
const char *command;
if( !PyArg_ParseTuple( args, "s", &command ) )
{
return NULL;
}
int const sts = ::system( command );
return Py_BuildValue( "i", sts );
}
};
}// namespace
CPPY_MODULE_CROUTINE( Spam, system, L"Execute a shell command" )
PyMODINIT_FUNC PyInit_spam()
{
return cppy::safeInit< Spam >();
}
Issues:
1. Wide string literals OK?
The basic Python API often requires UTF-8 encoded byte strings. With C++
source code encoded using e.g. Windows ANSI Western, string literals with
national characters such as Norwegian ÆØÅ then become gobbledegook or cause
outright failure. I balanced the hypothetical need for string literals with
national characters, versus perceived unnaturalness of wide string literals
for *nix developers, in favor of the former, i.e. L"wide string literals".
Related issue here: in Windows MinGW g++ can not compile utf-8 encoded
source with BOM, while MSVC requires a BOM in order to detect the encoding.
Is L"this" an acceptable decision if you were to use something like cppy?
2. Exception translation OK?
The code within the 'system' member routine could conceivably be reduced to
a single short line by adding some C++ support, but that would require use
of exceptions to report errors. Translating C++ exceptions to Python
exceptions does however add some overhead to every Python -> C++ call.
Currently I do this only for the module initialization code, but should it
be done for every exported method? Or perhaps as user choice? Benefit of
translation e.g. reducing 'system' above to sweet single line. Cost is
setup of try-block (negligible) and exception translation (inefficient).
3. Some unsafety OK?
In 'PyInit_spam' there may be a window of opportunity for client code to
Mess Things Up. Within the cppy::safeInit the C++ module object is created
and creates a Python module object, and if anything fails then the C++ side
frees the Python object. And after PyInit_spam has returned to Python the
cleanup responsibility resides with the Python interpreter: freeing the
Python module object causes the C++ object to be destroyed. But if say the
client code's equivalent of 'PyInit_spam' calls cppy::safeInit and just
discards the result and returns 0 (say) to Python, then it seems to not be
documented whether Python will free the Python module, i.e. possible leak.
4. Threading?
Is it necessary to make singletons/statics thread safe? Or does Python
ensure that no other threads execute while PyInit_spam is called? Can it
be called simultaneously by two or more threads?
5. Reload of interpreter?
My impression from the documentation is that finalization and reinit of the
interpreter is something that an application shouldn't really do, and that
an extension developer need not worry about that happening. Is it so?
Cheers,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* sturlamolden, on 12.07.2010 06:52: On 11 Jul, 21:37, "Alf P. Steinbach /Usenet" wrote: Oh, I wouldn't give that advice. It's meaningless mumbo-jumbo. Python works like Java in this respect, that's all; neither Java nor Python support 'swap'. x,y = y,x We're talking about defining a 'swap' routine that works on variables. Since Java/Python doesn't support pass by reference of variables it's not possible in these languages, i.e., you missed the point, or made a joke. :-) However, C# is very similar to Java, nearly all the way, except that in C# you can pass by reference. Remove that from C# and you have Java. Add that to Java and you have C#, roughly. No change in other aspects is needed. E.g. (ignore this if you've never heard about it, but it's a subtle point that you might be wondering about now) both Java and C# implement the definite assignment rule. I.e., there's nothing in the core semantics that prevents accessing/passing the variables by reference, although for Python and Java it could be a terminological nightmare, and for Python compounded to the n'th degree by the general confusion of a subset of the Python community about basic concepts. I don't know how C#'ers resolve the terminology... Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* MRAB, on 12.07.2010 04:09:
Alf P. Steinbach /Usenet wrote:
* MRAB, on 12.07.2010 00:37:
Alf P. Steinbach /Usenet wrote:
* Stephen Hansen, on 11.07.2010 21:00:
On 7/11/10 11:45 AM, wheres pythonmonks wrote:
Follow-up:
Is there a way to define compile-time constants in python and have
the
bytecode compiler optimize away expressions like:
if is_my_extra_debugging_on: print ...
when "is_my_extra_debugging" is set to false? I'd like to pay no
run-time penalty for such code when extra_debugging is disabled.
Any code wrapped in a __debug__ guard is utterly ommitted if you run
Python with the -O option. That, and asserts go away.
On #2: My point regarding the impossibility of writing the swap
function for ints is to explicitly understand that this isn't
possible, so as not to look for solutions along those lines when
trying to write python code.
Its impossible because Python's calling and namespace semantics simply
don't work like that. There's no references in the traditional sense,
because there's no variables-- boxes that you put values in. There's
just concrete objects. Objects are passed into the function and given
new names; that those objects have names in the enclosing scope is
something you don't know, can't access, and can't manipulate.. even
the
objects don't know what names they happen to be called.
Check out http://effbot.org/zone/call-by-object.htm
Oh, I wouldn't give that advice. It's meaningless mumbo-jumbo. Python
works like Java in this respect, that's all; neither Java nor Python
support 'swap'.
Of course there are variables, that's why the docs call them variables.
In Java a variable is declared and exists even before the first
assignment to it. In Python a 'variable' isn't declared and won't exist
until the first 'assignment' to it.
That is a misconception.
In Python a variable is declared by having an assignment to it, which
for a local variable may be anywhere within a routine.
If such a variable is used before it's been assigned to, then you get
an uninitialized variable exception. Clearly the variable must exist
in order for the exception to refer to it (not to mention the
exception occurring at all).
def foo():
print( blah )
blah = "this is both an assignment and a declaration causing it to exist"
foo()
Clearly when the exception is raised, referring to the variable, the
variable exists.
Contrary to your statement that is before the assignment.
However, as stated up-thread, I do not expect facts, logic or general
reasoning to have any effect whatsoever on such hard-core religious
beliefs. And I do not care whether I convince you or not. But I *do
not* want the religious subset of the community to succeed too much in
propagating nonsense idiot beliefs to newbies -- hence the concrete
example that any newbie can try.
How about this:
>>> def foo():
print("Before:", locals())
x = 0
print("After:", locals())
>>> foo()
Before: {}
After: {'x': 0}
How about it?
Note that you get the same result if you do
x = "blah"
def foo():
# print( x ) # Causes uninitialized variable exception here
print( "Before:", locals() )
x = 0
print( "After:", locals() )
However, if you remove the local assignment to x, then the out-commented print
statement will no longer cause an exception, it will then refer to the global.
The reason that it does throw an exception when you do have the local
assignment, is that the local x exists at that point. If it didn't exist it
could not have any effect. Things that don't exist generally have no effect,
except in the minds of the religious, like angels and so on.
On the basis of what locals() reports it should be OK to refer to the global x
as above. Judging by locals(), there's no local x that could get in the way. But
since it is not OK to refer to the global x, the result of locals() has nothing
to do with that: it doesn't tell you about the local x -- and no, the Python
interpreter does not look forward in time to see that it will appear.
In passing, I should perhaps have told you up front, your argument has nothing
substantial to do with the article you originally responded to, about the
semantics of variables. Your argument is the assertion that different languages
can't have similar or identical semantics for some feature. That's nonsense in
itself, plus, as you've seen, the difference that you focused on is not there,
and, third, what you do maintain is not there, doesn't exist, has a real effect.
Cheers & hth.,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* Stephen Hansen, on 12.07.2010 04:02: On 7/11/10 6:12 PM, Alf P. Steinbach /Usenet wrote: However, as stated up-thread, I do not expect facts, logic or general reasoning to have any effect whatsoever on such hard-core religious beliefs. Grow up, and/or get a grip, and/or get over yourself. Everyone who disagreed with you, disagreed with you with arguments, logic, facts, and reasoning. You disputed those facts, disagreed with the conclusions, but for you to then just dismiss people who don't agree with you as merely "religious", is childish. Exactly why I think you're wrong -- you're free to go re-read, I stand by my statements in this thread, and the others. The same arguments apply. Its not a religion, dear; my conclusions are not a matter of faith. That's all I have to say on this subject; the conversation has been had, at length (repeatedly). I swear, I'm just going to filter you and Rick out to /dev/null today and leave it at that at this rate. I'm getting worn out of these kinds of responses. Well, the above is flaming, which I predicted. The alleged facts etc. you're referring are just that, alleged, by you. In contrast, in debates among non-religious folks facts are /presented/, like I've done in this thread, e.g. concrete code, instead of like you alleging that facts have been presented, hinting about things, and so on -- it's pathetic. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* MRAB, on 12.07.2010 00:37: Alf P. Steinbach /Usenet wrote: * Stephen Hansen, on 11.07.2010 21:00: On 7/11/10 11:45 AM, wheres pythonmonks wrote: Follow-up: Is there a way to define compile-time constants in python and have the bytecode compiler optimize away expressions like: if is_my_extra_debugging_on: print ... when "is_my_extra_debugging" is set to false? I'd like to pay no run-time penalty for such code when extra_debugging is disabled. Any code wrapped in a __debug__ guard is utterly ommitted if you run Python with the -O option. That, and asserts go away. On #2: My point regarding the impossibility of writing the swap function for ints is to explicitly understand that this isn't possible, so as not to look for solutions along those lines when trying to write python code. Its impossible because Python's calling and namespace semantics simply don't work like that. There's no references in the traditional sense, because there's no variables-- boxes that you put values in. There's just concrete objects. Objects are passed into the function and given new names; that those objects have names in the enclosing scope is something you don't know, can't access, and can't manipulate.. even the objects don't know what names they happen to be called. Check out http://effbot.org/zone/call-by-object.htm Oh, I wouldn't give that advice. It's meaningless mumbo-jumbo. Python works like Java in this respect, that's all; neither Java nor Python support 'swap'. Of course there are variables, that's why the docs call them variables. In Java a variable is declared and exists even before the first assignment to it. In Python a 'variable' isn't declared and won't exist until the first 'assignment' to it. That is a misconception. In Python a variable is declared by having an assignment to it, which for a local variable may be anywhere within a routine. If such a variable is used before it's been assigned to, then you get an uninitialized variable exception. Clearly the variable must exist in order for the exception to refer to it (not to mention the exception occurring at all). def foo(): print( blah ) blah = "this is both an assignment and a declaration causing it to exist" foo() Clearly when the exception is raised, referring to the variable, the variable exists. Contrary to your statement that is before the assignment. However, as stated up-thread, I do not expect facts, logic or general reasoning to have any effect whatsoever on such hard-core religious beliefs. And I do not care whether I convince you or not. But I *do not* want the religious subset of the community to succeed too much in propagating nonsense idiot beliefs to newbies -- hence the concrete example that any newbie can try. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy questions from a python beginner
* Stephen Hansen, on 11.07.2010 21:00: On 7/11/10 11:45 AM, wheres pythonmonks wrote: Follow-up: Is there a way to define compile-time constants in python and have the bytecode compiler optimize away expressions like: if is_my_extra_debugging_on: print ... when "is_my_extra_debugging" is set to false? I'd like to pay no run-time penalty for such code when extra_debugging is disabled. Any code wrapped in a __debug__ guard is utterly ommitted if you run Python with the -O option. That, and asserts go away. On #2: My point regarding the impossibility of writing the swap function for ints is to explicitly understand that this isn't possible, so as not to look for solutions along those lines when trying to write python code. Its impossible because Python's calling and namespace semantics simply don't work like that. There's no references in the traditional sense, because there's no variables-- boxes that you put values in. There's just concrete objects. Objects are passed into the function and given new names; that those objects have names in the enclosing scope is something you don't know, can't access, and can't manipulate.. even the objects don't know what names they happen to be called. Check out http://effbot.org/zone/call-by-object.htm Oh, I wouldn't give that advice. It's meaningless mumbo-jumbo. Python works like Java in this respect, that's all; neither Java nor Python support 'swap'. Of course there are variables, that's why the docs call them variables. We've had this discussion before and I know from that that it is a religious issue with a small subset of the Python community, where reason, facts, logic does not apply and is not even recognized as such. So be it. So I'm not out to convince you or other of that sub-community, or trying to reason with you folks on this issue (futile, and generates flames pretty fast), but I do not want newbies brainwashed into that non-reasoning nonsense pure faith religion. For what it's worth, I'm sure that the effbot.org author, whose pages are otherwise quite technically meaningful & useful, in this case, after the flame war with some Java folks, decided that technical accuracy just wasn't worth it. So, I believe, he punted, which is an eminently rational choice when one's goals require acceptance in a society dominated by a religious clique. And just as I'm not out to engage you in any debate on this issue (futile), neither am I calling you irrational. Perhaps your choice is the same as that author's. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Naming Conventions, Where's the Convention Waldo?
* rantingrick, on 11.07.2010 09:26: Another source of asininity seems to be the naming conventions of the Python language proper! True/False start with an upper case and i applaud this. However str, list, tuple, int, float --need i go on...?-- start with lowercase. Q: Well what the hell is your problem Rick. Who cares right? WRONG, I tell you what my problem is. Now i cannot "wisely" use variables like... str="this is a string" list = [1,2,3] def make_random_objs(range=10) def show_message(str) int = 12 If we would have adopted proper naming conventions from dios numero uno all this nonsense would be rectified! Str, Float, List, Range, etc, etc. You think Python 3000 was a hump to climb over just wait for Python 4000. Just thoughts. Just do Str = str List = list Float = float and so on in module "myBasicTypes", and import that. :-) Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: integer >= 1 == True and integer.0 == False is bad, bad, bad!!!
* Stephen Hansen, on 11.07.2010 09:19: On 7/10/10 11:50 PM, rantingrick wrote: It was a typo not an on purpose misspelling If this had been the first time, perhaps. If you had not in *numerous* previous times spelled my name correctly, perhaps. If it were at all possible for "f" to be a typo of "ph", perhaps. It is a natural mistake to make in some languages. E.g. in Norwegian the Devil can be spelled Faen or Fanden (modern) or Phanden (old-fashioned, no longer in dictionaries but still used to sort of tone down the expression). It's even there in English, like "file" and "philosophy". So it's an error committed not by the limbic system but by a slightly higher level sound-to-text translator brain circuit. The text is generated from how the word sounds in one's head. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: integer >= 1 == True and integer.0 == False is bad, bad, bad!!!
* rantingrick, on 11.07.2010 08:50: On Jul 11, 1:22 am, Stephen Hansen wrote: Utter nonsense. No one does that unless they are coming from C or some other language without a True/False and don't know about it, or if they are using a codebase which is supporting a very old version of Python before True or False were introduced. Ah yes, when nothing else seems to work fall back to you default programming... FUD and ad hominem attacks I agree with Stephen, but for a different reason: that given desirability of implicit conversion to bool for some elementary types, then for uniformity there should be such conversion for all of them (and AFAIK there is), and given that, the rule should be the same, namely that default value of each type bool's to False, and other values to True, and so it is. The OP should simply represent "not found" as e.g. integer -1 instead of as a value of a different type. And write e.g. not_found = -1 ... if choiceIdx1 == choiceIdx2 == not_found: bah, none of them elif choice2Idx == not_found: use choice 1 elif choice1Idx == not_found: use choice 2 else: determine bestest choice Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: any issues with long running python apps?
* John Nagle, on 10.07.2010 20:54: On 7/9/2010 12:13 PM, Les Schaffer wrote: i have been asked to guarantee that a proposed Python application will run continuously under MS Windows for two months time. And i am looking to know what i don't know. The app would read instrument data from a serial port, If the device you're listening to is read-only, and you're just listening, make a cable to feed the serial data into two machines, and have them both log it. Put them on separate UPSs and in a place where nobody can knock them over or mess with them. "The Ramans do everything in triplicate" - Old jungle proverb Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Not-quite-the-module-name qualified names in extension modules? What?
Hi. I built the [xxmodule.c] from the source distribution, as suggested by the Python 3.1.1 docs. I named this [xx.pyd], as I believed the module name was just "xx". Indeed importing xx works fine, but when I do help(xx) I get ... >>> help( xx ) Help on module xx: NAME xx - This is a template module just for instruction. FILE c:\projects\progrock\lib\progrock\cppy_dev\examples\02_xx_apilevel\xx.pyd CLASSES builtins.Exception(builtins.BaseException) error builtins.object xxmodule.Null builtins.str(builtins.object) xxmodule.Str class Null(builtins.object) | Methods defined here: | ... with the name "xxmodule" somehow in there as qualification. Checking the standard "csv" module I similarly get ... CLASSES builtins.Exception(builtins.BaseException) _csv.Error builtins.object Dialect excel excel_tab DictReader DictWriter Sniffer ... with the name "_csv" in there. And I'm pretty sure that these not-quite-the-module-name names stem from the literal specification of names in the C code in the extension module, assuming that the csv module is indeed a C extension module. Is it really necessary to qualify names in the C code? And can it do harm when such name qualification is not using the final module name but instead something like "xxmodule" or "_csv"? More to the point, what's the point? Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Cpp + Python: static data dynamic initialization in *nix shared lib?
* Ian Collins, on 09.07.2010 23:22: On 07/10/10 03:52 AM, Alf P. Steinbach /Usenet wrote: [Cross-posted comp.lang.python and comp.lang.c++] I lack experience with shared libraries in *nix and so I need to ask... This is about "cppy", some support for writing Python extensions in C++ that I just started on (some days ago almost known as "pynis" (not funny after all)). For an extension module it seems that Python requires each routine to be defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern "C"' and C++ linkage, using a routine declared as 'static' in a class as a C callback, formally they're two different kinds, and I seem to recall that /some/ C++ compiler balks at that kind of mixing unless specially instructed to allow it. Perhaps it was the Sun compiler? Yes, it will (correctly) issue a warning. As the is a bit OT, contact me directly and we can work through it. I have had similar fun and games adding PHP modules! Thanks. I'm mailing you a zip with the code... The question, of course, whether it works in *nix. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Hello
* Dani Valverde, on 09.07.2010 18:31: Hello! I am new to python and pretty new to programming (I have some expertise wit R statistical programming language). I am just starting, so my questions may be a little bit stupid. Can anyone suggest a good editor for python? Cheers! If you're working in Windows the Notepad++ and PSPad and old Crimson Editor (all free) all work nicely and are reasonably light-weight. Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Cpp + Python: static data dynamic initialization in *nix shared lib?
[Cross-posted comp.lang.python and comp.lang.c++]
I lack experience with shared libraries in *nix and so I need to ask...
This is about "cppy", some support for writing Python extensions in C++ that I
just started on (some days ago almost known as "pynis" (not funny after all)).
For an extension module it seems that Python requires each routine to be defined
as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern "C"' and C++
linkage, using a routine declared as 'static' in a class as a C callback,
formally they're two different kinds, and I seem to recall that /some/ C++
compiler balks at that kind of mixing unless specially instructed to allow it.
Perhaps it was the Sun compiler?
Anyway, to be formally correct I cannot generate the required C routines via
templating, and I ended up using macros that the user must explicitly invoke,
like, here the Py doc's first extension module example recoded using cppy,
--
#include
#include
using namespace progrock;
class Spam: public cppy::Module
{
public:
Spam(): cppy::Module( "spam" )
{
setDocString( L"blåbærsyltetøy er blått" );
}
PyObject* system( PyObject* args )
{
const char *command;
int sts;
if( !PyArg_ParseTuple( args, "s", &command ) )
{
return NULL;
}
sts = ::system( command );
return Py_BuildValue( "i", sts );
}
};
CPPY_MODULE_CROUTINE( Spam, system, "Execute a shell command" )
PyMODINIT_FUNC PyInit_spam()
{
return cppy::init< Spam >();
}
--
It works in Windows.
But here CPPY_MODULE_CROUTINE does three things:
A Defining the 'extern "C"' routine.
I cannot think of any problem here.
B Defining installation data for that routine.
Possible problem: initializing a static with address of routine?
C -> Adding that install data record into a linked list!
Possible problem: are dynamic initialization actions guaranteed
to be performed in *nix shared library?
Problem (C) is outside the realm of the C++ standard, since the C++ standard
doesn't support shared libraries, and I've never actually used *nix shared
libraries so I don't /know/...
Is such dynamic initialization guaranteed?
For completeness, the macro definition (the 0 in there is a list next-pointer):
#define CPPY_MODULE_CROUTINE_DEF( cppClassName, name ) \
extern "C" \
static PyObject* cppClassName##_##name( PyObject*, PyObject* args ) \
{ \
return ::progrock::cppy::module().name( args ); \
}
#define CPPY_MODULE_CROUTINE_INSTALLDATA( cppClassName, name, docString ) \
static ::progrock::cppy::detail::ModuleRoutineDescriptor\
cppClassName##_##name##_descriptor = { \
0, \
#name, \
docString, \
&cppClassName##_##name \
}; \
\
static bool cppClassName##_##name##_descriptor_installed = \
::progrock::cppy::detail::addToList< cppClassName >(\
cppClassName##_##name##_descriptor \
);
#define CPPY_MODULE_CROUTINE( cppClassName, name, docString ) \
CPPY_MODULE_CROUTINE_DEF( cppClassName, name ) \
CPPY_MODULE_CROUTINE_INSTALLDATA( cppClassName, name, docString )
TIA.,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: How do I add method dynamically to module using C API?
* Martin v. Loewis, on 08.07.2010 09:13:
I tried (1) adding a __del__, but no dice, I guess
because it wasn't really an object method but just a free function in a
module; and (2) the m_free callback in the module definition structure,
but it was not called.
m_free will be called if the module object gets deallocated. So if
m_free really isn't called, the module got never deallocated.
Thanks again. I don't know what I did wrong. Now it's called. :-)
But I wasted much time googling to try to find out the /responsibilities/ of the
m_free callback, and what its void* argument was. E.g., should it deallocate the
module object, and if so, via what deallocation routine? I found some info, but
even your PEP, otherwise clear, was silent about this fine point.
Finally I looked at the source code that invokes it and found that it has no
responsibilities whatsoever, just a use-as-you-wish finalization callback. Nice!
But I think that could be more clear in the docs...
Code, for those who might be interested:
// progrock.cppy -- "C++ plus Python"
// A simple C++ framework for writing Python 3.x extensions.
//
// Copyright (C) Alf P. Steinbach, 2010.
#ifndef CPPY_MODULE_H
#define CPPY_MODULE_H
#include
//- Dependencies:
#include "Ptr.h"
#include
#include
//- Interface:
namespace progrock{ namespace cppy {
namespace detail {
struct InstanceData
{
std::list< PyMethodDef >methodDefs;
};
inline void* instanceMemoryOf( PyObject* pObject )
{
void* const result = PyModule_GetState( pObject );
assert( result != 0 );
return result;
}
inline InstanceData*& instanceDataPtrOf( PyObject* pObject )
{
return *reinterpret_cast< InstanceData** >(
instanceMemoryOf( pObject )
);
}
inline void on_freeModule( void* p )
{
PyObject* const pModule = reinterpret_cast< PyObject* >( p );
InstanceData* const pData = instanceDataPtrOf( pModule );
delete pData;
printf( "Deallocated!\n" ); // TODO: Remove.
}
inline PyModuleDef* moduleDefPtr()
{
static PyMethodDef methodDefs[] = {
//...
//{ "system", &pyni_system, METH_VARARGS, "Execute a shell
command." },
//{ "__del__", &onModuleDestroy, METH_NOARGS, "Destructor" },
//...
{ NULL, NULL, 0, NULL } // Sentinel
};
static PyModuleDef moduleDef = {
PyModuleDef_HEAD_INIT,
"cppy", // name of module
NULL, // m_doc, // module documentation in UTF-8
sizeof(void*), // size of per-interpreter state of the
module,
//or -1 if the module keeps state in global
variables.
methodDefs,
NULL, // m_reload
NULL, // m_traverse
NULL, // m_clear
&on_freeModule // m_free
};
return &moduleDef;
}
}
class Module
{
private:
Ptr p_;
detail::InstanceData* data_;
detail::InstanceData*& instanceDataPtr() const
{
return detail::instanceDataPtrOf( p_.get() );
}
public:
Module()
: p_( ::PyModule_Create( detail::moduleDefPtr() ) )
, data_( 0 )
{
assert( detail::moduleDefPtr()->m_size == sizeof( void* ) );
(p_.get() != 0) || cppx::throwX( "Module::: failed" );
instanceDataPtr() = data_ = new detail::InstanceData();
}
PyObject* rawPtr() const{ return p_.get(); }
PyObject* release() { return p_.release(); }
void setDocString( wchar_t const s[] )
{
Ptr const v = ::PyUnicode_FromWideChar( s, -1 );
(v.get() != 0)
|| cppx::throwX( "Module::setDocString: PyUnicode_FromWideChar
failed" );
::PyObject_SetAttrString( p_.get(), "__doc__", v.get() )
>> cppx::is( cppx::notMinusOne )
|| cppx::throwX( "Module::setDocString: PyObject_SetAttrString
failed" );
}
void addRoutine( char const name[], PyCFunction f, char const doc[] =
"" )
{
PyMethodDef const defData = { name, f, METH_VARARGS, doc };
data_->methodDefs.push_back( defData );
try
{
PyMethodDef* const pDef= &data_->methodDefs.back();
Ptr const pyName = ::PyUnicode_FromString( name );
(pyName.get() != 0)
|| cppx::throwX( "Module::addRoutine: PyUnicode_FromString
faile
Re: How do I add method dynamically to module using C API?
* Martin v. Loewis, on 08.07.2010 07:23:
And since things work for a single method when I declare 'def' as
'static', I suspect that means that the function object created by
PyCFunction_NewEx holds on to a pointer to the PyMethodDef structure?
Correct; it doesn't make a copy of the struct. So when you want the
function object to outlive the setRoutine call, you need to allocate
the PyMethodDef on the heap.
Thanks!
That's the direction I tentatively had started to investigate.
But second problem now is cleanup: I'd like to deallocate when the module is
freed. I tried (1) adding a __del__, but no dice, I guess because it wasn't
really an object method but just a free function in a module; and (2) the m_free
callback in the module definition structure, but it was not called.
Perhaps I don't need to to clean up?
Anyway, current looks like this:
// progrock.cppy -- "C++ plus Python"
// A simple C++ framework for writing Python 3.x extensions.
//
// Copyright (C) Alf P. Steinbach, 2010.
#ifndef CPPY_MODULE_H
#define CPPY_MODULE_H
#include
//- Dependencies:
#include "Ptr.h"
#include
#include
//- Interface:
namespace progrock{ namespace cppy {
namespace detail {
inline PyModuleDef* moduleDefPtr()
{
static PyMethodDef methodDefs[] = {
//...
//{ "system", &pyni_system, METH_VARARGS, "Execute a shell
command." },
//{ "__del__", &onModuleDestroy, METH_NOARGS, "Destructor" },
//...
{ NULL, NULL, 0, NULL } // Sentinel
};
static PyModuleDef moduleDef = {
PyModuleDef_HEAD_INIT,
"cppy", // name of module
NULL, // m_doc, // module documentation in UTF-8
sizeof(void*), // size of per-interpreter state of the
module,
//or -1 if the module keeps state in global
variables.
methodDefs,
NULL, // m_reload
NULL, // m_traverse
NULL, // m_clear
NULL// m_free
};
return &moduleDef;
}
}
class Module
{
private:
struct InstanceData
{
std::list< PyMethodDef >methodDefs;
};
void* instanceMemory() const
{
void* const result = PyModule_GetState( p_.get() );
assert( result != 0 );
return result;
}
InstanceData*& instanceDataPtr() const
{
return *reinterpret_cast< InstanceData** >( instanceMemory() );
}
Ptr p_;
InstanceData* data_;
public:
Module()
: p_( ::PyModule_Create( detail::moduleDefPtr() ) )
{
assert( def.m_size == sizeof( void* ) );
(p_.get() != 0) || cppx::throwX( "Module::: failed" );
instanceDataPtr() = data_ = new InstanceData();
}
PyObject* rawPtr() const{ return p_.get(); }
PyObject* release() { return p_.release(); }
void setDocString( wchar_t const s[] )
{
Ptr const v = ::PyUnicode_FromWideChar( s, -1 );
(v.get() != 0)
|| cppx::throwX( "Module::setDocString: PyUnicode_FromWideChar
failed" );
::PyObject_SetAttrString( p_.get(), "__doc__", v.get() )
>> cppx::is( cppx::notMinusOne )
|| cppx::throwX( "Module::setDocString: PyObject_SetAttrString
failed" );
}
void addRoutine( char const name[], PyCFunction f, char const doc[] =
"" )
{
PyMethodDef const defData = { name, f, METH_VARARGS, doc };
data_->methodDefs.push_back( defData );
try
{
PyMethodDef*pDef= &data_->methodDefs.back();
Ptr const pyName = ::PyUnicode_FromString( name );
Ptr r = ::PyCFunction_NewEx( pDef, p_.get(),
pyName.get());
::PyModule_AddObject( p_.get(), name, r.release() )
>> cppx::is( cppx::notMinusOne )
|| cppx::throwX( "Module::addRoutine: PyModule_AddObject
failed" );
}
catch( ... )
{
data_->methodDefs.pop_back();
throw;
}
}
};
} } // namespace progrock::cppy
#endif
I changed the module name from "pyni*" to "cppy"... ;-)
Cheers & thanks!, but how to clean up, or must I?
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
How do I add method dynamically to module using C API?
The code below, very much work in progress, just trying things, is C++.
Sorry about the formatting, I had to reformat manually for this posting:
class Module
{
private:
Ptr p_;
public:
Module( PyModuleDef const& def )
: p_( ::PyModule_Create( const_cast< PyModuleDef* >( &def ) ) )
{
(p_.get() != 0) || cppx::throwX( "Module::: failed" );
}
PyObject* rawPtr() const{ return p_.get(); }
PyObject* release() { return p_.release(); }
void setDocString( wchar_t const s[] )
{
Ptr const v = ::PyUnicode_FromWideChar( s, -1 );
(v.get() != 0)
|| cppx::throwX( "PyUnicode_FromWideChar failed" );
int const _ = ::PyObject_SetAttrString(
p_.get(), "__doc__", v.get()
);
(_ != -1 )
|| cppx::throwX( "PyObject_SetAttrString failed" );
}
void setRoutine(
char const name[], PyCFunction f, char const doc[] = ""
)
{
PyMethodDef def = { name, f, METH_VARARGS, doc };
Ptr const pyName = ::PyUnicode_FromString( name );
Ptr r = ::PyCFunction_NewEx(
&def, p_.get(), pyName.get()
);
int const _ = ::PyModule_AddObject( p_.get(), name, r.release() );
(_ != -1 )
|| cppx::throwX( "PyModule_AddObject failed" );
}
};
Problem: when a routine installed by 'setRoutine' above is called from Python,
then it fails with e.g. "SystemError: Bad call flags in PyCFunction_Call.
METH_OLDARGS is no longer supported!"
And since things work for a single method when I declare 'def' as 'static', I
suspect that means that the function object created by PyCFunction_NewEx holds
on to a pointer to the PyMethodDef structure?
I'm unable to find documentation of PyCFunction_NewEx, and more criticially, I'm
unable to find any documented way to turn a C or C++ function into a Python
function object?
Cheers,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Argh! Name collision!
* Alf P. Steinbach /Usenet, on 08.07.2010 01:47:
enum DoAddRef { doAddRef };
class Ptr
{
private:
PyObject* p_;
public:
Ptr( PyObject* p = 0 ): p_( p )
{}
Ptr( PyObject* p, DoAddRef ): p_( p )
{
assert( p != 0 );
Py_INCREF( p_ );
}
Ptr( Ptr const& other ): p_( other.p_ )
{
Py_XINCREF( p_ );
}
~Ptr()
{
Py_XDECREF( p_ );
}
void swapWith( Ptr& other ) { std::swap( p_, other.p_ ); }
Ptr& operator=( Ptr other ) { swapWith( other ); return *this; }
PyObject* get() const { return p_; }
PyObject* release()
{
PyObject* const result = p_;
Py_XDECREF( p_ );
Hark. This Py_XDECREF shouldn't be there, I don't know how it got there. The
whole point of 'release', with conventional semantics, is to /not/ decrement the
reference count.
p_ = 0;
return result;
}
};
Sorry for posting unfinished code,
- Alf
PS: "pyni" was a good name. But in use! When I thought about adding the "s" as
disambiguation I thought the pure shock value of that was funny in a way, but
now it doesn't seem funny. Is "pytes" (Python Extension Support) a good name?
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Argh! Name collision!
* rantingrick, on 07.07.2010 07:42:
On Jul 6, 9:11 pm, "Alf P. Steinbach /Usenet" wrote:
"pyni"! Pronounced like "tiny"! Yay!
hmm, how's about an alternate spelling... "pyknee", or "pynee", or
"pynie" ... considering those are not taken either?
Hm, for pure shock value I think I'll use the acronym PYthon Native Interface
Support.
pynis! :-)
A set of C++ classes to ease the writing of extensions.
Like,
// progrock.pynis -- "Python Native Interface Support"
// A simple C++ framework for writing Python 3.x extensions.
//
// Copyright (C) Alf P. Steinbach, 2010.
#ifndef PYNIS_PTR_H
#define PYNIS_PTR_H
#include
//- Dependencies:
#include
#include
#include
//- Interface:
namespace progrock{ namespace pynis {
enum DoAddRef { doAddRef };
class Ptr
{
private:
PyObject* p_;
public:
Ptr( PyObject* p = 0 ): p_( p )
{}
Ptr( PyObject* p, DoAddRef ): p_( p )
{
assert( p != 0 );
Py_INCREF( p_ );
}
Ptr( Ptr const& other ): p_( other.p_ )
{
Py_XINCREF( p_ );
}
~Ptr()
{
Py_XDECREF( p_ );
}
void swapWith( Ptr& other ) { std::swap( p_, other.p_ ); }
Ptr& operator=( Ptr other ) { swapWith( other ); return *this; }
PyObject* get() const { return p_; }
PyObject* release()
{
PyObject* const result = p_;
Py_XDECREF( p_ );
p_ = 0;
return result;
}
};
} } // namespace progrock::pynis
#endif
Cheers,
- Alf (shocked)
PS: Darn, forgot to google it. But I think it's unlikely the name's already in
use!
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* Alf P. Steinbach /Usenet, on 07.07.2010 23:19: However developing an extension with MSVC 10 the extension will use the 10.0 CRT, which is not necessarily present on the end user's system. As I see it there are five solutions with different trade-offs: A Already having Visual Studio 2008 (MSVC 9.0), or coughing up the money for an MSDN subscription, or visiting trade shows, so as to obtain that compiler version. -> Not an option for everybody. B Linking the CRT statically. -> Increased size, problems with CRT state such as file descriptors. C Linking the CRT dynamically and bundling the MSVC redistributables with the extension. -> Even more increased size for the download, but smaller total footprint for extensions in sum; same CRT state problems. D Linking the CRT dynamically and providing an optional download and install of the redistributables if they're not present. This would best be done with some support from the Python installation machinery. -> Small nice size for extensions, still same CRT state problems. E As D + a new compiler-independent native code interface that does not carry dependencies on CRT state such as file descriptors, like JNI. -> Really huge effort, and cannot be applied until some new Python version. And I think the clue here is that the CRT state problems can be avoided by careful coding. Hence, for those who cannot do A I think B is a realistic practical option, and D would be nice... Wait... F Possibly, as the docs say, "Developer Studio will throw in a lot of import libraries that you do not really need, adding about 100K to your executable. To get rid of them, use the Project Settings dialog, Link tab, to specify ignore default libraries. Add the correct msvcrtxx.lib to the list of libraries." Can anyone confirm whether this works in practice with MSVC 10? Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* Christian Heimes, on 07.07.2010 22:47: The main problem that the required MSVC redistributables are not necessarily present on the end user's system. It's not a problem for Python anymore. It took a while to sort all problems out. Martin and other developers have successfully figured out how to install the CRT for system wide and local user installations. A system wide installation installs the CRT in the side by side cache (WinSxS). A local installation keeps the msvcrt90.dll and the Microsoft.VC90.CRT.manifest next to the python.exe. Python extensions no longer embed a manifest so they share the CRT from the python.exe process. In order to ship a standalone exe you have to keep the CRT next to your exe. This should work for py2exe binaries as well. At our company we install our application stack entirely from subversion including Python 2.6.5, Sun JRE and lots of other stuff. This works perfectly fine for us even for servers without the MSVCRT redistributable. I think you're talking about a different problem. The CRT installed along with CPython works for extensions using the MSVC 9.0 CRT. However developing an extension with MSVC 10 the extension will use the 10.0 CRT, which is not necessarily present on the end user's system. As I see it there are five solutions with different trade-offs: A Already having Visual Studio 2008 (MSVC 9.0), or coughing up the money for an MSDN subscription, or visiting trade shows, so as to obtain that compiler version. -> Not an option for everybody. B Linking the CRT statically. -> Increased size, problems with CRT state such as file descriptors. C Linking the CRT dynamically and bundling the MSVC redistributables with the extension. -> Even more increased size for the download, but smaller total footprint for extensions in sum; same CRT state problems. D Linking the CRT dynamically and providing an optional download and install of the redistributables if they're not present. This would best be done with some support from the Python installation machinery. -> Small nice size for extensions, still same CRT state problems. E As D + a new compiler-independent native code interface that does not carry dependencies on CRT state such as file descriptors, like JNI. -> Really huge effort, and cannot be applied until some new Python version. And I think the clue here is that the CRT state problems can be avoided by careful coding. Hence, for those who cannot do A I think B is a realistic practical option, and D would be nice... Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* Martin v. Loewis, on 07.07.2010 21:56: Perhaps (if it isn't intentional) this is a bug of the oversight type, that nobody remembered to update the macro? Update in what way? I was guessing that at one time there was no PyMem_Malloc. And that it was introduced to fix Windows-specific problems, but inadvertently without updating the macro. It's just a guess as to reasons why the macro uses malloc directly. It might indeed be that the function version was introduced specifically for Windows. However, the macro was left intentionally: both for backwards compatibility, and for use inside Python itself. Except for the problems with file descriptors I think a practical interim solution for extensions implemented in C could be to just link the runtime lib statically. [...] When I wrote "link the runtime lib statically" that was an alternative to the usual link-as-DLL. Ok, I lost the thread. When you said "a practical interim solution" you were talking about what problem? I thought the discussion was about the need to link with the same DLL version as Python. The main problem that the required MSVC redistributables are not necessarily present on the end user's system. It wouldn't make sense to link the runtime lib statically as an alternative to linking it statically. However, it would surely make sense to link with a different DLL than the one that Python links with, assuming that would actually work. As for linking to a different /version/ of the CRT, if you really mean that, I think that's difficult. It's not necessarily impossible, after all there's STLPort. But I think that it must at the very least be rather difficult to do with Microsoft's tools, for otherwise people would have employed that solution before, and so I wouldn't trust the result, and wouldn't waste the time trying. It's actually straight-forward (or used to be, until they came up with the SxS madness). It was actually the case that people did so unexpectingly, and it seemed to work fine, except that it crashed when passing FILE*. Then we started explaining that mixing CRTs is risky. Oh. Well then. :-) Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* sturlamolden, on 07.07.2010 21:46: On 7 Jul, 21:41, "Alf P. Steinbach /Usenet" wrote: You still have two CRTs linked into the same process. So? CRT resources cannot be shared across CRT borders. That is the problem. Multiple CRTs are not a problem if CRT resources are never shared. Yeah, but then we're down to file descriptors, C library locales and such as the remaining problems. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* sturlamolden, on 07.07.2010 21:12: On 7 Jul, 06:54, "Alf P. Steinbach /Usenet" wrote: PyAPI_FUNC(void *) PyMem_Malloc(size_t); #define PyMem_MALLOC(n) (((n)< 0 || (n)> PY_SSIZE_T_MAX) ? NULL \ : malloc((n) ? (n) : 1)) I was afraid of that :( Except for the problems with file descriptors I think a practical interim solution for extensions implemented in C could be to just link the runtime lib statically. You still have two CRTs linked into the same process. So? Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* Martin v. Loewis, on 07.07.2010 21:10: Python 3.1.1, file [pymem.h]: PyAPI_FUNC(void *) PyMem_Malloc(size_t); #define PyMem_MALLOC(n)(((n)< 0 || (n)> PY_SSIZE_T_MAX) ? NULL \ : malloc((n) ? (n) : 1)) The problem with the latter that it seems that it's intended for safety but does the opposite... Why do you say that? It certainly *does* achieve safety, wrt. to certain errors, specifically: - passing sizes that are out-of-range - supporting malloc(0) on all systems It uses malloc instead of PyMem_Malloc. malloc may well be different and use a different heap in an extension DLL than in the Python interpreter and other extensions. That's one thing that the docs (rightly) warn you about. Perhaps (if it isn't intentional) this is a bug of the oversight type, that nobody remembered to update the macro? Update in what way? I was guessing that at one time there was no PyMem_Malloc. And that it was introduced to fix Windows-specific problems, but inadvertently without updating the macro. It's just a guess as to reasons why the macro uses malloc directly. Except for the problems with file descriptors I think a practical interim solution for extensions implemented in C could be to just link the runtime lib statically. For a minimal extension this increased the size from 8 KiB to 49 KiB. And generally with MS tools the size is acceptably small. If you think that's fine for your extension module (which may well be the case), go ahead. I have no comment on that except pointing it out as a somewhat stupid, somewhat evil social inclusion/exclusion argument, talking to the audience. Argh. You're wasting my time. But anyway, 49 KiB is small by today's standards. For example, you get 20 of those in a single MiB, and about 20.000 in a single GiB. But then, you could also just link with a different DLL version of the CRT instead. When I wrote "link the runtime lib statically" that was an alternative to the usual link-as-DLL. It wouldn't make sense to link the runtime lib statically as an alternative to linking it statically. As for linking to a different /version/ of the CRT, if you really mean that, I think that's difficult. It's not necessarily impossible, after all there's STLPort. But I think that it must at the very least be rather difficult to do with Microsoft's tools, for otherwise people would have employed that solution before, and so I wouldn't trust the result, and wouldn't waste the time trying. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* sturlamolden, on 06.07.2010 19:35: On 6 Jul, 19:09, Thomas Jollans wrote: Okay, you need to be careful with FILE*s. But malloc and free? You'd normally only alloc& free something within the same module, using the same functions (ie not mixing PyMem_Malloc and malloc), would you not? You have to be sure PyMem_Malloc is not an preprocessor alias for malloc (I haven't chaecked). Python 3.1.1, file [pymem.h]: PyAPI_FUNC(void *) PyMem_Malloc(size_t); #define PyMem_MALLOC(n) (((n) < 0 || (n) > PY_SSIZE_T_MAX) ? NULL \ : malloc((n) ? (n) : 1)) The problem with the latter that it seems that it's intended for safety but does the opposite... Perhaps (if it isn't intentional) this is a bug of the oversight type, that nobody remembered to update the macro? *** Except for the problems with file descriptors I think a practical interim solution for extensions implemented in C could be to just link the runtime lib statically. For a minimal extension this increased the size from 8 KiB to 49 KiB. And generally with MS tools the size is acceptably small. I think that this would be safe because since the C API has to access things in the interpreter I think it's a given that all the relevant functions delegate to shared library (DLL) implementations, but I have not checked the source code. As a more longterm solution, perhaps python.org could make available the redistributables for various MSVC versions, and then one could introduce some scheme for indicating the runtime lib dependencies of any given extension. Then when installing an extension the installer (distutils package functionality) could just check whether the required runtime is present, and if not give the user the choice of automatically downloading from python.org, or perhaps direct from Microsoft. This scheme would support dependencies on new runtime lib versions not yet conceived when the user's version of Python was installed. Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Argh! Name collision!
Donald Knuth once remarked (I think it was him) that what matters for a program is the name, and that he'd come up with a really good name, now all he'd had to do was figure out what it should be all about. And so considering Sturla Molden's recent posting about unavailability of MSVC 9.0 (aka Visual C++ 2008) for creating Python extensions in Windows, and my unimaginative reply proposing names like "pni" and "pynacoin" for a compiler independent Python native code interface, suddenly, as if out of thin air, or perhaps out of fish pudding, the name "pyni" occurred to me. "pyni"! Pronounced like "tiny"! Yay! I sat down and made my first Python extension module, following the tutorial in the docs. It worked! But, wait, perhaps some other extension is already named "piny"? Google. http://code.google.com/p/pyni/>, "PyNI is [a] config file reader/writer". Argh! - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
* sturlamolden, on 06.07.2010 17:50: Just a little reminder: Microsoft has withdrawn VS2008 in favor of VS2010. The express version is also unavailable for download.>:(( We can still get a VC++ 2008 compiler required to build extensions for the official Python 2.6 and 2.7 binary installers here (Windows 7 SDK for .NET 3.5 SP1): http://www.microsoft.com/downloads/details.aspx?familyid=71DEB800-C591-4F97-A900-BEA146E4FAE1&displaylang=en Download today, before it goes away! Microsoft has now published a download for Windows 7 SDK for .NET 4. It has the VC++ 2010 compiler. It can be a matter of days before the VC ++ 2008 compiler is totally unavailable. It is possible to build C and Fortran extensions for official Python 2.6/2.7 binaries on x86 using mingw. AFAIK, Microsoft's compiler is required for C++ or amd64 though. (Intel's compiler requires VS2008, which has now perished.) Remember Python on Windows will still require VS2008 for a long time. Just take a look at the recent Python 3 loath threads. Perhaps this all for the good. There is no *technical* problem creating a compiler-independent C/C++ language binding. I believe that Java's JNI works fine no matter what compiler you use, although it's many many years since I've done JNI things. Similarly, Python should IMHO just have a well defined compiler independent native code interface, e.g. "PNI", or "pynacoin", the PYthon NAtive COde INterface :-) Cheers, - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: The real problem with Python 3 - no business case for conversion
* Steven D'Aprano, on 03.07.2010 16:24: On Sat, 03 Jul 2010 08:46:57 -0400, D'Arcy J.M. Cain wrote: On Fri, 02 Jul 2010 22:40:34 -0700 John Nagle wrote: Not according to Vex's published package list: http://www.vex.net/info/tech/pkglist/ Hold on. That *is* the generated list and Python 3.1 is on it. We have both 2.6 and 3.1. The 3.1 version is listed right below the 2.6 one. The page is generated from pkg_info(1) and includes everything we have installed from FreeBSD ports. Pfft! Facts! You can prove anything you like with facts! :-) -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
Re: Decorators, with optional arguments
* Stephen Hansen, on 02.07.2010 19:41:
Okay, so!
I actually never quite got around to learning to do deep and useful
magic with decorators. I've only ever done the most basic things with
them. Its all been a little fuzzy in my head: things like what order
decorators end up being called in if there's more then one, etc.
But in my current situation, what I'm wanting to do is have a decorator
that wraps a function but which takes an *optional* argument, and sets
that argument as an attribute on said function if its there.
Here's what some tweaking and playing around has gotten me, as a recipe:
import functools
def my_decorator(arg):
if callable(arg): # Stuck on 2.5. Leavemealone. :)
protocol = None
else:
protocol = arg
def wrap(fn):
print "Wrapping."
fn.protocol = protocol
@functools.wraps(fn)
def wrapper(*args, **kwargs):
print "Calling."
result = fn(*args, **kwargs)
print "Called."
return result
return wrapper
if not protocol: # argument-less decorator
print "Calling wrap."
return wrap(arg)
else:
print "Returning wrap."
return wrap
To be used as:
class Thing(object):
@expose
def test1(self, arg1):
return arg1
@expose("testing")
def test2(self, arg2):
return arg2
So, my question: am I doing this right? :) Some play-through testing
appears to work. But, the dizzying array of nested def's up there leaves
me a bit dazed, so I'm wondering if there's a simpler way to accomplish
what I'm trying to do.
If you're willing to have slightly more explicit usage code, consider e.g.
#Py3
import functools
class expose:
def __init__( self, protocol = None ):
self._protocol = protocol
def __call__( self, f ):
print( "Wrapping." )
f.protocol = self._protocol
@functools.wraps( f )
def wrapper( *args, **kwargs ):
print( "Calling." )
result = f( *args, **kwargs )
print( "Called." )
return result
return wrapper
class Thing(object):
@expose()
def test1(self, arg1):
return arg1
@expose( "testing" )
def test2(self, arg2):
return arg2
o = Thing()
print( o.test1( 1.11 ) )
print( o.test2( 2.22 ) )
Cheers & hth.,
- Alf
--
blog at http://alfps.wordpress.com>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Using Classes
* Mag Gam, on 24.06.2010 13:58: I have been using python for about 1 year now and I really like the language. Obviously there was a learning curve but I have a programing background which made it an easy transition. I picked up some good habits such as automatic code indenting :-), and making my programs more modular by having functions. I know that Python is very OOP friendly, but I could not figure out why and when to use Classes in it. I mostly use it for simple text parsing I suppose when a program gets "complicated" I should start using Classes. Are there any tips or tricks people use to "force" them into the OOP mentality? I would like to force myself to learn the Python way but so far I could not figure out WHY I would need a class for this... Use whatever paradigm that yields code that you easily understand. Having said that, the main use of a class is to model some data with an associated set of operations. For this a class is merely a convenience, but sometimes the convenience can be so great that any other approach would be impractical in comparision. In some cases a non-class approach ends up with a number of routines foo, bar, zlopf etc. that all take some argument of a "general" type and all internally has some "if it's really a BRACHYKLURB, do A, assuming that it has BRACHYKLURB-specific data, but if it's really a KNETCHOFICHUS, then do B, assuming that it has KNETCHOFICHUS-specific data, and as default, if it's none of those, do C". This is a maintainance nightmare. You have to be sure that every such routine discriminates correctly on type, checking all possibilities and doing the right thing for each possibility. And you have to be sure of that in the face of possible additions of new subtypes, or removal of a subtype (maintainance). Then the answer is to introduce some classes and OOP stuff. In essence, instead of letting general routines awkwardly and error-prone choose the right type-specific routines depending on the object, let the object specify the right type-specific routines directly. :-) Cheers & hth., - Alf -- blog at http://alfps.wordpress.com> -- http://mail.python.org/mailman/listinfo/python-list
