date:20140801

On Fri, Aug 1, 2014 at 9:10 PM, Wolfgang Keller  wrote:
> Thankfully, all actually user-friendly operating systems (MacOS,
> TOS, RiscOS, probably AmigaOS, MacOS X) spare(d) their users the
> bottomless cesspit of "package management" and/or "installers".
>
> Because on such operating systems, each and every application is an
> entirely self-contained package that doesn't need any "packages" or
> "installers" to use it.

You mean everyone has to reinvent the proverbial wheel AND worry about
dependency collisions? Yeah, that's a heavenly thought.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What is best way to learn Python for advanced developer?

2014-08-01 Thread Rustom Mody

On 07/30/2014 02:20 PM, guirec corbel wrote:
> Hello,
> I am a Ruby developer and I want to program in Python. I know how to
> do simple things like create classes, methods, variables and all the
> basics. I want to know more. I want to know what is the Python
> philosophy, how to test, how to create maintenable software, etc.
> I'm looking for online courses and any ressources I can have on the
> subject.
> Can you help me?

Yes its true that most books and web material concentrate on toy
material. I guess thats inevitable.  I remember one of the CS greats
- I think it was Peter Naur - saying that programming pedagogy is too
much focused on language and too little on literature.

Two books that try to buck this trend a bit come to mind [Ive not
checked out either]

1. Expert Python Programming by Tarek Ziade
2. Hacker's guide by Danjou 
https://julien.danjou.info/books/the-hacker-guide-to-python
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs

Chris Angelico :

> On Fri, Aug 1, 2014 at 9:10 PM, Wolfgang Keller  wrote:
>> Because on such operating systems, each and every application is an
>> entirely self-contained package that doesn't need any "packages" or
>> "installers" to use it.
>
> You mean everyone has to reinvent the proverbial wheel AND worry about
> dependency collisions? Yeah, that's a heavenly thought.

I'm guessing he's referring to the modern fad of application sandboxing.
Each application is installed with everything it needs on top of the
basic OS.

If you have ten Python apps, you'll have ten Python installations. Also
the applications have no way to communicate outside their respective
sandboxes. They can't access each others' files, for example.

Personally, I tend to stick to this package management strategy: install
whatever is available with yum and write the rest yourself.

Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs

On Fri, Aug 1, 2014 at 10:19 PM, Marko Rauhamaa  wrote:
> I'm guessing he's referring to the modern fad of application sandboxing.
> Each application is installed with everything it needs on top of the
> basic OS.
>
> If you have ten Python apps, you'll have ten Python installations. Also
> the applications have no way to communicate outside their respective
> sandboxes. They can't access each others' files, for example.
>
> Personally, I tend to stick to this package management strategy: install
> whatever is available with yum and write the rest yourself.

Only if by "write" you also include compiling someone else's program
from source. I follow that strategy (except that I use apt rather than
yum), and there's a fair bit that I build from source but don't write.
Granted, that's partly because Debian Stable doesn't include a
sufficiently recent Python, for instance, but still, there's a lot to
be said for getting libraries (including dev versions) from the repo
and building some applications yourself.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

dict to boolean expression, how to?

2014-08-01 Thread Alex van der Spek

With a dict like so:

cond = {'a': 1, 'b': 1, 'c': 1,
'A': 0, 'B', 0, 'C':0}

how would you make a boolean expression like this:

bool = (('a' == 1) & ('A' == 0) |
('b' == 1) & ('B' == 0) |
('c' == 1) & ('C' == 0))

The fact that lowercase and uppercase keys are stringed together with & is 
intentional albeit the actual condition is a bit more tricky.

I've tried several approaches using eval() on a string built from the dict 
but landed with just spelling it out literally.


Any pointers welcome.
Alex
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

On Fri, Aug 1, 2014 at 10:45 PM, Alex van der Spek  wrote:
> how would you make a boolean expression like this:
>
> bool = (('a' == 1) & ('A' == 0) |
> ('b' == 1) & ('B' == 0) |
> ('c' == 1) & ('C' == 0))

Not sure what the use of this is, because 'a' will never be equal to
1. Are you trying to magically fetch names from your environment?
Because that's usually a bad idea.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs

Marko Rauhamaa wrote:

> Chris Angelico :
> 
>> On Fri, Aug 1, 2014 at 9:10 PM, Wolfgang Keller  wrote:
>>> Because on such operating systems, each and every application is an
>>> entirely self-contained package that doesn't need any "packages" or
>>> "installers" to use it.
>>
>> You mean everyone has to reinvent the proverbial wheel AND worry about
>> dependency collisions? Yeah, that's a heavenly thought.
> 
> I'm guessing he's referring to the modern fad of application sandboxing.
> Each application is installed with everything it needs on top of the
> basic OS.
>
> If you have ten Python apps, you'll have ten Python installations. 

A horrible thought. Hard drives are cheap, but not that cheap that one can
trivially afford to turn every 1K Python script into a 25,000K install
(based on the size of the Windows binary-only installer). On my system, the
obvious application directories (I may have missed some) total 460MB:

[steve@ando ~]$ du -hc /bin/ /sbin/ /usr/bin/ /usr/local/bin/
7.9M/bin/
38M /sbin/
76K /usr/bin/mergetools
380M/usr/bin/
35M /usr/local/bin/
460Mtotal

If those apps were an average of 10,000 times larger, that makes 4.6TB,
significantly larger than an entry level 1TB hard drive. It also makes
rescue DVDs and boot USB sticks impractical, to say nothing of the expense
of downloading upgrades. I can download (say) an entire Linux Mint system
in an hour or three, which is significantly better than the two years it
would take to download if everything was 10,000 times bigger.

But even more problematic... if there's a security vulnerability in Python,
would you rather wait for the vulnerability to patched once in a central
Python binary, or individually in each and every single Python script that
comes with a bundled Python binary?

> Also 
> the applications have no way to communicate outside their respective
> sandboxes. They can't access each others' files, for example.

If two applications can both write to the file system, they can communicate.
If they have sufficient file system privileges, they can even reach into
each other's bundle and modify anything they want.

> Personally, I tend to stick to this package management strategy: install
> whatever is available with yum and write the rest yourself.

+0.8 on that. Sometimes I install software outside of the package management
system, but I always feel a tad dirty when I do so :-)

-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Roy Smith

In article <53db8bd8$0$2976$e4fe5...@news2.news.xs4all.nl>,
 Alex van der Spek  wrote:

> With a dict like so:
> 
> cond = {'a': 1, 'b': 1, 'c': 1,
> 'A': 0, 'B', 0, 'C':0}
> 
> how would you make a boolean expression like this:
> 
> bool = (('a' == 1) & ('A' == 0) |
> ('b' == 1) & ('B' == 0) |
> ('c' == 1) & ('C' == 0))
> 
> The fact that lowercase and uppercase keys are stringed together with & is 
> intentional albeit the actual condition is a bit more tricky.
> 
> I've tried several approaches using eval() on a string built from the dict 
> but landed with just spelling it out literally.

I would certainly avoid eval(), especially if there's any chance 
(including ways you can't possibly imagine right now) of external data 
getting included.  Google for "little bobby tables".

Maybe something like (coding on the fly, not tested):

terms = []
for key in 'abc':
term = cond[key] and not cond[key.upper()]
terms.append(term)

bool_value = any(terms)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

On Fri, 01 Aug 2014 12:45:12 +, Alex van der Spek wrote:

> With a dict like so:
> 
> cond = {'a': 1, 'b': 1, 'c': 1,
> 'A': 0, 'B', 0, 'C':0}
> 
> how would you make a boolean expression like this:
> 
> bool = (('a' == 1) & ('A' == 0) |
> ('b' == 1) & ('B' == 0) |
> ('c' == 1) & ('C' == 0))

That's False. It's always False, because 'a' does not equal 1, etc. Also, 
you're using bitwise operators & and | rather than boolean operators. 
Finally, you are shadowing the built-in bool() type, which is probably a 
bad idea.

In the first case, I think you mean cond['a'] == 1 rather than just 
'a'==1; in the second case, the bool operators are called "and" and "or"; 
and in the third case, there are many equally good names for a generic 
boolean flag, like "flag".

Putting those together, I think you probably mean something like this:

flag = (
(cond['a'] == 1 and cond['A'] == 0) or
(cond['b'] == 1 and cond['B'] == 0) or
(cond['c'] == 1 and cond['C'] == 0)
)

which can be simplified to:

flag = any( cond[c] == 1 and cond[c.upper()] for c in ['a', 'b', 'c'] )

If you *really* mean bitwise-and, you can change the "and"s to & and 
"or"s to | but honesty that's pointless and inefficient because all your 
conditions are bools, not arbitrary integers.

> The fact that lowercase and uppercase keys are stringed together with &
> is intentional albeit the actual condition is a bit more tricky.
> 
> I've tried several approaches using eval() on a string built from the
> dict but landed with just spelling it out literally.

Ayyyeee Don't use eval(). Really. Any time you think you need to use 
eval(), smash your hand with a hammer until the urge goes away.

*wink*

But seriously: if you need to ask "why not use eval?", you're not ready 
to use eval safely. But in a nutshell:

- using eval is a large performance hit (about 10 times slower);

- eval is dangerous and can introduce code injection vulnerabilities.

eval is almost never the right solution to any problem, and in the very 
few exceptions, it needs careful handling by an expert to ensure you're 
not introducing serious security bugs.

-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs

On Fri, Aug 1, 2014 at 11:10 PM, Steven D'Aprano
 wrote:
> Marko Rauhamaa wrote:
>
>> I'm guessing he's referring to the modern fad of application sandboxing.
>> Each application is installed with everything it needs on top of the
>> basic OS.
>>
>> If you have ten Python apps, you'll have ten Python installations.
>
> A horrible thought. Hard drives are cheap, but not that cheap that one can
> trivially afford to turn every 1K Python script into a 25,000K install
> (based on the size of the Windows binary-only installer). On my system, the
> obvious application directories (I may have missed some) total 460MB:
>
> [steve@ando ~]$ du -hc /bin/ /sbin/ /usr/bin/ /usr/local/bin/
> 7.9M/bin/
> 38M /sbin/
> 76K /usr/bin/mergetools
> 380M/usr/bin/
> 35M /usr/local/bin/
> 460Mtotal
>
> If those apps were an average of 10,000 times larger, that makes 4.6TB,
> significantly larger than an entry level 1TB hard drive. It also makes
> rescue DVDs and boot USB sticks impractical, to say nothing of the expense
> of downloading upgrades. I can download (say) an entire Linux Mint system
> in an hour or three, which is significantly better than the two years it
> would take to download if everything was 10,000 times bigger.

There is a solution. If all those binaries are marked as read-only,
you could have a file system that stores things based on their hashes,
effectively hardlinking (automatically) all the duplicates. Of course,
that only works if they really are duplicates. If one ships Python
2.7.3 and another ships 2.7.4, there'll be a lot of almost-duplicated
files that technically identical.

> But even more problematic... if there's a security vulnerability in Python,
> would you rather wait for the vulnerability to patched once in a central
> Python binary, or individually in each and every single Python script that
> comes with a bundled Python binary?

This is exactly the problem that sandboxing "fixes", though. As soon
as you upgrade the central Python binary, you risk breaking that
application that depended on the exact version that it shipped with.

Encouraging laziness and sloppy versioning, IMO.

>> Also
>> the applications have no way to communicate outside their respective
>> sandboxes. They can't access each others' files, for example.
>
> If two applications can both write to the file system, they can communicate.
> If they have sufficient file system privileges, they can even reach into
> each other's bundle and modify anything they want.

If you chroot to the sandbox, they shouldn't be able to. (Not to say
there's no such thing as chroot leakage, of course, but they
shouldn't.)

>> Personally, I tend to stick to this package management strategy: install
>> whatever is available with yum and write the rest yourself.
>
> +0.8 on that. Sometimes I install software outside of the package management
> system, but I always feel a tad dirty when I do so :-)

I don't. There's plenty that I've done that way - but only ever
applications, or libraries that completely don't exist in the repos.
I've never installed a newer version of a library than I can get from
repo.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

On Fri, 01 Aug 2014 14:17:41 +1000, Ben Finney wrote:

> Steven D'Aprano  writes:
> 
>> On Thu, 31 Jul 2014 20:12:12 -0700, Dan Stromberg wrote:
>>
>> > I removed some quotes, and noticed that 1 and 1.0 hash the same.
>> > That's a bit unexpected, but I suppose it's not completely
>> > unreasonable.
> 
> I would expect it, for the reasons Steven explains. Why is it
> unexpected?
> 
>> You should expect that two equal objects should hash to the same value
>> in different versions of Python, or even from one run of Python to the
>> next.
> 
> I suspect there's a “not” missing there.

Thanks for spotting that. Correct, I meant that you should *not* expect 
hashing to return any specific value.


[...]
> If you expect the hash valuesto be the same, you're making a mistake; if
> you expect the hash values to be different, you're making a mistake.

Yes. Although Python promises (at least for classes in the standard 
library) that x == y should imply that hash(x) == hash(y), it says 
nothing about the other way:

x == y implies that hash(x) == hash(y)

but hash(x) == hash(y) does NOT imply that x == y.


py> hash(2**128) == hash(1)
True


> The correct attitude, with regard to implementation details, is to
> expect uncertainty and not assume anything :-)

Yes, this!



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Roy Smith

In article <53db95e6$0$29986$c3e8da3$54964...@news.astraweb.com>,
 Steven D'Aprano  wrote:

> eval is almost never the right solution to any problem, and in the very 
> few exceptions, it needs careful handling by an expert to ensure you're 
> not introducing serious security bugs.

Corollary to that rule: All the people who are smart enough to actually 
understand how to use eval() safety, are also smart enough to know not 
to use it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting a list of all modules

2014-08-01 Thread Robert Kern


On 2014-07-31 11:41, Steven D'Aprano wrote:

On Wed, 30 Jul 2014 21:22:18 +0800, Leo Jay wrote:


On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano 
wrote:

I'm looking for a programmatic way to get a list of all Python modules
and packages. Not just those already imported, but all those which
*could* be imported.



If you don't actually import it, how can you know it could be imported?
Not all .so files are valid python modules. Not all .py files could be
imported by all python interpreters.


You're right, of course, but I'm not concerned by whether or not the
module is error-free and can be imported successfully.

I'm working on tab completion for module names. I have some alpha-quality
code working, so if I hit TAB after typing "import ma" I get this:


py> import ma
macpath  macurl2path  mailbox  mailcap  mangle
markupbase   math

For what it's worth, importing "mangle" fails with a SyntaxError. But
that's okay, I don't expect tab completion to only show *valid*
modules :-)

Over the next few days I'll make an official announcement, but if anyone
wants a sneak-peek, check out:

http://code.google.com/p/tabhistory/source/browse/tabhistory.py


where I have indenting, code completion, filename completion, and module
completion all working to some degree or another.


Take a look at what has already been implemented in IPython:

https://github.com/ipython/ipython/blob/master/IPython/core/completerlib.py#L208

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

On Fri, Aug 1, 2014 at 11:31 PM, Steven D'Aprano
 wrote:
> Yes. Although Python promises (at least for classes in the standard
> library) that x == y should imply that hash(x) == hash(y), it says
> nothing about the other way:
>
> x == y implies that hash(x) == hash(y)

This is the entire point of the hashing system. If equal values can
hash differently, why bother calculating the hashes?

Or if you prefer, the point of hashing is the logical converse of the
above: hash(x) != hash(y) implies that x != y. If the hashes are
different, you can skip the equality check, ergo you can build a data
type that claims to search for the key that == the looked-up key, but
actually does a much faster hash check first, and skips everything
with a different hash.

> but hash(x) == hash(y) does NOT imply that x == y.
>

Hello, pigeonhole principle :) If this were false - that is, if equal
hashes DID imply equal objects - it would be necessary to completely
encode an object's state in its hash, and hashes would be impossibly
large. This would, in fact, destroy their value completely.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

2014-08-01 Thread Nicholas Cole

On Fri, Aug 1, 2014 at 12:22 PM, Chris Angelico  wrote:
> On Fri, Aug 1, 2014 at 9:10 PM, Wolfgang Keller  wrote:
>> Thankfully, all actually user-friendly operating systems (MacOS,
>> TOS, RiscOS, probably AmigaOS, MacOS X) spare(d) their users the
>> bottomless cesspit of "package management" and/or "installers".
>>
>> Because on such operating systems, each and every application is an
>> entirely self-contained package that doesn't need any "packages" or
>> "installers" to use it.
>
> You mean everyone has to reinvent the proverbial wheel AND worry about
> dependency collisions? Yeah, that's a heavenly thought.

Actually, that's not right.  RiscOS had and OS X has (I'm sure the
others do as well) a concept that is similar to a shared library.  But
the joy of an application bundle is that installing an application
does not scatter its own files all over the file-system, putting
configuration files here, binary resources there, library files
somewhere else, executable files somewhere else again.  The result on
one of these other systems is that uninstalling an application is a
simple matter of deleting the relevant bundle, which contains all of
the resources necessary for that application.  All that remains are
whatever files exist within user directories.

I've worked with both.  Quite honestly, I really wish that other
operating systems had gone down this route. MS didn't possibly to make
it harder to steal software, and Unix...well, *nix has the concept of
the "distribution" that will manage all of this for you.  We all know
the problems that this causes.

N.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Alex van der Spek

On Fri, 01 Aug 2014 12:45:12 +, Alex van der Spek wrote:

> With a dict like so:
> 
> cond = {'a': 1, 'b': 1, 'c': 1,
> 'A': 0, 'B', 0, 'C':0}
> 
> how would you make a boolean expression like this:
> 
> bool = (('a' == 1) & ('A' == 0) |
> ('b' == 1) & ('B' == 0) |
> ('c' == 1) & ('C' == 0))
> 
> The fact that lowercase and uppercase keys are stringed together with &
> is intentional albeit the actual condition is a bit more tricky.
> 
> I've tried several approaches using eval() on a string built from the
> dict but landed with just spelling it out literally.
> 
> 
> Any pointers welcome.
> Alex

I am sorry, the problem is ill posed.

'a', 'A' and so forth are my failed attempt to shorthand.

In reality the dict's keys are column names in a pandas dataframe df.

The boolean expression would therefore look like:

bool = ((df['a'] == 1) & (df['A'] == 0) |
 (df['b'] == 1) & (df['B'] == 0) |
 (df['c'] == 1) & (df['C'] == 0))

I do know eval() lends itself to code injection but can't say I am
fully aware of its dangers. It seemed like a handy tool to me.

This newsgroup scares me, it appears to be for professional computer 
scientists only, the theoretical part is sometimes too much for this 
practical physicist with an old background in FORTRAN.

Is there a better place to ask questions of this nature?

Alex van der Spek
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Ned Batchelder


On 8/1/14 8:45 AM, Alex van der Spek wrote:

With a dict like so:

cond = {'a': 1, 'b': 1, 'c': 1,
 'A': 0, 'B', 0, 'C':0}

how would you make a boolean expression like this:

bool = (('a' == 1) & ('A' == 0) |
 ('b' == 1) & ('B' == 0) |
 ('c' == 1) & ('C' == 0))

The fact that lowercase and uppercase keys are stringed together with & is
intentional albeit the actual condition is a bit more tricky.

I've tried several approaches using eval() on a string built from the dict
but landed with just spelling it out literally.


Any pointers welcome.
Alex



Are you looking for this?

bool = (
(cond['a'] == 1 and cond['A'] == 0) or
(cond['b'] == 1 and cond['B'] == 0) or
(cond['c'] == 1 and cond['C'] == 0)
)

If so, what's wrong with that expression exactly?  I'm not sure how eval 
could enter into it, so I feel like I'm missing something.


--
Ned Batchelder, http://nedbatchelder.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

2014-08-01 Thread Ian Kelly

On Thu, Jul 31, 2014 at 9:28 PM, Terry Reedy  wrote:
> On 7/31/2014 5:15 PM, Ian Kelly wrote:
>>
>> On Thu, Jul 31, 2014 at 5:24 AM, Dilu Sasidharan 
>> wrote:
>>>
>>> Hi,
>>>
>>> I am wondering why the dictionary in python not returning multi value key
>>> error when i define something like
>>>
>>> p = {'k':"value0",'k':"value1"}
>>>
>>> key is string immutable and sometimes shares same id.
>>>
>>> also if the key is immutable and have different ids.
>>>
>>> like
>>>
>>> p = {'1':"value0",'1.0':"value1"}
>>
>>
>> In this latter case note that '1' and '1.0' are not equal, so this
>> will simply result in two separate entries in the dict anyway.
>
>
> Dilu presumably meant
>
 p = {1:"value0", 1.0:"value1"}
 p
> {1: 'value1'}

Maybe, but it was explicitly stated that the keys were strings.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

On Sat, Aug 2, 2014 at 12:28 AM, Nicholas Cole  wrote:
> Actually, that's not right.  RiscOS had and OS X has (I'm sure the
> others do as well) a concept that is similar to a shared library.  But
> the joy of an application bundle is that installing an application
> does not scatter its own files all over the file-system, putting
> configuration files here, binary resources there, library files
> somewhere else, executable files somewhere else again.

Shared libraries are a code execution model. The question is, how do
you locate them? Let's suppose I write a program that requires the
Nettle cryptography library. I can't depend on you already having that
on your system; you might, but I can't depend on it. What do I do? Do
I include it in my application bundle, or do I have some small record
that says "and by the way, I need libnettle"? Including it in the
application bundle means there'll be duplicates everywhere, possibly
of slightly different versions. Not including it means there needs to
be some kind of system for resolving dependencies, which is exactly
what apt-get, yum, pacman, and so on are for.

The installer has basically three choices.
1) Install libnettle inside the application directory
2) Install libnettle to some system library directory
3) Don't install libnettle, and demand that someone else (perhaps the
user, or the system package manager) install it.

Option 1 results in duplications. (Unless one application is allowed
to access a library in another application's directory, which is a
HORRIBLE mess.) Option 2 is exactly what you're complaining about,
scattering files all over the FS. And option 3 is what package
managers are for. What are you advocating?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

On Fri, 01 Aug 2014 09:32:36 -0400, Roy Smith wrote:

> In article <53db95e6$0$29986$c3e8da3$54964...@news.astraweb.com>,
>  Steven D'Aprano  wrote:
> 
>> eval is almost never the right solution to any problem, and in the very
>> few exceptions, it needs careful handling by an expert to ensure you're
>> not introducing serious security bugs.
> 
> Corollary to that rule: All the people who are smart enough to actually
> understand how to use eval() safety, are also smart enough to know not
> to use it.

... smart enough to know WHEN to use it (which is *rarely*).

That's in production code, of course. There's nothing wrong with using 
eval in the interactive interpreter for quick and dirty exploration. But 
even then, I find that it's usually easier to write a line or two of 
Python code to process something than to try using eval.

There are uses for eval (or exec), even if production code. See 
collections.namedtuple, doctest, and timeit, for example. In the first 
place, namedtuple takes extra care to sanitise the data being used. In 
the case of doctest and timeit, the whole point of them is to run trusted 
code. If you can't trust your own code that you're timing, what can you 
trust?

-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread marco . nawijn

Hi,

Are you aware of the Python operator module?
It provides function equivalents of all (most?)
python operator. So instead of a==b, you can
state operator.eq(a,b). As a result, you can
loop over the key/value pairs in the dict and
built your logic with the operator.eq, 
operator.and_, and operator.or_ (notice the 
trailing underscore to avoid clashing with
normal "and" and "or" statements.

Marco
 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs

Steven D'Aprano :

> Marko Rauhamaa wrote:
>> If you have ten Python apps, you'll have ten Python installations. 
>
> A horrible thought. Hard drives are cheap, but not that cheap

Well, iPhones aren't *that* expensive...

>> Also the applications have no way to communicate outside their
>> respective sandboxes. They can't access each others' files, for
>> example.
>
> If two applications can both write to the file system,

They can't. They are sandboxed.

My WP8 phone has Python3 installed. Absolutely useless since it can't
get out of its sandbox (except for Microsoft's cloud drive, I think).

Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

On Fri, 01 Aug 2014 14:26:38 +, Alex van der Spek wrote:

[...]
> bool = ((df['a'] == 1) & (df['A'] == 0) |
>  (df['b'] == 1) & (df['B'] == 0) |
>  (df['c'] == 1) & (df['C'] == 0))
> 
> I do know eval() lends itself to code injection but can't say I am fully
> aware of its dangers. It seemed like a handy tool to me.

And in expert hands, it can be. But I can't see how you're using eval in 
this case. Since you're not having any success, perhaps it's not as handy 
as you imagine?

Did you try the suggestion to use any( ... )? Did it work?

> This newsgroup scares me, it appears to be for professional computer
> scientists only, 

Not even close! Some of us aren't even professional programmers!

> the theoretical part is sometimes too much for this
> practical physicist with an old background in FORTRAN.

Oh, I sympathise. I know it can be hard to avoid feeling overwhelmed when 
you are dropped far outside of your comfort zone in a topic you know 
nothing about. But you're a physicist, and therefore you've already 
proven you have the logical skills to reason about quite complex and 
mathematical topics. If you're lost now, I wager it's because you're 
unfamiliar with the territory, not because it's beyond you.

So please, don't be shy about asking people to explain in simpler terms. 
Sometimes we forget that things which are second nature to us are not 
familiar to somebody new to the language. Or we don't want to patronise 
them by assuming that every person asking a question needs their hand-
held and every little detail spelled out in painfully small steps. It's 
quite hard to find the right balance, especially when we don't know your 
level of knowledge.

So please, ask concrete questions, the more specific the better. Don't be 
afraid to ask what unfamiliar terms means.

> Is there a better place to ask questions of this nature?

I don't know of anyone here who is an expert in pandas, so if you ask 
questions which are specific to pandas, we may run into the limits of our 
knowledge. If you can find a dedicated pandas mailing list or other 
forum, they may help too, but I don't know if they will be more or less 
willing to explain the basics about Python.

-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

Chris Angelico :

>> but hash(x) == hash(y) does NOT imply that x == y.
>
> Hello, pigeonhole principle :) If this were false - that is, if equal
> hashes DID imply equal objects - it would be necessary to completely
> encode an object's state in its hash, and hashes would be impossibly
> large. This would, in fact, destroy their value completely.

Well, modern computing assumes precisely:

   hash(x) == hash(y) => x == y

That principle is at play with strong authentication (HTTPS et al),
version control (git et al),

A handful of bits uniquely identify an object regardless of its size.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting a list of all modules

On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote:

> Take a look at what has already been implemented in IPython:
> 
> https://github.com/ipython/ipython/blob/master/IPython/core/
completerlib.py#L208


Awesome! Thank you!


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Peter Otten

Alex van der Spek wrote:

> I do know eval() lends itself to code injection but can't say I am
> fully aware of its dangers. It seemed like a handy tool to me.

In a lab if you don't need to protect your script against attacks from 
outside eval() (and exec()) is fine. If the data fed to eval() is completely 
under your control (think collections.namedtuple) eval() is also fine.

Adding a public web interface on such a lab application means trouble.

> This newsgroup scares me, 

If you dare say that you are not scared enough ;)

> it appears to be for professional computer
> scientists only, the theoretical part is sometimes too much for this
> practical physicist with an old background in FORTRAN.

That sounds like you are experienced enough to say "There may be problems 
with this code, but I choose not to care about them this time -- at my own 
risk"

> Is there a better place to ask questions of this nature?

There is the tutor mailing list which is mostly geared at absolute 
beginners, does more hand-holding, and the threads are more likely to stay 
on topic. You might take a look, but with your background you are probably 
better off here.

> I am sorry, the problem is ill posed.
> 
> 'a', 'A' and so forth are my failed attempt to shorthand.
> 
> In reality the dict's keys are column names in a pandas dataframe df.
> 
> The boolean expression would therefore look like:
> 
> bool = ((df['a'] == 1) & (df['A'] == 0) |
>  (df['b'] == 1) & (df['B'] == 0) |
>  (df['c'] == 1) & (df['C'] == 0))

This is how it might look without eval():

#untested
result = functools.reduce(operator.or_, ((v == 1) & (df[k.upper()] == 0) for 
k, v in df.items() if k.islower()))

And here is an eval-based solution:

# untested
expr = "|".join(
"((df[{}] == 1) | (df[{}] == 0))".format(c, c.upper()) 
for c in df is c.islower())
result = eval(expr)

konsole.desktop
Description: application/desktop
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Mark Lawrence


On 01/08/2014 14:28, Steven D'Aprano wrote:

On Fri, 01 Aug 2014 12:45:12 +, Alex van der Spek wrote:


With a dict like so:

cond = {'a': 1, 'b': 1, 'c': 1,
 'A': 0, 'B', 0, 'C':0}

how would you make a boolean expression like this:

bool = (('a' == 1) & ('A' == 0) |
 ('b' == 1) & ('B' == 0) |
 ('c' == 1) & ('C' == 0))


That's False. It's always False, because 'a' does not equal 1, etc. Also,
you're using bitwise operators & and | rather than boolean operators.
Finally, you are shadowing the built-in bool() type, which is probably a
bad idea.

In the first case, I think you mean cond['a'] == 1 rather than just
'a'==1; in the second case, the bool operators are called "and" and "or";
and in the third case, there are many equally good names for a generic
boolean flag, like "flag".

Putting those together, I think you probably mean something like this:

flag = (
 (cond['a'] == 1 and cond['A'] == 0) or
 (cond['b'] == 1 and cond['B'] == 0) or
 (cond['c'] == 1 and cond['C'] == 0)
 )

which can be simplified to:

flag = any( cond[c] == 1 and cond[c.upper()] for c in ['a', 'b', 'c'] )



Shouldn't that be cond[c.upper()] == 0 ?

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Mark Lawrence


On 01/08/2014 16:24, Steven D'Aprano wrote:

I don't know of anyone here who is an expert in pandas, so if you ask
questions which are specific to pandas, we may run into the limits of our
knowledge. If you can find a dedicated pandas mailing list or other
forum, they may help too, but I don't know if they will be more or less
willing to explain the basics about Python.



https://mail.python.org/mailman/listinfo/pandas-dev or 
gmane.comp.python.pydata


As it's so handy see here 
http://news.gmane.org/index.php?prefix=gmane.comp.python for the all 
python lists on gmane.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Mark Lawrence


On 01/08/2014 15:26, Alex van der Spek wrote:

On Fri, 01 Aug 2014 12:45:12 +, Alex van der Spek wrote:


With a dict like so:

cond = {'a': 1, 'b': 1, 'c': 1,
 'A': 0, 'B', 0, 'C':0}

how would you make a boolean expression like this:

bool = (('a' == 1) & ('A' == 0) |
 ('b' == 1) & ('B' == 0) |
 ('c' == 1) & ('C' == 0))

The fact that lowercase and uppercase keys are stringed together with &
is intentional albeit the actual condition is a bit more tricky.

I've tried several approaches using eval() on a string built from the
dict but landed with just spelling it out literally.


Any pointers welcome.
Alex


I am sorry, the problem is ill posed.

'a', 'A' and so forth are my failed attempt to shorthand.

In reality the dict's keys are column names in a pandas dataframe df.

The boolean expression would therefore look like:

bool = ((df['a'] == 1) & (df['A'] == 0) |
  (df['b'] == 1) & (df['B'] == 0) |
  (df['c'] == 1) & (df['C'] == 0))


See this 
http://stackoverflow.com/questions/19482970/python-get-list-from-pandas-dataframe-column-headers 
to get the column names.  Combine this with Steven D'Aprano's earlier 
answer using any() and I suspect you're there.




I do know eval() lends itself to code injection but can't say I am
fully aware of its dangers. It seemed like a handy tool to me.


It strikes me as being like looking down the gun barrel to see if 
there's a bullet loaded whilst holding the trigger.  Best avoided but 
there's always the "consenting adults" approach here :)




This newsgroup scares me, it appears to be for professional computer
scientists only, the theoretical part is sometimes too much for this
practical physicist with an old background in FORTRAN.


If you're smart enough to state that your first question was ill posed, 
you're smart enough to ask questions when you don't understand.




Is there a better place to ask questions of this nature?


No :)



Alex van der Spek



--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

On Fri, 01 Aug 2014 16:50:51 +0100, Mark Lawrence wrote:

>> which can be simplified to:
>>
>> flag = any( cond[c] == 1 and cond[c.upper()] for c in ['a', 'b', 'c'] )
>>
>>
> Shouldn't that be cond[c.upper()] == 0 ?

Yes it should be, thank you!


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Peter Pearson

On 01 Aug 2014 14:26:38 GMT, Alex van der Spek  wrote:
[snip]
> This newsgroup scares me, it appears to be for professional computer 
> scientists only, the theoretical part is sometimes too much for this 
> practical physicist with an old background in FORTRAN.
>
> Is there a better place to ask questions of this nature?

Relax.  This newsgroup cheerfully deals with noobs much more noobish
than you.  It's an amazingly polite and helpful place, and many of the
regulars are top experts on Python.  Hang around, you'll like it.

-- 
To email me, substitute nowhere->spamcop, invalid->net.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

2014-08-01 Thread Dietmar Schwertberger


Am 01.08.2014 13:10, schrieb Wolfgang Keller:

Because on such operating systems, each and every application is an
entirely self-contained package that doesn't need any "packages" or
"installers" to use it.

For people who have never used such a system it's probably difficult to see
the  advantages.

Besides the easy installation, backup and replication of software the 
RISC OS

way also had the advantage that you were able to organize your
applications in folders just like other folders and files.
There was no need for separate File and Program managers.
MS never got this right. Instead, they tried to fix things later with the
start menu and finally the box to type the software name to start it ...

One effect was that under DOS/Windows people usually saved their
documents in folders per application whereas under RISC OS they
were usually grouped by content/project.


When it came to usability, RISC OS had many advantages over the
other systems, e.g.
 - real drag'n'drop for loading *and* saving of files/selections
 - drag'n'drop also for transfer between applications
 - a standard vector graphics format that all applications supported
   (and for which an application was provided by default with the OS)
 - good font display (still better than e.g. MS Windows today)
 - three mouse buttons for select/menu/adjust
 - no menu bars
 - the icon bar for running applications, drives, shares and other 
resources

 - consistent, orthogonal & logical user interfaces instead of assistants
   and wizards for each and every task
 - complete programmers reference manual



Regards,

Dietmar

--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

2014-08-01 Thread Michael Torrie

On 08/01/2014 08:39 AM, Chris Angelico wrote:
> The installer has basically three choices.
> 1) Install libnettle inside the application directory
> 2) Install libnettle to some system library directory
> 3) Don't install libnettle, and demand that someone else (perhaps the
> user, or the system package manager) install it.
> 
> Option 1 results in duplications. (Unless one application is allowed
> to access a library in another application's directory, which is a
> HORRIBLE mess.) Option 2 is exactly what you're complaining about,
> scattering files all over the FS. And option 3 is what package
> managers are for. What are you advocating?

Option 1 also is a huge security hole.  A prime example of this was the
so-called heartbleed bug.  In such a model, each app that distributes
openssl in the app bundle has to be updated or it is at risk.  This
turns out to be a huge vulnerability.

-- 
https://mail.python.org/mailman/listinfo/python-list

Creating a 2s compliment hex string for negitive numbers

2014-08-01 Thread bSneddon

I need to calculate an error correction code for an old protocol.

I calculate the integer 4617 and want to code the 2s compliment in ASCII
hex EDF7.  When issue the following.
>>> hex(-4617)
'-0x1209'   

Does anyone know a clean way to get to the desired results?   My ECC will always
be 16 bit (4 nibble) hex number.

Bill

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Creating a 2s compliment hex string for negitive numbers

2014-08-01 Thread MRAB


On 2014-08-01 21:35, bSneddon wrote:

I need to calculate an error correction code for an old protocol.

I calculate the integer 4617 and want to code the 2s compliment in ASCII
hex EDF7.  When issue the following.

hex(-4617)

'-0x1209'

Does anyone know a clean way to get to the desired results?   My ECC will always
be 16 bit (4 nibble) hex number.


Use a bitwise AND:

>>> hex(-4617 & 0x)
'0xedf7'

--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

2014-08-01 Thread MRAB


On 2014-08-01 18:16, Dietmar Schwertberger wrote:

Am 01.08.2014 13:10, schrieb Wolfgang Keller:

Because on such operating systems, each and every application is an
entirely self-contained package that doesn't need any "packages" or
"installers" to use it.

For people who have never used such a system it's probably difficult to see
the  advantages.

Besides the easy installation, backup and replication of software the
RISC OS
way also had the advantage that you were able to organize your
applications in folders just like other folders and files.
There was no need for separate File and Program managers.
MS never got this right. Instead, they tried to fix things later with the
start menu and finally the box to type the software name to start it ...

One effect was that under DOS/Windows people usually saved their
documents in folders per application whereas under RISC OS they
were usually grouped by content/project.


When it came to usability, RISC OS had many advantages over the
other systems, e.g.
   - real drag'n'drop for loading *and* saving of files/selections
   - drag'n'drop also for transfer between applications
   - a standard vector graphics format that all applications supported
 (and for which an application was provided by default with the OS)
   - good font display (still better than e.g. MS Windows today)
   - three mouse buttons for select/menu/adjust
   - no menu bars
   - the icon bar for running applications, drives, shares and other
resources
   - consistent, orthogonal & logical user interfaces instead of assistants
 and wizards for each and every task
   - complete programmers reference manual


I'd heard people say how user-friendly Apple Macs were, but when I got
to use one I was somewhat disappointed.

When opening files, it used old-fashioned dialog boxes like RISC OS's
precursor from several years earlier. In RISC OS, if I had a directory
window open, I could save to it with a simple drag-and-drop, but in
MacOS, even if I had a directory window open, I had to navigate to the
directory in the Save dialog. (OK, not quite true, because of a
3rd-party extension called "Click There It Is".)

And don't mention the menu bar across the top, separated from the
window to which it belonged.

Or the way that clicking on any window of an application or the Finder
brought not only it but also all of the its siblings to the front. On
RISC OS, windows came to the front only when *I* wanted them to.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Creating a 2s compliment hex string for negitive numbers

2014-08-01 Thread bSneddon

On Friday, August 1, 2014 4:47:20 PM UTC-4, MRAB wrote:
> On 2014-08-01 21:35, bSneddon wrote:
> 
> > I need to calculate an error correction code for an old protocol.
> 
> >
> 
> > I calculate the integer 4617 and want to code the 2s compliment in ASCII
> 
> > hex EDF7.  When issue the following.
> 
>  hex(-4617)
> 
> > '-0x1209'
> 
> >
> 
> > Does anyone know a clean way to get to the desired results?   My ECC will 
> > always
> 
> > be 16 bit (4 nibble) hex number.
> 
> >
> 
> Use a bitwise AND:
> 
> 
> 
>  >>> hex(-4617 & 0x)
> 
> '0xedf7'

Thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

2014-08-01 Thread Terry Reedy


On 8/1/2014 11:39 AM, Marko Rauhamaa wrote:

Chris Angelico :


but hash(x) == hash(y) does NOT imply that x == y.


Hello, pigeonhole principle :) If this were false - that is, if equal
hashes DID imply equal objects - it would be necessary to completely
encode an object's state in its hash, and hashes would be impossibly
large. This would, in fact, destroy their value completely.


Well, modern computing assumes precisely:

hash(x) == hash(y) => x == y


Assuming that a false statement is true does not make it true, and can 
and has gotten 'computing' into trouble.



That principle is at play with strong authentication (HTTPS et al),
version control (git et al),


The principle for these applications is

stronghash(x) == stronghash(y) => x == y with probability 1 (or 
indistinguishable from 1).


For mercurial, with no treat model, a 160 bit hash is used. Internet 
applications need more bits and carefully vetted algorithms to hopefully 
make the actual principle true.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: dict to boolean expression, how to?

2014-08-01 Thread Terry Reedy


On 8/1/2014 11:02 AM, Steven D'Aprano wrote:

On Fri, 01 Aug 2014 09:32:36 -0400, Roy Smith wrote:


In article <53db95e6$0$29986$c3e8da3$54964...@news.astraweb.com>,
  Steven D'Aprano  wrote:


eval is almost never the right solution to any problem, and in the very
few exceptions, it needs careful handling by an expert to ensure you're
not introducing serious security bugs.


Corollary to that rule: All the people who are smart enough to actually
understand how to use eval() safety, are also smart enough to know not
to use it.


Eval is a specialized version of exec. Everything you do on a computer 
is evaluating and executing code. Here is a highly simplified version of 
a Python interpreter:


user_program = get_user_python_code()
__main__ = make_global_namespace()
exec(user_program, __main__, __main__)

CPython codes an expansion of the above in C. Idle uses an expansion of 
the above to do the same thing.


The threat model is executing code from someone who does not have 
physical access to a machine to just run code and who should not be trusted.



... smart enough to know WHEN to use it (which is *rarely*).

That's in production code, of course. There's nothing wrong with using
eval in the interactive interpreter for quick and dirty exploration. But
even then, I find that it's usually easier to write a line or two of
Python code to process something than to try using eval.

There are uses for eval (or exec), even if production code. See
collections.namedtuple, doctest, and timeit, for example. In the first
place, namedtuple takes extra care to sanitise the data being used. In
the case of doctest and timeit, the whole point of them is to run trusted
code. If you can't trust your own code that you're timing, what can you
trust?


Exactly. If someone can start Python or Idle on a machine, they can 
start Windows Explorer and Command Prompt. Idle exec's user code because 
emulating the interactive interpreter is part of its purpose. It 
sometimes evals expression within user code in response to user requests.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

2014-08-01 Thread Gregory Ewing


Chris Angelico wrote:

The installer has basically three choices.
1) Install libnettle inside the application directory
2) Install libnettle to some system library directory
3) Don't install libnettle, and demand that someone else (perhaps the
user, or the system package manager) install it.

Option 2 is exactly what you're complaining about,
scattering files all over the FS.


Not really. On MacOSX, if you installed a shared library
called libnettle, *all* the files relating to it
would be kept in one directory called Nettle.framework
(either in /Library/Frameworks or ~/Library/Frameworks
depending on whether it's system-wide or for a single user).

MacOSX doesn't currently have an automatic dependency
manager, but if it did, things would still be a lot neater
and tidier than they are in Linux or Windows, where what
is conceptually a single object (a package) gets split up
and its parts scattered around several obscure locations.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

On Sat, Aug 2, 2014 at 6:22 AM, Michael Torrie  wrote:
> On 08/01/2014 08:39 AM, Chris Angelico wrote:
>> The installer has basically three choices.
>> 1) Install libnettle inside the application directory
>> 2) Install libnettle to some system library directory
>> 3) Don't install libnettle, and demand that someone else (perhaps the
>> user, or the system package manager) install it.
>>
>> Option 1 results in duplications. (Unless one application is allowed
>> to access a library in another application's directory, which is a
>> HORRIBLE mess.) Option 2 is exactly what you're complaining about,
>> scattering files all over the FS. And option 3 is what package
>> managers are for. What are you advocating?
>
> Option 1 also is a huge security hole.  A prime example of this was the
> so-called heartbleed bug.  In such a model, each app that distributes
> openssl in the app bundle has to be updated or it is at risk.  This
> turns out to be a huge vulnerability.

More generally, that's exactly what Steven said about needing every
package to update before you can confidently say it's updated. But
that's also the greatest feature of the first option: you can't break
this application by upgrading that library, because only upgrading the
application (which hopefully will have been tested by the author) will
upgrade the library it uses.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

On Sat, Aug 2, 2014 at 9:14 AM, Gregory Ewing
 wrote:
> Chris Angelico wrote:
>>
>> The installer has basically three choices.
>> 1) Install libnettle inside the application directory
>> 2) Install libnettle to some system library directory
>> 3) Don't install libnettle, and demand that someone else (perhaps the
>> user, or the system package manager) install it.
>>
>> Option 2 is exactly what you're complaining about,
>> scattering files all over the FS.
>
>
> Not really. On MacOSX, if you installed a shared library
> called libnettle, *all* the files relating to it
> would be kept in one directory called Nettle.framework
> (either in /Library/Frameworks or ~/Library/Frameworks
> depending on whether it's system-wide or for a single user).
>
> MacOSX doesn't currently have an automatic dependency
> manager, but if it did, things would still be a lot neater
> and tidier than they are in Linux or Windows, where what
> is conceptually a single object (a package) gets split up
> and its parts scattered around several obscure locations.

That's fine if I explicitly install libnettle - that's the third
option. What happens if I install Foo's Cool Chat App, and FCCA uses
libnettle to encrypt conversations? Is FCCA allowed to install
libnettle into /Library/Frameworks? If so, its files will be split
between there and its own app directory.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

On Sat, Aug 2, 2014 at 7:42 AM, Terry Reedy  wrote:
> For mercurial, with no treat model, a 160 bit hash is used. Internet
> applications need more bits and carefully vetted algorithms to hopefully
> make the actual principle true.

Ditto git, which also has no threat model. I don't know of any
situation in HTTPS that has this, but the classic concept of hashed
passwords (quite independent of HTTPS) basically says "if I take an
arbitrary/random salt and combine it with your password, and hash
that, then the probability of a hash collision involving the same salt
and a different password approaches 0". And any time "approaches 0" is
provably false (or doesn't approach 0 closely enough), you have weak
passwords, which is why it's a really bad idea to use MD5 passwording.
Ergo MD5 is not (any more, at least) a "carefully vetted algorithm".
(That said, though, I will happily use md5sum across a huge pile of
files to find duplicates. It's a lot quicker than sha*sum, and I don't
have reason to expect malicious hash collisions on my own hard drive.
Plus, I can always just check some other way.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

2014-08-01 Thread Gregory Ewing


MRAB wrote:

I'd heard people say how user-friendly Apple Macs were, but when I got
to use one I was somewhat disappointed.


Well, they were compared to MS-DOS and the like, which was
all that was within reach of the general public when the
first Mac appeared. RISCOS came along somewhat later.


in MacOS, even if I had a directory window open, I had to navigate to the
directory in the Save dialog.


Yes, that was annoying. It wasn't a problem to begin with,
because the original Mac was strictly single-tasking --
you couldn't *have* a directory window and an application
open at the same time. And all your files were on floppies
in a flat file system -- folders only existed in the
Finder's imagination -- so the only real choice to be
made when saving a file was "which disk do I put it on".

When multitasking, hard disks and hierarchical file
systems came along, there was an opportunity for a
rethink, but it never really happened.

Things are somewhat better in MacOSX, where you can drag
a folder from a Finder window onto a file dialog to take
you there, but there is still more of a distinction between
Finder windows and save dialogs than there needs to be.


And don't mention the menu bar across the top, separated from the
window to which it belonged.


That seems to be a matter of taste. There are some
advantages to the menu-bar-at-top model. It's an easier
target to hit, because you can just flick the mouse up
to the top. It only takes up space once, instead of
once per window. It makes it possible for an app to
be running without having any windows, and still be
able to interact with it.


Or the way that clicking on any window of an application or the Finder
brought not only it but also all of the its siblings to the front.


MacOSX has fixed that one, thankfully. Only the window
you click comes to the front, now.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and IDEs [was Re: Python 3 is killing Python]

On Sat, Aug 2, 2014 at 10:00 AM, Gregory Ewing
 wrote:
> MRAB wrote:
>> in MacOS, even if I had a directory window open, I had to navigate to the
>> directory in the Save dialog.
>
> Yes, that was annoying. It wasn't a problem to begin with,
> because the original Mac was strictly single-tasking --
> you couldn't *have* a directory window and an application
> open at the same time. And all your files were on floppies
> in a flat file system -- folders only existed in the
> Finder's imagination -- so the only real choice to be
> made when saving a file was "which disk do I put it on".

Okay, so it was like DOS 1.0...

> When multitasking, hard disks and hierarchical file
> systems came along, there was an opportunity for a
> rethink, but it never really happened.

... and it didn't get improved when it grew directories like DOS 2.0
did. It's like how the default DOS prompt is actually $N$G when $P$G
is a lot more useful, except that changing the default prompt is
pretty easy and applies globally (not to mention that you might well
want to enhance the prompt beyond just $P$G).

>> And don't mention the menu bar across the top, separated from the
>> window to which it belonged.
>
> That seems to be a matter of taste. There are some
> advantages to the menu-bar-at-top model. It's an easier
> target to hit, because you can just flick the mouse up
> to the top. It only takes up space once, instead of
> once per window. It makes it possible for an app to
> be running without having any windows, and still be
> able to interact with it.

Downside: It separates (graphically and logically) a window from its
menu bar. The "easier target for the mouse" argument is valuable ONLY
when you use the mouse to access the menu bar. If you use the keyboard
(and take advantage of mnemonic letters), it's much more useful to
have the menu bar attached to its window. In the rare case of an app
that runs without any windows, incidentally, how do you tell the
system that you want that app's menu bar instead of (say) Finder,
which comes up when you click on the desktop?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

Marko Rauhamaa wrote:

> Chris Angelico :
> 
>>> but hash(x) == hash(y) does NOT imply that x == y.
>>
>> Hello, pigeonhole principle :) If this were false - that is, if equal
>> hashes DID imply equal objects - it would be necessary to completely
>> encode an object's state in its hash, and hashes would be impossibly
>> large. This would, in fact, destroy their value completely.
> 
> Well, modern computing assumes precisely:
> 
>hash(x) == hash(y) => x == y

I really don't think that the behaviour of Python's built-in hash() function
is of fundamental concern to all of modern computing. Python's hash()
function is of fundamental concern to Python dicts, and that's all.

> That principle is at play with strong authentication (HTTPS et al),
> version control (git et al),

git is not even written in Python. HTTPS is a protocol, so it could be
written in Python, but typically isn't. Neither of them use Python's
hash(), because the people writing these applications aren't insane.
Collisions in Python's hash() are trivially easy to find, and the existence
of such collisions is well-known.

Perhaps you're talking about cryptographically strong hash functions, of
which there are a few? If so, a brief note that you were changing the
subject would be appreciated :-)

The pigeonhole principle applies to crypto hash functions too, and there is
active research into creating newer and stronger hash functions as the
older ones are broken. For instance, md4, md5, md2 and sha-0 hash functions
have all be broken, and shouldn't be used for anything where security is of
concern.

> A handful of bits uniquely identify an object regardless of its size.

No. The principle being applied is not that a handful of bits *uniquely*
identifies an object, since that is clearly and obviously untrue: consider
objects of size 1K, there are 2**1024 possible such objects. I don't know
what "a handful" of bits is, but let's say 128 bits, so there are 2**128
possible hashes. It doesn't take much mathematics to see that you cannot
*uniquely* hash 2**1024 objects into only 2**128 checksums without quite a
few collisions.

The actual principle being applied is that, with a cryptographically strong
hash function, a handful of bits can *practically* identify an object
regardless of its size. Let's take that 128 bit checksum again: that means
that there are 340282366920938463463374607431768211456 possible checksums.
If the input files are unpredictably hashed over that entire range, on
average you would have to generate half that number of hashes before
stumbling across a collision purely by chance. Half of 2**128 is still a
rather big number: if you generated a million hashes a second, on average
it would take you considerably more than a billion years to accidentally
hit a collision.

For practical purposes, that's good enough.

But note that there are numerous provisos there, including:

- It relies on the hash function generating a large enough handful of bits.
An 8 bit checksum only has 2**8 = 256 unique values, so you would find a
collision pretty quickly.

- It relies on the mapping of original object to checksum being randomly
distributed. If they are not randomly distributed, then there will be sets
of objects which collide more often than expected. For instance, suppose
that every object containing a 0 byte hashed to the same value. Then, on
average, you would not have to wait very long before finding your first
collision.

- It relies on the checksum being unpredictable, to prevent substitution
attacks: you're expecting object x with checksum a, but somebody
substitutes object y with checksum a instead.

Python's hash() function is not cryptographically strong and makes no claim
to be.

-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error

On Sat, Aug 2, 2014 at 11:47 AM, Steven D'Aprano
 wrote:
> - It relies on the checksum being unpredictable, to prevent substitution
> attacks: you're expecting object x with checksum a, but somebody
> substitutes object y with checksum a instead.

Note that this requirement is only an issue when there are actual
attacks involved. In many cases, hashing is used to detect either
errors in transfer (eg truncated files) or meaningfully different
files (eg MP3s of different songs). A collision between those won't be
the result of someone deliberately crafting a file with the right
checksum; it'll happen only by chance, so md5sum can be safely used
there.

But if you're trying to use this to prove that a file was downloaded
correctly, you do need to worry about that. If I say that you can
download the binary of my program from , and that the MD5
checksum is 123456...DEF, then someone could do a DNS hack (cache
poisoning, proxy interference, whatever) to capture your attempted
download, and send you instead to his own server, where he has a
carefully crafted binary that does everything mine does, plus it tells
him all your passwords - and it has arbitrary junk buried in it to
make sure the MD5 sum matches. So an MD5 checksum is broken for
anything from the internet, but is quite usable for certain specific
cases.

There is one aspect of the unpredictability that's important even in
the simple cases, though, and that's the avalanche effect. If anything
changes in the file, the whole hash should completely and arbitrarily
change. That means you don't need to stare at the whole hash, trying
to see if that 8 became a 0; any change to the file will probably make
the first few digits visibly different, so it's easily obvious that
the hash has changed.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug with help (was Re: Getting a list of all modules)

2014-08-01 Thread Akira Li

Mark Lawrence  writes:

> On 31/07/2014 19:55, Akira Li wrote:
>> Steven D'Aprano  writes:
>>
>>> I'm looking for a programmatic way to get a list of all Python modules
>>> and packages. Not just those already imported, but all those which
>>> *could* be imported.
>> ...
>>> Is this problem already solved? Can anyone make any suggestions?
>>
>> Look at how `help('modules')` is implemented. Though it crashes on my
>> system.
>>
>
> Have you reported this at bugs.python.org or is there already an issue
> for the problem that you see?

It is this issue for python2.7:

https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/896836

python3 doesn't crash with currently installed packages.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: converting ISO8601 date and time string representations to datetime

2014-08-01 Thread Akira Li

Wolfgang Maier  writes:

> On 08/01/2014 01:30 AM, Roy Smith wrote:
>> In article ,
>>   Albert-Jan Roskam  wrote:
>>
 In article ,
 Wolfgang Maier  wrote:

> Hi,
> I'm trying to convert ISO8601-compliant strings representing dates or
> dates and times into datetime.datetime objects.

 https://pypi.python.org/pypi/iso8601
>>>
>>> Yikes, what a regex. It must have been painstaking to get that right.
>>> https://bitbucket.org/micktwomey/pyiso8601/src/2bd28b5d6cd2481674a8b0c54a8bba6
>>> 4ab775f81/iso8601/iso8601.py?at=default
>>
>> It is a thing of beauty.
>>
>
> No wonder I found it hard to write something that seemed bulletproof !

It seems it supports only some custom subset of ISO 8601. There is rfc
3339 [1] that describes a profile of the ISO 8601 standard.

rfc 3339 combines human readability with the simplicity of machine parsing.

[1] http://tools.ietf.org/html/rfc3339


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

eval [was Re: dict to boolean expression, how to?]

On Fri, 01 Aug 2014 17:44:27 +0200, Peter Otten wrote:

> Alex van der Spek wrote:
> 
> 
>> I do know eval() lends itself to code injection but can't say I am
>> fully aware of its dangers. It seemed like a handy tool to me.
> 
> In a lab if you don't need to protect your script against attacks from
> outside eval() (and exec()) is fine. If the data fed to eval() is
> completely under your control (think collections.namedtuple) eval() is
> also fine.

I'm not entirely happy with describing eval as "fine", even when you're 
not concerned with security. There are at least two problems with eval 
that should make it a tool of last resort:

* its less direct, typically more convoluted, which makes it 
  hard to read, harder to write, and harder to reason about;

* it's slower than running code directly (in my experience, 
  about ten times slower).

If you have the choice between running code directly, or running eval() 
or exec() that runs the same code, you should nearly always prefer to run 
the code directly.

y = x + 1  # Yes.
y = eval("x + 1")  # No.
y = eval("eval('x + 1')")  # Good grief what are you thinking???

Now obviously nobody sensible is going to use eval in production code for 
such simple expressions as "x+1", but the same principle applies even for 
more complex examples. If you can write the code once, as source code, 
it's usually better than generating the source code at runtime and 
running it with eval or exec.

Consider the namedtuple implementation in the standard library. There's a 
lot of criticism of it, some of it justified. It uses exec extensively, 
which means the code is dominated by a giant string template. This 
defeats your editor's syntax colouring, makes refactoring harder, and 
makes how the namedtuple works rather less understandable. It seems to me 
that it's only generating the __new__ method which genuinely needs to use 
exec, the rest of the namedtuple could and should use just an ordinary 
class object (although I concede that some of this is just a matter of 
personal taste).

Raymond Hettinger's original, using exec for the entire inner class:

http://code.activestate.com/recipes/500261-named-tuples/

My refactoring, with the bare minimum use of exec necessary:

https://code.activestate.com/recipes/578918-yet-another-namedtuple/

[...]
>> bool = ((df['a'] == 1) & (df['A'] == 0) |
>>  (df['b'] == 1) & (df['B'] == 0) |
>>  (df['c'] == 1) & (df['C'] == 0))
>  
> This is how it might look without eval():
> 
> #untested
> result = functools.reduce(operator.or_, ((v == 1) & (df[k.upper()] == 0)
> for k, v in df.items() if k.islower()))

For those who agree with Guido that reduce makes code unreadable:

result = True
for key in df:
if key.islower():
result = result or (df[key] == 1 and df[key.upper()] == 0)

Or if you insist on a single expression:

result = any(df[k] == 1 and df[k.upper()] == 0 for k in df if k.islower())

> And here is an eval-based solution:
> 
> # untested
> expr = "|".join(
> "((df[{}] == 1) | (df[{}] == 0))".format(c, c.upper()) for c in df
> is c.islower())
> result = eval(expr)

I really don't believe that there is any benefit to that in readability, 
power, flexibility, or performance. Also, you're using bitwise operators 
instead of shortcut bool operators. Any reason why?

-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3 is killing Python

2014-08-01 Thread Glenn Linderman

On 7/16/2014 7:27 AM, Frank Millman wrote:

I just tried an experiment in my own project. Ned Batchelder, in his
Pragmatic Unicode presentation, http://nedbatchelder.com/text/unipain.html,
suggests that you always have some unicode characters in your data, just to
ensure that they are handled correctly. He has a tongue-in-cheek example
which spells the word PYTHON using various exotic unicode characters. I used
this to populate a field in my database, to see if it would display in my
browser-based client.

The hardest part was getting it in. There are 6 characters, but utf-8
requires 16 bytes to store it -

b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4'.decode('utf-8')

However, that was it. Without any changes to my program, it read it from the
database and displayed it on the screen. IE8 could only display 2 out of the
6 characters correctly, and Chrome could display 5 out of 6, but that is a
separate issue. Python3 handled it perfectly.

wrapping the above in a print(), on Windows, I get:

Traceback (most recent call last):
File "D:\my\py\python-utf8.py", line 1, in
print(b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4'.decode('utf-8'))
File "C:\Python33\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position
0-5: character maps to

So Python3 doesn't handle it perfectly on Windows. And I saw someone
blame the Windows console for that... but the Windows console can
properly display all those characters if the proper APIs are used. The
bug is 7 years old: http://bugs.python.org/issue1602 and hasn't been
fixed, although the technology for fixing it is available, and various
workarounds (with limitations) have been available for 5 years, and
patches have been available for 3 years that work pretty good. However,
just a few days ago, 26 July 2014, Drekin had an insight that may
possibly lead to a patch that will work well enough to be integrated
into some future version of Python... I hope he follows up on it. This
is a serious limitation, and it is, and always has been, a bug in Python
3 Unicode handling on Windows.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Dict when defining not returning multi value key error