Output not defined

[jonas . thornvall]

Can any web programmer tell me why output is not defined?
I know this is python group but my code is so rudimentary that probably any 
sufficient good webprogrammer can figure it out, but i am lost."I've tried 
comp.lang.javascript"

Problem is I've looked upon this for a week or two, and i just can't figure out 
why output is not defined. 

My hangup is probably because it works without problem when i run it in under 
windows XP browser, and that make me confused."I started program it in XP but 
now i want to transition into more modern browsers, Linux and windows 10" 

https://3d16zci4vuzyxt8hxvovrg-on.drv.tw/Website/midi.html 

Any help or even suggestion would be greatly appreciated. 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with OrderedDict - progress report

[Gregory Ewing]

Chris Angelico wrote:

It is an error to mutate the dictionary *and then continue to iterate over it*.


But if you're processing the last key, and you add one so
that it's no longer the last key, what should happen?

My feeling is that this should be an error, because it's
not clear whether iteration should stop at that point
or not.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with OrderedDict - progress report

[Chris Angelico]

On Fri, Jun 1, 2018 at 5:54 PM, Gregory Ewing
 wrote:
> Chris Angelico wrote:
>>
>> It is an error to mutate the dictionary *and then continue to iterate over
>> it*.
>
>
> But if you're processing the last key, and you add one so
> that it's no longer the last key, what should happen?
>
> My feeling is that this should be an error, because it's
> not clear whether iteration should stop at that point
> or not.
>

Right, but if you 'break' immediately after a mutation, that isn't
continuing to iterate. Even though you're inside the loop, there's no
further action taken to process the loop, and no problem.

If it so happens that mutating the last one and then continuing the
loop doesn't cause a problem, it's because you got lucky, not because
the language guarantees it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Indented multi-line strings

[Gregory Ewing]

Here's my take on what an indented multi-line string
syntax should look like.

string foo:
|  This is a multi-line indented string declaration.
|  Note that it's a statement, not an expression. In
|  this example, each line of the final string starts
|  with two spaces.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with OrderedDict - progress report

[INADA Naoki]

On Fri, Jun 1, 2018 at 4:56 PM Gregory Ewing 
wrote:

> Chris Angelico wrote:
> > It is an error to mutate the dictionary *and then continue to iterate
> over it*.
>
> But if you're processing the last key, and you add one so
> that it's no longer the last key, what should happen?
>
> My feeling is that this should be an error, because it's
> not clear whether iteration should stop at that point
> or not.
>
> --
> Greg
> --
> https://mail.python.org/mailman/listinfo/python-list
>

​Regardless how CPython works, it's prohibited, and behavior is
**undefined**.​
There are no "what should happen".

Python interpreters may or may not raise error.  And any error
(RuntimeError,
MemoryError, interpreter freeze) may happen.

Python programmer shouldn't rely on the behavior.

​Regards,​

-- 
INADA Naoki  
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with OrderedDict - progress report

[Frank Millman]

"Gregory Ewing"  wrote in message news:fnccd8ff3s...@mid.individual.net...


Chris Angelico wrote:
> It is an error to mutate the dictionary *and then continue to iterate 
> over it*.


But if you're processing the last key, and you add one so
that it's no longer the last key, what should happen?

My feeling is that this should be an error, because it's
not clear whether iteration should stop at that point
or not.



I agree. A normal dictionary raises an exception. I think that an 
OrderedDict should do so as well.


In fact this was the reason for my intermittent error. I was adding a key 
while iterating, which was an error. But if I happened to be on the last key 
at the time, it did not raise an exception. If I was on a previous key, it 
did raise an exception.


Frank



--
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with OrderedDict - progress report

[Steven D'Aprano]

On Thu, 31 May 2018 16:37:39 +0200, Frank Millman wrote:

[...]
> Agreed, but my gut feel, and the following example, suggest that when
> processing the last key in a dictionary while iterating over it, you
> have not yet stopped iterating.
> 
 d = {}
 d[1] = 'one'
 d[2] = 'two'

Before Python 3.7, regular dicts are unordered. Just because you add 2 
last, doesn't mean that 2 is the last item. The position of each item is 
unpredictable, and 2 could end up in the first position.

In this example:


 for k in d:
> ...   if k == 2:
> ... d[3] = 'three'
> ...
> Traceback (most recent call last):
>   File "", line 1, in 
> RuntimeError: dictionary changed size during iteration

just because k == 2 does not mean this matches on the last iteration. It 
could be the first.


> OrderedDict seems to behave differently in this regard -

Neither OrderedDict nor dict guarantee to detect all mutations.


-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

pex console_scripts unavailable

[WaffleSouffle]

I'd like to embed console_scripts in a pex file and be able to use them by 
specifying them when invoking the pex file.

The details are...
Python module with the following file structure:

.
├── mefoo
│   ├── __init__.py
│   ├── bar.py
│   └── command_line
│   ├── __init__.py
│   ├── cli1.py
│   └── cli2.py
└── setup.py

Stripped down `setup.py` is 

#!/usr/bin/env python3

import setuptools

setuptools.setup(
name='mefoo',
version='0.0.1',
entry_points={
'console_scripts' : [
'mefoo1 = mefoo.command_line.cli1:main',
'mefoo2 = mefoo.command_line.cli2:main'
]
},
scripts=['scripts/mefooscript1'],
packages=setuptools.find_packages()
)

Build a wheel with `python3 setup.py sdist bdist_wheel` which outputs to `dist` 
directory.
Install wheel with `pip install --no-index dist/mefoo-0.0.1-py3-none-any.whl`
This correctly creates my `console_script` entry point so from command prompt I 
can call:

> mefoo1
Could not open mefoo1 in the environment [./themefoo.pex]: [Errno 2] No 
such file or directory: 'mefoo1'

Then I move on and create a pex file:

pex --python=python3 --disable-cache --no-index --find-links=./dist mefoo 
--output-file themefoo.pex -v

This creates a pex file, but attempts to use the `console_script` entry point 
fails.

> ./themefoo.pex mefoo1

*How do I make this work ?*

What I can do is specify the `console_script` entry point as an entry point to 
pex


pex --python=python3 --disable-cache --no-index --find-links=./dist -c 
mefoo1 mefoo --output-file themefoo.pex -v


Unzipping the pex file doesn't shows it doesn't contain any extra archived 
content, so I'm guessing there's some extra config embedded in the file itself 
which is interpreted appropriately.

The [wheel documentation](https://wheel.readthedocs.io/en/stable/) talks about 
`install-scripts`, but I cannot work out how to use it.
The docs mention:
> If the scripts are needed, use a real installer like pip. The wheel
> tool python -m wheel install-scripts package [package …] can also be
> used at any time to call setuptools to write the appropriate scripts
> wrappers.

I'm not sure what writing the appropriate script wrappers means.

There's some discussion leading to `install-scripts` being implemented 
[here](https://github.com/pypa/pip/issues/1067), and I had a look at the code 
but couldn't work out whether this addresses my problem.
-- 
https://mail.python.org/mailman/listinfo/python-list

pex console_scripts unavailable

[WaffleSouffle]

Python module with the following file structure:

.
├── mefoo
│   ├── __init__.py
│   ├── bar.py
│   └── command_line
│   ├── __init__.py
│   ├── cli1.py
│   └── cli2.py
└── setup.py

Stripped down `setup.py` is 

#!/usr/bin/env python3

import setuptools

setuptools.setup(
name='mefoo',
version='0.0.1',
entry_points={
'console_scripts' : [
'mefoo1 = mefoo.command_line.cli1:main',
'mefoo2 = mefoo.command_line.cli2:main'
]
},
scripts=['scripts/mefooscript1'],
packages=setuptools.find_packages()
)

Build a wheel with `python3 setup.py sdist bdist_wheel` which outputs to `dist` 
directory.
Install wheel with `pip install --no-index dist/mefoo-0.0.1-py3-none-any.whl`
This correctly creates my `console_script` entry point so from command prompt I 
can call:

> mefoo1

Then I move on and create a pex file:

pex --python=python3 --disable-cache --no-index --find-links=./dist mefoo 
--output-file themefoo.pex -v

This creates a pex file, but attempts to use the `console_script` entry point 
fails.

> ./themefoo.pex mefoo1
Could not open mefoo1 in the environment [./themefoo.pex]: [Errno 2] No 
such file or directory: 'mefoo1'

*How do I make this work ?*

What I can do is specify the `console_script` entry point as an entry point to 
pex


pex --python=python3 --disable-cache --no-index --find-links=./dist -c 
mefoo1 mefoo --output-file themefoo.pex -v


Unzipping the pex file doesn't shows it doesn't contain any extra archived 
content, so I'm guessing there's some extra config embedded in the file itself 
which is interpreted appropriately.

The [wheel documentation](https://wheel.readthedocs.io/en/stable/) talks about 
`install-scripts`, but I cannot work out how to use it.
The docs mention:
> If the scripts are needed, use a real installer like pip. The wheel
> tool python -m wheel install-scripts package [package …] can also be
> used at any time to call setuptools to write the appropriate scripts
> wrappers.

I'm not sure what writing the appropriate script wrappers means.

There's some discussion leading to `install-scripts` being implemented 
[here](https://github.com/pypa/pip/issues/1067), and I had a look at the code 
but couldn't work out whether this addresses my problem.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Barry Scott]

On Thursday, 31 May 2018 14:03:01 BST Marko Rauhamaa wrote:
> Chris Angelico :
> > On Thu, May 31, 2018 at 10:03 PM, Marko Rauhamaa  wrote:
> >> This surprising exception can even be a security issue:
> >>>>> os.path.exists("\0")
> >>
> >>Traceback (most recent call last):
> >>  File "", line 1, in 
> >>  File "/usr/lib64/python3.6/genericpath.py", line 19, in exists
> >>  
> >>os.stat(path)
> >>
> >>ValueError: embedded null byte
> > 
> > [...]
> > 
> > A Unix path name cannot contain a null byte, so what you have is a
> > fundamentally invalid name. ValueError is perfectly acceptable.
> 
> At the very least, that should be emphasized in the documentation. The
> pathname may come from an external source. It is routine to check for
> "/", "." and ".." but most developers (!?) would not think of checking
> for "\0". That means few test suites would catch this issue and few
> developers would think of catching ValueError here. The end result is
> unpredictable.

I think the reason for the \0 check is that if the string is passed to the 
operating system with the \0 you can get surprising results.

If \0 was not checked for you would be able to get True from:

os.file.exists('/home\0ignore me')

This is because a posix system only sees '/home'.
Surely ValueError is reasonable?

Once you know that all of the string you provided is given to the operating 
system it can then do whatever checks it sees fit to and return a suitable 
result.

As an aside Windows has lots of special filenames that you have to know about 
if you are writting robust file handling. AUX, COM1, \this\is\also\COM1 etc.

Barry

> 
> 
> Marko




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Paul Moore]

On 1 June 2018 at 13:15, Barry Scott  wrote:
> I think the reason for the \0 check is that if the string is passed to the
> operating system with the \0 you can get surprising results.
>
> If \0 was not checked for you would be able to get True from:
>
> os.file.exists('/home\0ignore me')
>
> This is because a posix system only sees '/home'.

So because the OS API can't handle filenames with \0 in (because that
API uses null-terminated strings) Python has to special case its
handling of the check. That's fine.

> Surely ValueError is reasonable?

Well, if the OS API can't handle filenames with embedded \0, we can be
sure that such a file doesn't exist - so returning False is
reasonable.

> Once you know that all of the string you provided is given to the operating
> system it can then do whatever checks it sees fit to and return a suitable
> result.

As the programmer, I don't care. The Python interpreter should take
care of that for me, and if I say "does file 'a\0b' exist?" I want an
answer. And I don't see how anything other than "no it doesn't" is
correct. Python allows strings with embedded \0 characters, so it's
possible to express that question in Python - os.path.exists('a\0b').
What can be expressed in terms of the low-level (C-based) operating
system API shouldn't be relevant.

Disclaimer - the Python "os" module *does* expose low-level
OS-dependent functionality, so it's not necessarily reasonable to
extend this argument to other functions in os. But it seems like a
pretty solid argument in this particular case.

> As an aside Windows has lots of special filenames that you have to know about
> if you are writting robust file handling. AUX, COM1, \this\is\also\COM1 etc.

I don't think that's relevant in this context.

Paul
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do I list only the methods I define in a class?

[bruceg113355]

On Thursday, May 31, 2018 at 10:18:53 PM UTC-4, bob gailer wrote:
> On 5/31/2018 3:49 PM, bruceg113...@gmail.com wrote:
>  > How do I list only the methods I define in a class?
> Here's a class with some method, defined in various ways:
> 
>  >>> class x():
> ... a=3
> ... def f():pass
> ... g = lambda: None
> ...
> 
>  >>> l=[v for v in x.__dict__.items()]; print(l)
> [('a', 3), ('f', ), ('__module__', 
> '__main__'), ('__dict__', ), 
> ('__doc__', None), ('__weakref__',  objects>)]
> 
>  >>> import inspect
>  >>> [(key, value) for key, value in l if inspect.isfunction(i[1])]
> [('f', ), ('g',  
> at 0x01DEDD693620>)]
> 
> HTH




After more researching, I found the below code that works for me.

https://stackoverflow.com/questions/1911281/how-do-i-get-list-of-methods-in-a-python-class


from types import FunctionType

class Foo:
def bar(self): pass
def baz(self): pass

def methods(cls):
return [x for x, y in cls.__dict__.items() if type(y) == FunctionType]


Using the above code, I now get the following.
['__init__', 'apples', 'peaches', 'pumpkin']

Bruce

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Marko Rauhamaa]

Paul Moore :
> On 1 June 2018 at 13:15, Barry Scott  wrote:
>> Once you know that all of the string you provided is given to the
>> operating system it can then do whatever checks it sees fit to and
>> return a suitable result.
>
> As the programmer, I don't care. The Python interpreter should take
> care of that for me, and if I say "does file 'a\0b' exist?" I want an
> answer. And I don't see how anything other than "no it doesn't" is
> correct. Python allows strings with embedded \0 characters, so it's
> possible to express that question in Python - os.path.exists('a\0b').
> What can be expressed in terms of the low-level (C-based) operating
> system API shouldn't be relevant.

Interestingly, you get a False even for existing files if you don't have
permissions to access the file. Arguably, that answer is misleading, and
an exception would be justified. But since os.path.exists() returns a
False even in that situation, it definitely should return False for a
string containing a NUL character.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Richard Damon]

On 5/31/18 1:43 PM, Grant Edwards wrote:
> On 2018-05-31, Paul Moore  wrote:
>> On 31 May 2018 at 15:01, Chris Angelico  wrote:
>>
>> Honestly, I think the OP's point is correct. os.path.exists should
>> simply return False if the filename has an embedded \0 - at least on
>> Unix.
> Except on the platform in quetion filenames _don't_ contain an
> embedded \0.  What was passed was _not_ a path/filename.
>
> You might as well have passed a floating point number or a dict.
>
>
I think this is a key point. os.path.exists needs to pass a null
terminated string to the system to ask it about the file. The question
comes what should we do if we pass it a value that can't (or we won't)
represent as such a string.

The confusion is that in python, a string with an embedded null is
something pretty much like a string without an embedded null, so the
programmer might not think of it as being the wrong type. Thus we have
several options.

1) we can treat os.path.exists('foo\0bar') the same as
os.path.exists(1.5) and raise the exception.

2) we can treat os.path.exists('foo\0bar') as specifying a file that can
never exists and bypass the system call are return false.

3) we can process os.path.exists('foo\0bar') by just passing the string
to the system call, making it the same as os.path.exists('foo')

The last is probably the one that we can say is likely wrong, but
arguments could be made for either of the first two.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Chris Angelico]

On Fri, Jun 1, 2018 at 11:41 PM, Richard Damon  wrote:
> The confusion is that in python, a string with an embedded null is
> something pretty much like a string without an embedded null, so the
> programmer might not think of it as being the wrong type. Thus we have
> several options.
>
> 1) we can treat os.path.exists('foo\0bar') the same as
> os.path.exists(1.5) and raise the exception.

1.5 raises TypeError, which is correct. But the type of "foo\0bar" is
str, which is a perfectly valid type. ValueError is more correct here.
And that's what currently happens.

Possibly more confusing, though, is this:

>>> os.path.exists(1)
True
>>> os.path.exists(2)
True
>>> os.path.exists(3)
False

I think it's testing that the file descriptors exist, because
os.path.exists is defined in terms of os.stat, which can stat a path
or an FD. So os.path.exists(fd) is True if that fd is open, and False
if it isn't. But os.path.exists is not documented as accepting FDs.
Accident of implementation or undocumented feature? Or maybe
accidental feature?

> 2) we can treat os.path.exists('foo\0bar') as specifying a file that can
> never exists and bypass the system call are return false.

That's what's being proposed.

> 3) we can process os.path.exists('foo\0bar') by just passing the string
> to the system call, making it the same as os.path.exists('foo')
>
> The last is probably the one that we can say is likely wrong, but
> arguments could be made for either of the first two.

More than "likely wrong"; it's definitely wrong, and deceptively so. I
don't think anyone would support this case.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Marko Rauhamaa]

Chris Angelico :

> Possibly more confusing, though, is this:
>
 os.path.exists(1)
> True
 os.path.exists(2)
> True
 os.path.exists(3)
> False
>
> I think it's testing that the file descriptors exist, because
> os.path.exists is defined in terms of os.stat, which can stat a path
> or an FD. So os.path.exists(fd) is True if that fd is open, and False
> if it isn't. But os.path.exists is not documented as accepting FDs.
> Accident of implementation or undocumented feature? Or maybe
> accidental feature?

What's more:

   >>> os.path.exists(-100)
   False
   >>> os.path.exists(-1)
   Traceback (most recent call last):
 File "", line 1, in 
 File "/usr/lib64/python3.6/genericpath.py", line 19, in exists
   os.stat(path)
   OverflowError: fd is less than minimum

One could argue -100 is less than minimum...

The common denominator is that "\0" and -1 are caught by
Python's standard library while "" and -100 are caught by the OS.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Indented multi-line strings

[Dan Strohl via Python-list]

> 
> I would prefer to remove the padding, like this:
> 
> Test = """
> |Hello, this is a
> | Multiline indented
> |String
> """.outdent(padding='|')
> 
> Or write it like this?
> 
> Test = """|Hello, this is a
>   | Multiline indented
>   |String
>   """.outdent(padding='|')
> 
> Hmm, the sign of Zorro! :-)
> 
> I'm starting to like outdent(), but that may be my TIMTOWTDIism speaking.
> 

I can see both outdent(padding="|") and outdent(size=4) being useful in various 
cases.  If this ends up being the recommendation, I would also suggest adding 
str.indent(), it evens out the methods and most of the time I use the textwrap 
functions it is either indent or outdent, so this would clean up those pieces.

So... how does one go about suggesting changes to the built in types?  I could 
take a whack at the code for it, but my C skills are no where near what should 
probably be needed for something this close to the core of the language.  I'm 
not sure if adding a couple of methods is a PEP type of thing.

Dan Strohl


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Grant Edwards]

On 2018-06-01, Paul Moore  wrote:

> Python allows strings with embedded \0 characters, so it's possible
> to express that question in Python - os.path.exists('a\0b').  What
> can be expressed in terms of the low-level (C-based) operating
> system API shouldn't be relevant.

Python allows floating point numbers, so it is possible to express
this question in python: os.path.exists(3.14159).  Is the fact that
the underlying OS/filesystem can't identify files via a floating point
number relevent?  Should it return False or raise ValueError?

-- 
Grant Edwards   grant.b.edwardsYow! How do I get HOME?
  at   
  gmail.com

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Indented multi-line strings

[Paul Moore]

On 1 June 2018 at 15:36, Dan Strohl via Python-list
 wrote:
> So... how does one go about suggesting changes to the built in types?  I 
> could take a whack at the code for it, but my C skills are no where near what 
> should probably be needed for something this close to the core of the 
> language.  I'm not sure if adding a couple of methods is a PEP type of thing.

It would probably have to go via python-ideas, but if it gets the OK
there I doubt it would need a PEP.

There are a few key questions I'd expect to see come up.

Why does this need to be a string method? Why can't it be a standalone
function? Maybe you should publish an implementation on PyPI, collect
some data on how popular it is, and then if it's widely used, propose
it for inclusion in the stdlib at that point? By making it a string
method, you're also restricting its use to users of recent versions of
Python, whereas a PyPI implementation would work for everyone.

None of these are showstoppers - many proposals have got past them -
but it's worth having at least thought through your answers to them,
so you can present the idea in the best light.

Paul
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Paul Moore]

On 1 June 2018 at 15:38, Grant Edwards  wrote:
> On 2018-06-01, Paul Moore  wrote:
>
>> Python allows strings with embedded \0 characters, so it's possible
>> to express that question in Python - os.path.exists('a\0b').  What
>> can be expressed in terms of the low-level (C-based) operating
>> system API shouldn't be relevant.
>
> Python allows floating point numbers, so it is possible to express
> this question in python: os.path.exists(3.14159).  Is the fact that
> the underlying OS/filesystem can't identify files via a floating point
> number relevent?  Should it return False or raise ValueError?

I'm not sure if you're asking a serious question here, or trying to
make some sort of point, but os.path.exists is documented as taking a
string, so passing a float should be a TypeError. And it is.

But as I already said, this is a huge amount of effort spent on a
pretty trivial corner case, so I'll duck out of this thread now.
Paul
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ... (ellipsis)

[Mike McClain]

On Thu, May 31, 2018 at 11:00:13PM -0400, Terry Reedy wrote:
> On 5/31/2018 10:26 PM, Mike McClain wrote:
> > I'm having understanding the use if the ellipsis.
> >I keep reading that it is used in slices
>
> By numpy for numpy multidimensional arrays, which have their own
> __getitem__, which recognizes and gives meaning to ...
>
Thank you Mr. Reedy
I've not yet looked at numpy but no doubt will get there.
Mike
--
The depression won't end till we grow a generation that knows how to
live on what they got.
- Will Rogers
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Indented multi-line strings

[Dan Strohl via Python-list]

> 
> It would probably have to go via python-ideas, but if it gets the OK there I
> doubt it would need a PEP.
> 

Cool, thanks!

> There are a few key questions I'd expect to see come up.
> 
> Why does this need to be a string method? Why can't it be a standalone
> function? Maybe you should publish an implementation on PyPI, collect some
> data on how popular it is, and then if it's widely used, propose it for 
> inclusion
> in the stdlib at that point? By making it a string method, you're also 
> restricting
> its use to users of recent versions of Python, whereas a PyPI implementation
> would work for everyone.

Good point, so, basically, there already is a function for this built in 
textwrap.dedent() and textwrap.indent(), I would think (hope) that that would 
answer that question.



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Indented multi-line strings

[Chris Angelico]

On Sat, Jun 2, 2018 at 12:57 AM, Paul Moore  wrote:
> On 1 June 2018 at 15:36, Dan Strohl via Python-list
>  wrote:
>> So... how does one go about suggesting changes to the built in types?  I 
>> could take a whack at the code for it, but my C skills are no where near 
>> what should probably be needed for something this close to the core of the 
>> language.  I'm not sure if adding a couple of methods is a PEP type of thing.
>
> It would probably have to go via python-ideas, but if it gets the OK
> there I doubt it would need a PEP.

Yeah, take it to -ideas but don't worry about a PEP.

> There are a few key questions I'd expect to see come up.
>
> Why does this need to be a string method? Why can't it be a standalone
> function? Maybe you should publish an implementation on PyPI, collect
> some data on how popular it is, and then if it's widely used, propose
> it for inclusion in the stdlib at that point? By making it a string
> method, you're also restricting its use to users of recent versions of
> Python, whereas a PyPI implementation would work for everyone.

The biggest reason to make it a string method is to give the
possibility of optimization. Python cannot optimize int(1.2) down to
the constant 1 because you might have shadowed int; but a method on a
string literal cannot be shadowed, and could potentially be
constant-folded. Include that in the initial post to preempt this
recommendation.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: version

[Mike McClain]

On Thu, May 31, 2018 at 07:44:35PM -0700, Mike McClain wrote:

> Is there a way in a script to know which version of python is being
> run so I can write:
> If (version == 2.7):
> do it this way
> elsif (version == 3.2):
> do it another way
>

Thanks for the responses,
Mike
--
The depression won't end till we grow a generation that knows how to
live on what they got.
- Will Rogers
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Richard Damon]

On 6/1/18 9:58 AM, Chris Angelico wrote:
> On Fri, Jun 1, 2018 at 11:41 PM, Richard Damon  
> wrote:
>> The confusion is that in python, a string with an embedded null is
>> something pretty much like a string without an embedded null, so the
>> programmer might not think of it as being the wrong type. Thus we have
>> several options.
>>
>> 1) we can treat os.path.exists('foo\0bar') the same as
>> os.path.exists(1.5) and raise the exception.
> 1.5 raises TypeError, which is correct. But the type of "foo\0bar" is
> str, which is a perfectly valid type. ValueError is more correct here.
> And that's what currently happens.
>
> Possibly more confusing, though, is this:
>
 os.path.exists(1)
> True
 os.path.exists(2)
> True
 os.path.exists(3)
> False
>
> I think it's testing that the file descriptors exist, because
> os.path.exists is defined in terms of os.stat, which can stat a path
> or an FD. So os.path.exists(fd) is True if that fd is open, and False
> if it isn't. But os.path.exists is not documented as accepting FDs.
> Accident of implementation or undocumented feature? Or maybe
> accidental feature?
>
>> 2) we can treat os.path.exists('foo\0bar') as specifying a file that can
>> never exists and bypass the system call are return false.
> That's what's being proposed.
>
>> 3) we can process os.path.exists('foo\0bar') by just passing the string
>> to the system call, making it the same as os.path.exists('foo')
>>
>> The last is probably the one that we can say is likely wrong, but
>> arguments could be made for either of the first two.
> More than "likely wrong"; it's definitely wrong, and deceptively so. I
> don't think anyone would support this case.
>
> ChrisA

I would say that one way to look at it is that os.path.exists
fundamentally (at the OS level) expects a parameter of the 'type' of
either a nul terminated string or a file descriptor (aka fixed width
integer). One issue we have is that these 'types' don't directly map to
Python types.

We can basically make a call to os.path.exists with 4 different types of
parameter:

1) The parameter has a totally wrong type of type that just doesn't map
to one of the expected type. This gives a TypeError exeception.

2) The parameter has a Python type that maps to right OS 'type' but has
a value that prevents us from properly converting it to a corresponding
value of that type. This could be a integral value out of range for the
fixed width type used, or a string which contains an embedded nul.
Currently these generate an OverflowError for out of range integer and a
ValueError for a bad string

3) The parameter can be mapped to the proper type but the value is
somehow illegal (the number fits the type, but isn't legal for a file
descriptor, or a string has a value that can't represent a real file).
In this case, os.path.exists doesn't try to validate the parameter but
just passes it along and returns a value based on the answer it gets.

4) The parameter represents a legal value of a right type, so as above
we pass the value and get back the answer.

The fundamental question is about case 2. Should os.path.exist, having
been give a value of the right 'Python Type' but not matching the type
of the operating system parameter identify this as an error (as it
currently does), or should it be changed to decide that if it could
somehow get that parameter to the os, then it would say that the file
doesn't exist, and so return false. I would say that if you accept that,
should we also say that if we pass a totally wrong type, why shouldn't
we again return false instead of a TypeError, after all, if we pass it a
dictionary, they certainly is no file like that in existence,

The real question comes which method is more useful, which is most apt
to be the one we want, and which one is the better building block for a
program.

One thing to note as an advantage for the current method, it is trivial
with the current decision to write a mypathexists that would accept
strings with nuls embedded and return false, just call os.path.exists
inside a try, and catch the ValueError and return false. You could also
extend it to catch OverflowError and/or TypeError. On the other hand, if
os.path.exists swallows these errors and just returns false, then it is
a lot more work to make a wrapper that throws the errors, you basically
would need to precheck for bad values and throw, and then if you move to
a system that happened to allow nuls in the file name (and the python
code knew that), your wrapper code now is wrong as you had to build in
implementation knowledge into the user code.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Indented multi-line strings

[Paul Moore]

On 1 June 2018 at 16:36, Chris Angelico  wrote:
> On Sat, Jun 2, 2018 at 12:57 AM, Paul Moore  wrote:

>> Why does this need to be a string method? Why can't it be a standalone
>> function? Maybe you should publish an implementation on PyPI, collect
>> some data on how popular it is, and then if it's widely used, propose
>> it for inclusion in the stdlib at that point? By making it a string
>> method, you're also restricting its use to users of recent versions of
>> Python, whereas a PyPI implementation would work for everyone.
>
> The biggest reason to make it a string method is to give the
> possibility of optimization. Python cannot optimize int(1.2) down to
> the constant 1 because you might have shadowed int; but a method on a
> string literal cannot be shadowed, and could potentially be
> constant-folded. Include that in the initial post to preempt this
> recommendation.

So the optimisation should probably be an explicit part of the
proposal. Without the optimisation, factors like "won't be usable in
code that wants to support older Python", "why not just make it a
standalone function", etc. will probably result in the proposal not
getting accepted.

On 1 June 2018 at 16:20, Dan Strohl  wrote:
>
> Good point, so, basically, there already is a function for this built in 
> textwrap.dedent() and textwrap.indent(), I would think (hope) that that would 
> answer that question.

OK, so unless the argument is "provide a string method, that's
guaranteed to be constant folded"[1] I suspect that it's pretty
unlikely that a proposal to add a string method that simply replicated
the textwrap functions would get very far.

But regardless, there's no point in me trying to second guess what
might come up on python-ideas, you should just post there and see what
reception you get.

Paul

[1] There's two possibilities here, of course. First, provide an
implementation for CPython that includes the constant folding, or
second, make constant folding a language guarantee that other
implementations also have to implement. I doubt the second option is
going to be practical, though.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Indented multi-line strings

[Chris Angelico]

On Sat, Jun 2, 2018 at 2:10 AM, Paul Moore  wrote:
> On 1 June 2018 at 16:36, Chris Angelico  wrote:
>> On Sat, Jun 2, 2018 at 12:57 AM, Paul Moore  wrote:
>
>>> Why does this need to be a string method? Why can't it be a standalone
>>> function? Maybe you should publish an implementation on PyPI, collect
>>> some data on how popular it is, and then if it's widely used, propose
>>> it for inclusion in the stdlib at that point? By making it a string
>>> method, you're also restricting its use to users of recent versions of
>>> Python, whereas a PyPI implementation would work for everyone.
>>
>> The biggest reason to make it a string method is to give the
>> possibility of optimization. Python cannot optimize int(1.2) down to
>> the constant 1 because you might have shadowed int; but a method on a
>> string literal cannot be shadowed, and could potentially be
>> constant-folded. Include that in the initial post to preempt this
>> recommendation.
>
> So the optimisation should probably be an explicit part of the
> proposal. Without the optimisation, factors like "won't be usable in
> code that wants to support older Python", "why not just make it a
> standalone function", etc. will probably result in the proposal not
> getting accepted.

There's already an existing function, albeit not the most
discoverable. But people avoid it because it's a run-time cost, and
deleting the leading spaces in the source code eliminates that
run-time cost.

> OK, so unless the argument is "provide a string method, that's
> guaranteed to be constant folded"[1] I suspect that it's pretty
> unlikely that a proposal to add a string method that simply replicated
> the textwrap functions would get very far.
>
> [1] There's two possibilities here, of course. First, provide an
> implementation for CPython that includes the constant folding, or
> second, make constant folding a language guarantee that other
> implementations also have to implement. I doubt the second option is
> going to be practical, though.

I wouldn't make it an actual guarantee. As long as CPython has the
optimization, people will start using it; if Jython or MicroPython or
Brython does a run-time method call, so be it. It's still likely to be
faster than importing a module, but most importantly, people's code
will look better this way than manually unindented.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Attachments? Re: Indented multi-line strings (was: "Data blocks" syntax specification draft)

[Peter J. Holzer]

On 2018-05-31 14:42:39 -0700, Paul wrote:
> I have heard that attachments to messages are not allowed on this list,
> which makes sense. However I notice that messages from Peter do have an
> attachment, i.e., a signature.asc file.

No this is isn't an attachment. It's a signature. Your MUA probably
displays everything it can't handle as an "attachment".
> 
> I'm just curious; why and how do those particular attachments get through?
> And should they get through, I guess? E.G., what if I attach a malicious
> file labeled as .asc?

The mailinglist software can probably distinguish between
multipart/signed and multipart/mixed.

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Indented multi-line strings

[Peter Pearson]

On Fri, 1 Jun 2018 15:57:58 +0100, Paul Moore  wrote:
> On 1 June 2018 at 15:36, Dan Strohl via Python-list
> wrote:
>> So... how does one go about suggesting changes to the built in types?
[snip]
>
> Why does this need to be a string method? Why can't it be a standalone
> function?

Yes, please, let's content ourselves with a standalone function.

Adding features to a language imposes costs in several ways, including
hindering newcomers and making code version-dependent.  To full-time
Python developers, these costs appear small, because they are amortized
over a lot of Python activity; but they are encumbrances to the spread
and casual use of the language.  It seems as if the destiny of every
language is to be adorned by its enthusiasts with so many arcane and
specialized optimizations -- every one an obvious improvement -- that
the world's interest drifts to something newer and cleaner.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Indented multi-line strings

[Chris Angelico]

On Sat, Jun 2, 2018 at 7:00 AM, Peter Pearson  wrote:
> On Fri, 1 Jun 2018 15:57:58 +0100, Paul Moore  wrote:
>> On 1 June 2018 at 15:36, Dan Strohl via Python-list
>> wrote:
>>> So... how does one go about suggesting changes to the built in types?
> [snip]
>>
>> Why does this need to be a string method? Why can't it be a standalone
>> function?
>
> Yes, please, let's content ourselves with a standalone function.
>
> Adding features to a language imposes costs in several ways, including
> hindering newcomers and making code version-dependent.  To full-time
> Python developers, these costs appear small, because they are amortized
> over a lot of Python activity; but they are encumbrances to the spread
> and casual use of the language.  It seems as if the destiny of every
> language is to be adorned by its enthusiasts with so many arcane and
> specialized optimizations -- every one an obvious improvement -- that
> the world's interest drifts to something newer and cleaner.

How will a method be worse than a standalone function? Please explain
this. A method is a lot easier to discover than a stdlib module
function, which is in turn IMMENSELY more discoverable than anything
on pypi.

If you dislike adding features to a language on the basis that it
makes the language harder to learn, remember that you instead force
one of three even worse options:

1) Messy code because people unindent inside their source code,
creating wonky indentation (which Python usually avoids)

2) Forcing readers to look up the third-party module you're using
before they can understand your code

3) Forcing readers to look up your ad-hoc function before
understanding your code.

All of these make it harder to understand your code, specifically
BECAUSE the language doesn't have the requisite feature. Well-written
language features are good, not bad, for readability.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Attachments? Re: Indented multi-line strings (was: "Data blocks" syntax specification draft)

[Chris Angelico]

On Sat, Jun 2, 2018 at 7:03 AM, Peter J. Holzer  wrote:
> On 2018-05-31 14:42:39 -0700, Paul wrote:
>> I have heard that attachments to messages are not allowed on this list,
>> which makes sense. However I notice that messages from Peter do have an
>> attachment, i.e., a signature.asc file.
>
> No this is isn't an attachment. It's a signature. Your MUA probably
> displays everything it can't handle as an "attachment".

Which, I might point out, is a decent way to handle unknown parts. The
user is shown that there's an extra part, and can view it as an
external file.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Steven D'Aprano]

On Fri, 01 Jun 2018 09:41:04 -0400, Richard Damon wrote:

> I think this is a key point. os.path.exists needs to pass a null
> terminated string to the system to ask it about the file.

That's an implementation detail utterly irrelevant to Python programmers. 
If the OS expects a pointer to a block of UTF-16 bytes, would you say "oh 
well, since Python doesn't have low level pointers, we simply can't 
provide this functionality?"

No of course we would not. os.path.exists should take a regular Python 
string and adapt it to whatever the implementation requires.


> The question
> comes what should we do if we pass it a value that can't (or we won't)
> represent as such a string.

You get a TypeError:


py> os.path.exists([])
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python3.5/genericpath.py", line 19, in exists
os.stat(path)
TypeError: argument should be string, bytes or integer, not list



> The confusion is that in python, a string with an embedded null is
> something pretty much like a string without an embedded null, so the
> programmer might not think of it as being the wrong type.

But it *isn't* the wrong type. It is the same type:


py> type("abc") is type("a\0c")
True


So TypeError is out, since the type is right, only the value is wrong. 
But ValueError is wrong too:

What does os.path.exists return when given "the wrong value" (i.e. a 
string that doesn't match an existing path)? It returns False, not raise 
an exception.


> Thus we have several options.

Only one of which is consistent with the rest of os.path.exist()'s 
behaviour, only one of which is sensible. Return False, like every other 
pathname that doesn't exist.




-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Chris Angelico]

On Sat, Jun 2, 2018 at 8:10 AM, Steven D'Aprano
 wrote:
> But it *isn't* the wrong type. It is the same type:
>
>
> py> type("abc") is type("a\0c")
> True
>
>
> So TypeError is out, since the type is right, only the value is wrong.

I agree with you so far. This parallels the fact that math.asin(5)
doesn't raise TypeError, since an integer is perfectly valid input to
the arcsine function.

> But ValueError is wrong too:
>
> What does os.path.exists return when given "the wrong value" (i.e. a
> string that doesn't match an existing path)? It returns False, not raise
> an exception.

Hang on, how is "a string that doesn't match an existing path" the
wrong value? The point is to ask if a path exists. A path that doesn't
exist is still valid. There are four possibilities:

1) You're asking a question that doesn't even make sense, like
os.path.exists(Ellipsis)

2) You're asking about a path that exists and you can prove that it
does. Return True.

3) You're asking about a path that doesn't exist and you can prove
that it doesn't. Return False.

4) You're asking about a path that you can't be sure about - maybe
there's a permissions error.

With perms errors, it's less clear what's the right thing to do. I
think letting an OSError bubble up would be appropriate here, but
others may disagree. Similarly if you try to access a network mount
that is currently disconnected, a device that has crashed, etc.
Returning False (as in the current implementation) is plausible; I
don't think it's the ideal, but since it would break backward
compatibility to change it now, I'm not advocating making that change.

The real question is whether os.path.exists("a\0c") is more akin to
os.path.exists(Ellipsis) or to os.path.exists("/mnt/broken/spam"). It
is never going to be valid to ask whether Ellipsis exists, and it is
never going to be valid to ask whether a\0c exists. The broken mount
might potentially become valid, and the same path name could change in
meaning if you are chrooted, so it is a sane question to ask.

My ideal preference would be for True to mean "we know for certain
that this exists" and False "we know for certain that this doesn't
exist" (which might be because one of its parent components doesn't
exist - if /spam doesn't exist, then /spam/ham doesn't either, and
that's just straight False). Everything else should raise an
exception. That said, though, I'm not hugely fussed and don't have
actual use-cases here, beyond the back-of-the-mind feeling that saying
"no that doesn't exist" is deceptively confident about something that
isn't actually certain.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Gregory Ewing]

Marko Rauhamaa wrote:

Interestingly, you get a False even for existing files if you don't have
permissions to access the file.


Obviously in that case, instead of True or False it should
return FileNotFound. :-)

https://thedailywtf.com/articles/What_Is_Truth_0x3f_

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with OrderedDict - progress report

[Gregory Ewing]

Chris Angelico wrote:

if you 'break' immediately after a mutation, that isn't
continuing to iterate. Even though you're inside the loop, there's no
further action taken to process the loop, and no problem.


Yes, but you're also not calling next() again, so no
exception would be triggered.

My point is that I don't see a justification for treating
the last next() call (the one that notices you've reached
the end and raises StopIteration) any differently from
the others. It seems like either a bug or a misfeature
to me.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Steven D'Aprano]

On Fri, 01 Jun 2018 14:38:56 +, Grant Edwards wrote:

> On 2018-06-01, Paul Moore  wrote:
> 
>> Python allows strings with embedded \0 characters, so it's possible to
>> express that question in Python - os.path.exists('a\0b').  What can be
>> expressed in terms of the low-level (C-based) operating system API
>> shouldn't be relevant.
> 
> Python allows floating point numbers, so it is possible to express this
> question in python: os.path.exists(3.14159).  Is the fact that the
> underlying OS/filesystem can't identify files via a floating point
> number relevent?  Should it return False or raise ValueError?

Certainly not a ValueError, that would be silly. The fact that it is an 
illegal value is subordinate to the fact that it is the wrong type. 

It should either return False, or raise TypeError. Of the two, since 
3.14159 cannot represent a file on any known OS, TypeError would be more 
appropriate.

But since "\0" is the correct type (a string), and the fact that it 
happens to be illegal on POSIX is a platform-dependent detail of no more 
importance than the fact that "?" is illegal on Windows, it should be 
treated as any other platform-dependent illegal file and return False.


-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Steven D'Aprano]

On Thu, 31 May 2018 17:43:28 +, Grant Edwards wrote:

> Except on the platform in quetion filenames _don't_ contain an embedded
> \0.  What was passed was _not_ a path/filename.

"/wibble/rubbish/nobodyexpectsthespanishinquistion" is not a pathname on 
my system either, and os.path.exists() returns False for that. As it is 
supposed to.

I'd be willing to bet that:

import secrets  # Python 3.6+
s = "/" + secrets.token_hex(1024) + "/spam"

is not a pathname on any computer in the world. (If it is even legal.) 
And yet os.path.exists(s) returns False.

The maximum number of file components under POSIX is (I believe) 256. And 
yet:

py> os.path.exists("/a"*100)
False

"/a" by one million cannot possibly be a path under POSIX.


> the thread will continue for months and generate hundreds of followup.

Only because some people insist on exercising their right to be wrong.


-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Gregory Ewing]

Grant Edwards wrote:

Python allows floating point numbers, so it is possible to express
this question in python: os.path.exists(3.14159).  Is the fact that
the underlying OS/filesystem can't identify files via a floating point
number relevent?  Should it return False or raise ValueError?


I don't know about that, but it's clear that
os.path.exists(1j) shoud raise OnlyInYourDreamsError.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[eryk sun]

On Fri, Jun 1, 2018 at 3:58 PM, Richard Damon  wrote:
>
> The fundamental question is about case 2. Should os.path.exist, having
> been give a value of the right 'Python Type' but not matching the type
> of the operating system parameter identify this as an error (as it
> currently does), or should it be changed to decide that if it could
> somehow get that parameter to the os, then it would say that the file
> doesn't exist, and so return false.

AFAIK, this behavior hasn't been documented. So it can either be
documented, and thus never allow NUL in paths, or else every call that
currently raises ValueError for this case should raise a pretend
FileNotFoundError. No change to exists(), isdir(), and isfile() would
be required.

For Windows, there's another case that's in more of a grey area.
Python 3.6+ uses UTF-8 as the file-system encoding in Windows.
Internally it transcodes between UTF-8 and the native UTF-16 encoding.
The "surrogatepass" error handler is used in order to faithfully
handle invalid surrogates, which the system allows. This leaves no
simple way to smuggle invalid UTF-8 sequences into the filename and
rountrip back to bytes, so UnicodeDecodeError (a subclass of
ValueError) is raised. The same invalid UTF-8 would pass silently in
POSIX, which uses bytes paths and the "surrogateescape" handler.

Trivia:
The native NT API of Windows can use device names that contain NUL
characters because it uses counted strings in the OBJECT_ATTRIBUTES
record that's used to access named objects (e.g. Device, Section, Job,
Event, Semaphore, etc). I've tested that this works. A file system
could also allow NUL in names, but Microsoft's drivers reserve NUL as
an invalid character, as would any driver that uses the file-system
runtime library. That said, native NT applications have a limited
scope, so it's almost pointless to speculate.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Steven D'Aprano]

On Fri, 01 Jun 2018 11:58:42 -0400, Richard Damon wrote:

> I would say that one way to look at it is that os.path.exists
> fundamentally (at the OS level) expects a parameter of the 'type' of
> either a nul terminated string or a file descriptor (aka fixed width
> integer). One issue we have is that these 'types' don't directly map to
> Python types.


What a strange and unhelpful way to look at it. When calling 
os.path.exists() from Python, are you interacting directly with the OS in 
low-level C or assembly, or using a high-level language like Python?

That's not a rhetorical question. I would like to know what you think we 
are doing when we type into the Python interpreter

import os
os.path.exists("pathname")

Well actually no I lie. Of course its a rhetorical question: I'm sure you 
know full well that you're using Python, a high-level language.


> We can basically make a call to os.path.exists with 4 different types of
> parameter:
> 
> 1) The parameter has a totally wrong type of type that just doesn't map
> to one of the expected type. This gives a TypeError exeception.
> 
> 2) The parameter has a Python type that maps to right OS 'type' but has
> a value that prevents us from properly converting it to a corresponding
> value of that type. This could be a integral value out of range for the
> fixed width type used, 

That is a reasonable case for either ValueError or OverflowError, 
OverflowError being a more specific (and therefore useful) exception to 
raise. Being a low-level detail, file descriptors are inherently limited 
to a fixed width and it truly is an error to supply something outside of 
that width.

> or a string which contains an embedded nul.

The fact that this is illegal under POSIX is an irrelevant platform-
dependent detail. Irrelevant in the sense that it shouldn't change the 
API of the function. "<" doesn't raise ValueError on Windows, and "" 
doesn't raise ValueError on any platform. Why should POSIX nulls be 
treated as special?


> Currently these generate an OverflowError for out of range integer and a
> ValueError for a bad string

Incorrect: as Paul and MRAB (and myself) have pointed out, bad strings 
return False, with the surprising exception of null-embedded strings 
under POSIX.

Illegal strings like "<" on Windows return False. Excessive long strings, 
or strings with too many path components, return False. The empty string 
returns False.

There is no good reason to raise ValueError on strings with null in them.


> 3) The parameter can be mapped to the proper type but the value is
> somehow illegal (the number fits the type, but isn't legal for a file
> descriptor, or a string has a value that can't represent a real file).
> In this case, os.path.exists doesn't try to validate the parameter but
> just passes it along and returns a value based on the answer it gets.

Right. And if the answer it gets is "illegal value", it returns False.



> 4) The parameter represents a legal value of a right type, so as above
> we pass the value and get back the answer.
> 
> The fundamental question is about case 2. Should os.path.exist, having
> been give a value of the right 'Python Type' but not matching the type
> of the operating system parameter identify this as an error (as it
> currently does), 

That's wrong, that is not what it does, except in the surprising case of 
nulls under POSIX.


> or should it be changed to decide that if it could
> somehow get that parameter to the os, then it would say that the file
> doesn't exist, and so return false.

Of course.


> I would say that if you accept that,
> should we also say that if we pass a totally wrong type, why shouldn't
> we again return false instead of a TypeError, after all, if we pass it a
> dictionary, they certainly is no file like that in existence,

Because os.path.exists is documented as accepting strings, bytes and 
ints, and everything else is a TypeError.

If you want to make a case for relaxing the type restrictions on 
os.path.exists, go right ahead, but I won't be supporting you.



-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Steven D'Aprano]

On Sat, 02 Jun 2018 08:50:38 +1000, Chris Angelico wrote:

> My ideal preference would be for True to mean "we know for certain that
> this exists" and False "we know for certain that this doesn't exist"

We cannot make that promise, because we might not have permission to view 
the file. Since we don't have a three-state True/False/Maybe flag, 
os.path.exists() just returns False.

> (which might be because one of its parent components doesn't exist - if
> /spam doesn't exist, then /spam/ham doesn't either, and that's just
> straight False).

And if the path is an illegal value, it also straight doesn't exist. Like 
the empty string, like "<" on Windows, like strings with too many path 
components.


-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Chris Angelico]

On Sat, Jun 2, 2018 at 9:37 AM, Steven D'Aprano
 wrote:
> On Thu, 31 May 2018 17:43:28 +, Grant Edwards wrote:
>
>> Except on the platform in quetion filenames _don't_ contain an embedded
>> \0.  What was passed was _not_ a path/filename.
>
> "/wibble/rubbish/nobodyexpectsthespanishinquistion" is not a pathname on
> my system either, and os.path.exists() returns False for that. As it is
> supposed to.
>
> I'd be willing to bet that:
>
> import secrets  # Python 3.6+
> s = "/" + secrets.token_hex(1024) + "/spam"
>
> is not a pathname on any computer in the world. (If it is even legal.)
> And yet os.path.exists(s) returns False.

With both of these, the path cannot exist because its first component
does not exist. Absent a /wibble on your system, the entire long path
is unable to exist. That is a natural consequence of the hierarchical
structure of file systems. I'm fairly sure 2KB of path is valid on all
major OSes today, which means that it's exactly the same as /wibble -
the first component doesn't exist, ergo the path doesn't exist.

> The maximum number of file components under POSIX is (I believe) 256. And
> yet:
>
> py> os.path.exists("/a"*100)
> False
>
> "/a" by one million cannot possibly be a path under POSIX.

I can't actually find that listed anywhere. Citation needed. But
assuming you're right, POSIX is still a set of minimum requirements -
not maximums, to my knowledge. If some operating system permits longer
paths with more components, it won't be non-compliant on that basis.
So it's still plausible to ask "does this path exist", and it's
perfectly correct to look at the first "/a/" and check if there's
anything named "a" in your root directory, and return False upon
finding none. The question is sane, unlike os.path.exists([]).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Chris Angelico]

On Sat, Jun 2, 2018 at 9:54 AM, Steven D'Aprano
 wrote:
> On Sat, 02 Jun 2018 08:50:38 +1000, Chris Angelico wrote:
>
>> My ideal preference would be for True to mean "we know for certain that
>> this exists" and False "we know for certain that this doesn't exist"
>
> We cannot make that promise, because we might not have permission to view
> the file. Since we don't have a three-state True/False/Maybe flag,
> os.path.exists() just returns False.

Being unable to view the file is insignificant, but presumably you
mean we might not have permission to view the containing directory.
And we do get that information:

>>> os.stat("/root/.bashrc")
Traceback (most recent call last):
  File "", line 1, in 
PermissionError: [Errno 13] Permission denied: '/root/.bashrc'
>>> os.stat("/var/.bashrc")
Traceback (most recent call last):
  File "", line 1, in 
FileNotFoundError: [Errno 2] No such file or directory: '/var/.bashrc'

The permissions error is reported differently by stat, but then
exists() just says "oh that's an OS error so we're going to say it
doesn't exist". If you truly want the reliable form of os.path.exists,
it would be this:

try:
os.stat(path)
return True
except FileNotFoundError:
return False

Anything that ISN'T "this file exists" or "this file doesn't exist"
will be signalled with an exception.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Steven D'Aprano]

On Sat, 02 Jun 2018 09:56:58 +1000, Chris Angelico wrote:

> On Sat, Jun 2, 2018 at 9:37 AM, Steven D'Aprano
>  wrote:
>> On Thu, 31 May 2018 17:43:28 +, Grant Edwards wrote:
>>
>>> Except on the platform in quetion filenames _don't_ contain an
>>> embedded \0.  What was passed was _not_ a path/filename.
>>
>> "/wibble/rubbish/nobodyexpectsthespanishinquistion" is not a pathname
>> on my system either, and os.path.exists() returns False for that. As it
>> is supposed to.
>>
>> I'd be willing to bet that:
>>
>> import secrets  # Python 3.6+
>> s = "/" + secrets.token_hex(1024) + "/spam"
>>
>> is not a pathname on any computer in the world. (If it is even legal.)
>> And yet os.path.exists(s) returns False.
> 
> With both of these, the path cannot exist because its first component
> does not exist.

Since /wibble doesn't exist, neither does /wibble/a\0b


py> os.path.exists("/wibble")
False
py> os.path.exists("/wibble/a\0b")
Traceback (most recent call last):
  File "", line 1, in 
  File "/storage/torrents/torrents/python/Python-3.6.4/Lib/
genericpath.py", line 19, in exists
os.stat(path)
ValueError: embedded null byte


Oops.


> Absent a /wibble on your system, the entire long path is
> unable to exist. That is a natural consequence of the hierarchical
> structure of file systems. I'm fairly sure 2KB of path is valid on all
> major OSes today, 

But probably not 2K in a single path component.

But that's not really my point: I was responding to Grant, who claimed 
that \0 is not a pathname (or filename) and therefore ValueError is the 
correct response. But there are lots of things which aren't pathnames, or 
even which *cannot be* pathnames, and yet they return False. What makes 
\0 so special?


> which means that it's exactly the same as /wibble -
> the first component doesn't exist, ergo the path doesn't exist.
> 
>> The maximum number of file components under POSIX is (I believe) 256.
>> And yet:
>>
>> py> os.path.exists("/a"*100)
>> False
>>
>> "/a" by one million cannot possibly be a path under POSIX.
> 
> I can't actually find that listed anywhere. Citation needed. 

https://eklitzke.org/path-max-is-tricky



> But
> assuming you're right, POSIX is still a set of minimum requirements -
> not maximums, to my knowledge. 

It isn't even a set of minimum requirements. "<" is legal under POSIX, 
but not Windows.



> If some operating system permits longer
> paths with more components, it won't be non-compliant on that basis. So
> it's still plausible to ask "does this path exist", and it's perfectly
> correct to look at the first "/a/" and check if there's anything named
> "a" in your root directory, and return False upon finding none. The
> question is sane, unlike os.path.exists([]).

Correct.

Just as it is sane to ask if path "a\0b" exists. If it happens to be 
illegal on POSIX, just as "<" is illegal under Windows, it is still sane 
to ask, and you should get False returned.




-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem with OrderedDict - progress report

[Chris Angelico]

On Sat, Jun 2, 2018 at 9:10 AM, Gregory Ewing
 wrote:
> Chris Angelico wrote:
>>
>> if you 'break' immediately after a mutation, that isn't
>> continuing to iterate. Even though you're inside the loop, there's no
>> further action taken to process the loop, and no problem.
>
>
> Yes, but you're also not calling next() again, so no
> exception would be triggered.
>
> My point is that I don't see a justification for treating
> the last next() call (the one that notices you've reached
> the end and raises StopIteration) any differently from
> the others. It seems like either a bug or a misfeature
> to me.
>

I'm guessing it's pure chance, and that certain types of mutation
would still trigger the exception.

A bit of background: These exceptions exist to prevent interpreter
crashes. I believe originally you'd only get an exception if the
mutation triggered a dictionary resize; in other words, if NOT giving
an exception would have actually segfaulted the interpreter right then
and there. (I'm talking CPython here, obviously. No idea what other
Pythons did at the time.) It's more consistent now; but it looks like
you may have come across a residual inconsistency. ISTM it would
indeed be more logical to raise for this situation too.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Chris Angelico]

On Sat, Jun 2, 2018 at 10:14 AM, Steven D'Aprano
 wrote:
>> But
>> assuming you're right, POSIX is still a set of minimum requirements -
>> not maximums, to my knowledge.
>
> It isn't even a set of minimum requirements. "<" is legal under POSIX,
> but not Windows.

Windows isn't POSIX compliant.

Anyhow, I've come to the conclusion that we're all about equally wrong
here, so I'm going to just stop arguing. Anyone who wants the
behaviour I described can get it easily enough via os.stat; and if you
want ValueError to turn into False, that's also easy enough.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[eryk sun]

On Sat, Jun 2, 2018 at 12:14 AM, Steven D'Aprano
 wrote:
>
> It isn't even a set of minimum requirements. "<" is legal under POSIX,
> but not Windows.

"<" (i.e. the DOS_STAR wildcard character) is valid in device and
stream names. It's only invalid in filenames, since it's reserved for
wildcard matching in file-system calls.

For example, in the following we have a device named
':' (note the name contains a forward slash) and a
stream named ''. The stream component requires NTFS,
ReFS, or CDFS; FAT doesn't support it. Stream names also allow ASCII
control characters, except NUL.

>>> DefineDosDevice(0, ':', 'C:\\Temp')
>>> f = open(r'\\?\:\FILENAME.TXT:', 'w')
>>> os.path.exists(r'\\?\:\FILENAME.TXT:')
True
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: version

[Mike McClain]

On Fri, Jun 01, 2018 at 08:02:27AM -0700, Mike McClain wrote:
> On Thu, May 31, 2018 at 07:44:35PM -0700, Mike McClain wrote:
> 
> > Is there a way in a script to know which version of python is being
> > run so I can write:
> > If (version == 2.7):
> > do it this way
> > elsif (version == 3.2):
> > do it another way
> >
>
> Thanks for the responses,

Those responses were sys.version_info.major and the module 'six'.
The first works here:
if( sys.version_info.major == 3 ):
choice = input(" to continue, or  to abort")
else:
choice = raw_input(" to continue, or  to abort")   #  OK in 2.7
but not here:
if( sys.version_info.major == 3 ):
print( row, sep=', ')
else:
print ', '.join(row)#   failing in 3.2

Both of the above print statements throw syntax errors, one under 2.7
the other under 3.2.

I looked at 'six' and it putting wrappers around calls would mean a
rewrite of what is basically just a playground for exploring what
python is and certainly not worth the trouble to rewrite much.

It looks like what I was wanting is something like 'C's #if, a
compiler conditional.

Does python have anything like that to tell the interpreter to ignore
a line that is not a comment or a quoted string?

Thanks,
Mike
--
The depression won't end till we grow a generation that knows how to
live on what they got.
- Will Rogers
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Grant Edwards]

On 2018-06-01, Steven D'Aprano  wrote:

> But since "\0" is the correct type (a string), and the fact that it 
> happens to be illegal on POSIX is a platform-dependent detail of no more 
> importance than the fact that "?" is illegal on Windows, it should be 
> treated as any other platform-dependent illegal file and return False.

That sounds reasonable.

What about the case where somebody calls

os.path.exists("/tmp/foo\x00bar")

If /tmp/foo exists should it return True?  That's what would happen if
you passed that string directly to the libc call.

--
Grant



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: version

[Chris Angelico]

On Sat, Jun 2, 2018 at 11:34 AM, Mike McClain  wrote:
> On Fri, Jun 01, 2018 at 08:02:27AM -0700, Mike McClain wrote:
>> On Thu, May 31, 2018 at 07:44:35PM -0700, Mike McClain wrote:
>> 
>> > Is there a way in a script to know which version of python is being
>> > run so I can write:
>> > If (version == 2.7):
>> > do it this way
>> > elsif (version == 3.2):
>> > do it another way
>> >
>>
>> Thanks for the responses,
>
> Those responses were sys.version_info.major and the module 'six'.
> The first works here:
> if( sys.version_info.major == 3 ):
> choice = input(" to continue, or  to abort")
> else:
> choice = raw_input(" to continue, or  to abort")   #  OK in 
> 2.7

Even better:

try: input = raw_input
except NameError: pass

and then just use input() everywhere.

> but not here:
> if( sys.version_info.major == 3 ):
> print( row, sep=', ')
> else:
> print ', '.join(row)#   failing in 3.2
>
> Both of the above print statements throw syntax errors, one under 2.7
> the other under 3.2.

Even better:

from __future__ import print_function

and then use the version 3 style everywhere.

> I looked at 'six' and it putting wrappers around calls would mean a
> rewrite of what is basically just a playground for exploring what
> python is and certainly not worth the trouble to rewrite much.
>
> It looks like what I was wanting is something like 'C's #if, a
> compiler conditional.
>
> Does python have anything like that to tell the interpreter to ignore
> a line that is not a comment or a quoted string?

No, but for each potential problem where you'd need it, you should
have an alternative that works within the current syntax. At least so
long as the oldest version you need to support is 2.7, anyway.

I would recommend upgrading to an even newer version of Python,
incidentally. Python 3.2 is itself extremely old. I'm not sure if
there are any Wheezy backports of newer versions of Python, but when I
was using Wheezy, I just compiled my own Python 3.x from source; the
Debian-provided /usr/bin/python3 isn't touched (and probably isn't
used anyway - on Wheezy, system scripts are all using 2.7 IIRC), and
you have a new /usr/local/bin/python3 that would be a lot newer.

The newer Python versions make cross-version compatibility a LOT easier.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Grant Edwards]

On 2018-06-01, Steven D'Aprano  wrote:
> On Thu, 31 May 2018 17:43:28 +, Grant Edwards wrote:
>
>> Except on the platform in quetion filenames _don't_ contain an embedded
>> \0.  What was passed was _not_ a path/filename.
>
> "/wibble/rubbish/nobodyexpectsthespanishinquistion" is not a pathname on 
> my system either,

I disagree.  On Unix systems that _is_ a path.  There may or may not
be a file that exists with that path.  OTOH "/wibble\x00/whatever" is
not a Unix path.

--
Grant




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: version

[Dan Stromberg]

On Fri, Jun 1, 2018 at 6:34 PM, Mike McClain  wrote:

> On Fri, Jun 01, 2018 at 08:02:27AM -0700, Mike McClain wrote:
> > On Thu, May 31, 2018 at 07:44:35PM -0700, Mike McClain wrote:
> > 
> > > Is there a way in a script to know which version of python is being
> > > run so I can write:
> > > If (version == 2.7):
> > > do it this way
> > > elsif (version == 3.2):
> > > do it another way
>


> I looked at 'six' and it putting wrappers around calls would mean a
> rewrite of what is basically just a playground for exploring what
> python is and certainly not worth the trouble to rewrite much.
>
> It looks like what I was wanting is something like 'C's #if, a
> compiler conditional.
>
> Does python have anything like that to tell the interpreter to ignore
> a line that is not a comment or a quoted string?
>

I've had some luck using m4 for preprocessing Python and Cython from the
same .py file.  However, it's often better to have 2 different files, as
evidenced by some of C's #ifdef'd files that get really complicated and
hard to test.

You're probably better off using 3.x at this point.

BTW, if you just want a simplistic way of doing print's that works on 2.x
and 3.x, there are some options:
1) sys.stdout.write(string)
2) print(string)
3) from __future__ import print_function

#2 is a bit of a trick/shenanigan that can be useful; it works because for
single-argument print's, the print function and the print statement both
work with parentheses.  You can even:

print('abc %s def %s' % ('123', 456))

On both.  Two 2.x, that's a parenthesized string passed to the print
statement, and to 3.x it's a single-argument print function.

HTH
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Serhiy Storchaka]

01.06.18 16:58, Chris Angelico пише:

Possibly more confusing, though, is this:


os.path.exists(1)

True

os.path.exists(2)

True

os.path.exists(3)

False

I think it's testing that the file descriptors exist, because
os.path.exists is defined in terms of os.stat, which can stat a path
or an FD. So os.path.exists(fd) is True if that fd is open, and False
if it isn't. But os.path.exists is not documented as accepting FDs.
Accident of implementation or undocumented feature? Or maybe
accidental feature?


Accident of implementation. In Python 3.3 os.stat() became accepting 
integer file descriptors (as os.fstat()). os.path.exists() just passes 
its argument to os.stat().


--
https://mail.python.org/mailman/listinfo/python-list

Re: Why exception from os.path.exists()?

[Serhiy Storchaka]

02.06.18 03:05, Chris Angelico пише:

The permissions error is reported differently by stat, but then
exists() just says "oh that's an OS error so we're going to say it
doesn't exist". If you truly want the reliable form of os.path.exists,
it would be this:

try:
 os.stat(path)
 return True
except FileNotFoundError:
 return False

Anything that ISN'T "this file exists" or "this file doesn't exist"
will be signalled with an exception.


And this is how pathlib.Path.exists() was implemented. But for 
os.path.exists() we are limited by backward compatibility.


--
https://mail.python.org/mailman/listinfo/python-list