Re: File names with slashes [was Re: error in os.chdir]

[Gene Heskett]

On Thursday 05 July 2018 11:57:18 Mikhail V wrote:

> Steven D'Aprano wrote:
> > In Explorer and the open-file dialog of most applications, they will
> > see paths like this:
> >
> > directory\file name with spaces
> >
> > with the extension (.jpg, .pdf, .docx etc) suppressed. So by your
> > argument, Python needs to accept strings without quotes:
> >
> > open(directory\file name with spaces, 'r')
> >
> > and guess which extension you mean.
> >
> > That would be fun to watch in action.
>
> I think it's what is called 'English humor'?
> Funny (just a little bit).
>
> Wait a moment - you are well aware of Windows features,
> that's suspicious...
>
>
> BTW Explorer with hidden extension that's a sign of
> "double-click, drag-and-drop" category of users.
> Definitely more advanced users turn on extensions,
> or don't even use explorer as a file manager.
>
> > Linux programmers will
> > see paths with spaces and other metacharacters escaped:
> >
> > directory/file\ name\ with\ spaces.txt
>
> ick. what I see is escaping of the most frequent Latin character.

ick  is a very weak adjective.

For a change, why can't windows do it right?  Oh wait, its windows so the 
question is moot.  Sigh, and just one of the reasons there are zero 
windows machines on my premises.


-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 
-- 
https://mail.python.org/mailman/listinfo/python-list

File names with slashes [was Re: error in os.chdir]

[Mikhail V]

Steven D'Aprano wrote:

> In Explorer and the open-file dialog of most applications, they will see
> paths like this:
>
> directory\file name with spaces
>
> with the extension (.jpg, .pdf, .docx etc) suppressed. So by your
> argument, Python needs to accept strings without quotes:
>
> open(directory\file name with spaces, 'r')
>
> and guess which extension you mean.
>
> That would be fun to watch in action.


I think it's what is called 'English humor'?
Funny (just a little bit).

Wait a moment - you are well aware of Windows features,
that's suspicious...


BTW Explorer with hidden extension that's a sign of
"double-click, drag-and-drop" category of users.
Definitely more advanced users turn on extensions,
or don't even use explorer as a file manager.

> Linux programmers will
> see paths with spaces and other metacharacters escaped:
>
> directory/file\ name\ with\ spaces.txt


ick. what I see is escaping of the most frequent Latin character.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Steven D'Aprano]

On Thu, 05 Jul 2018 00:56:22 +0300, Mikhail V wrote:

> for the user it is most important to
> *see* and copy-paste the path string exactly as it is displayed
> everywhere else on windows.

So in Windows, you see:

dir directory\file.pdf

so in Python, we have to use exactly the same path with absolutely no 
changes:

open(directory\file.pdf, 'r')

In Explorer and the open-file dialog of most applications, they will see 
paths like this:

directory\file name with spaces


with the extension (.jpg, .pdf, .docx etc) suppressed. So by your 
argument, Python needs to accept strings without quotes:

open(directory\file name with spaces, 'r')

and guess which extension you mean.

That would be fun to watch in action.

Python programmers on Macs will see lots of file paths that display with 
colons, like this:

directory:file.txt

and mentally translate it to have a forward slash. Linux programmers will 
see paths with spaces and other metacharacters escaped:

directory/file\ name\ with\ spaces.txt

and mentally translate it to remove the backslashes. If they're not 
already doing this, they'll surely have to do it soon, when unused 
backslash escapes become an error.

https://bugs.python.org/issue32912


Do you believe Python programmers on Windows are especially dumber than 
Python programmers on Macs and Linux? I don't, but you seem to want to 
coddle them and protect them from having to think about what they're 
doing. I think that's patronising of your fellow Windows users.


I'm reminded of a quote by the late Terry Pratchett, an English author, 
talking about his experience publishing books in the US. If I remember 
correctly, he said that American readers aren't dumb, but American 
publishers are convinced that they are.



-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Gregory Ewing]

Mikhail V wrote:

There is one issue that I can't write \ on the end:
r"C:\programs\util\"

But since I know it's a path and not a file, I just write without trailing \.


Indeed. There's never a need to put a backslash on the end of
a path, as long as you always use os.path functions or
equivalent to manipulate them. Which is a good idea anyway,
since it will make things easier if anyone ever wants to
run the code on something other than Windows.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

File names with slashes [was Re: error in os.chdir]

[Mikhail V]

ChrisA wrote:

> Mikhail V wrote:
>> Yes, and the answer was a week ago: just put "r" before  the string.
>> r"C:\programs\util"
>>
>> And it worked till now. So why should I replace backslashes with
>> forward slashes?
>> There is one issue that I can't write \ on the end:
>> r"C:\programs\util\"
>>
>> But since I know it's a path and not a file, I just write without trailing \.

> That's exactly the issue.

If its about trailing \ - then this is issue with the syntax - not
with the path.


> But if you just always separate paths with
> forward slashes, you never have a problem. There is no "replace"
> happening; you simply use the correct path separation character from
> the start.

I am on windows, and the OP is (othrwise why would he ask?).
If next version of windows will use forward slashes everywhere,
then yes - there will be no "replace" happening.

To the subject of the question - for the user it is most important to
*see* and copy-paste the path string exactly as it is displayed everywhere
else on windows.
Other problems, like designing cross-platform utils, etc. are not really
relevant to the simple use case (on windows).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Mark Lawrence]

On 04/07/18 21:30, Chris Angelico wrote:

On Thu, Jul 5, 2018 at 6:25 AM, Mikhail V  wrote:

Joe Pfeiffer wrote:


On Windows a path is e.g.:
C:\programs\util\
So what is reasonable about using forward slashes?
It happens to me that I need to copy-paste real paths like 100 times
a day into scripts - do you propose to convert to forward slashes each time?



That's what started the thread -- using backslashes caused a \a to be
interpreted as a special character instead of two characters in the
path.


Yes, and the answer was a week ago: just put "r" before  the string.
r"C:\programs\util"

And it worked till now. So why should I replace backslashes with
forward slashes?
There is one issue that I can't write \ on the end:
r"C:\programs\util\"

But since I know it's a path and not a file, I just write without trailing \.


That's exactly the issue. But if you just always separate paths with
forward slashes, you never have a problem. There is no "replace"
happening; you simply use the correct path separation character from
the start. You can assemble strings from pieces without having to
remember to slap an "r" in front of each literal; you can have slashes
at the ends of strings; everything will just work.

ChrisA



Maybe it would be smarter to build paths using os.sep, os.path.join and 
os.path.split rather than use literal strings.  There is also os.altsep 
https://docs.python.org/3/library/os.html#os.altsep which if I'd heard 
about I'd forgotten about :)  And others.  Failing that there's always 
pathlib https://docs.python.org/3/library/pathlib.html


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Chris Angelico]

On Thu, Jul 5, 2018 at 6:25 AM, Mikhail V  wrote:
> Joe Pfeiffer wrote:
>
>>> On Windows a path is e.g.:
>>> C:\programs\util\
>>> So what is reasonable about using forward slashes?
>>> It happens to me that I need to copy-paste real paths like 100 times
>>> a day into scripts - do you propose to convert to forward slashes each time?
>
>> That's what started the thread -- using backslashes caused a \a to be
>> interpreted as a special character instead of two characters in the
>> path.
>
> Yes, and the answer was a week ago: just put "r" before  the string.
> r"C:\programs\util"
>
> And it worked till now. So why should I replace backslashes with
> forward slashes?
> There is one issue that I can't write \ on the end:
> r"C:\programs\util\"
>
> But since I know it's a path and not a file, I just write without trailing \.

That's exactly the issue. But if you just always separate paths with
forward slashes, you never have a problem. There is no "replace"
happening; you simply use the correct path separation character from
the start. You can assemble strings from pieces without having to
remember to slap an "r" in front of each literal; you can have slashes
at the ends of strings; everything will just work.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

File names with slashes [was Re: error in os.chdir]

[Mikhail V]

Joe Pfeiffer wrote:

>> On Windows a path is e.g.:
>> C:\programs\util\
>> So what is reasonable about using forward slashes?
>> It happens to me that I need to copy-paste real paths like 100 times
>> a day into scripts - do you propose to convert to forward slashes each time?

> That's what started the thread -- using backslashes caused a \a to be
> interpreted as a special character instead of two characters in the
> path.

Yes, and the answer was a week ago: just put "r" before  the string.
r"C:\programs\util"

And it worked till now. So why should I replace backslashes with
forward slashes?
There is one issue that I can't write \ on the end:
r"C:\programs\util\"

But since I know it's a path and not a file, I just write without trailing \.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Joe Pfeiffer]

Mikhail V  writes:

> [Steven D'Aprano]
>
>> (The same applies to Unix/Linux systems too, of course.) But while you're
>> using Python to manipulate files, you should use Python rules, and that
>> is "always use forward slashes".
>>
>> Is that reasonable?
>>
>> Under what circumstances would a user calling open(pathname) in Python
>> need to care about backslashes?
>
> Cough cough
>
> On Windows a path is e.g.:
> C:\programs\util\
>
> So why should I use forward slashes in a Python literal?
> I don't remember any problem caused by using backslashes in paths in Python -
> are there problems?
> (Apart from the fact that Python dos not have true raw string literals)
>
> So what is reasonable about using forward slashes?
> It happens to me that I need to copy-paste real paths like 100 times
> a day into scripts - do you propose to convert to forward slashes each time?

That's what started the thread -- using backslashes caused a \a to be
interpreted as a special character instead of two characters in the
path.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Thomas Jollans]

On 2018-07-03 14:06, Mikhail V wrote:
> Greg wrote:
> 
>> Mikhail V wrote:
>>>   s= "\"s\""   ->
>>>   s=  {"s"}
>>
>> But now you need to find another way to represent set literals.
> 
> 
> I need to find? That comment was not about (current) Python but
> rather how I think string should have been from the beginning.
> 
> So you already like it and want in Python ?  :-)
> Fitting it into current Python would need some prefix e.g.
> 
> s = !{hello}

Nothing new under the sun.

Our old frenemy Perl has
  q{abc""} == "abc\"\""
  q`{}` == "{}"

Perl's sexy Japanese alter-ego Ruby has %Q{}, %q{}.

I'm sure there are more.
-- 
https://mail.python.org/mailman/listinfo/python-list

File names with slashes [was Re: error in os.chdir]

[Mikhail V]

Greg wrote:

> Mikhail V wrote:
> >   s= "\"s\""   ->
> >   s=  {"s"}
>
> But now you need to find another way to represent set literals.


I need to find? That comment was not about (current) Python but
rather how I think string should have been from the beginning.

So you already like it and want in Python ?  :-)
Fitting it into current Python would need some prefix e.g.

s = !{hello}
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Gregory Ewing]

Mikhail V wrote:

  s= "\"s\""   ->
  s=  {"s"}


But now you need to find another way to represent set literals.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

File names with slashes [was Re: error in os.chdir]

[Mikhail V]

[Chris A]

> [Mikhail]
> > So Imo default syntax should be something like:
> >
> > S = "A:{x41}B:{x42}"
> >
> > instead of backslashes and Co.
>
> So how do you represent brace characters in a string?


\{ and \}


just kidding :-D
I would be ok with {L} and {R} - easy on eye and easy to remember.
Why don't ask how would I represent double-quote character?
It's more important, but I would not use double quote as string delimiter,
I'd use {} as delimiter:

  s= "\"s\""   ->
  s=  {"s"}

is much better.

> > 2. raw strings, including multiline raw strings which should be PEP-8 
> > compliant.
>
> ???

Raw strings - as I suggested here like 3 weeks ago or so?

S = `op`
multi-line raw string block
parsed by indentation
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[MRAB]

On 2018-07-02 18:16, Richard Damon wrote:

On 7/2/18 9:20 AM, Mikhail V wrote:

[Richard Damon]


The one major issue with backslashes is that they are a special
character in string literals, so you either need to use raw literals a
remember the few cases they still act as special characters, or remember
to convert them to double back slashes, at a minimum for all the
characters that they are special for (easier to double them all).
I think it was originally an error to make the backslash followed by a
character not defined as special with a backslash as keeping the
backslash as a literal as it causes a number of these issues. Yes, it
allows you to not need to double it in many cases but that just sets you
up for the mistakes that started the thread. It is probably too late to
change that behavior now though.

Yes this would at least make less mistakes.
I find the whole situation with strings a bit disappointing.
On the one hand - there were so many string types added, on the other
hand - there are still many inconveniences. There is an english proverb,
"it does too much, but still too little". (or something like that)

The initial string syntax -- I think it's direct copy of C strings syntax.
And it sucks. (path dependency?)

I'd say there should be just two main string types:
1. string literal where all special character goes into, say, figure braces {};
(if only there was a time machine)
2. raw strings, including multiline raw strings which should be PEP-8 compliant.

So Imo default syntax should be something like:

S = "A:{x41}B:{x42}"

instead of backslashes and Co.


Yes, the backslash notation is just like what C uses (an likely the
source of it). The exact same problem pops up in the C forums too. Not
sure where it got the notation from.


C is descended from BCPL, which used '*', e.g. newline was "*N".
--
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Richard Damon]

On 7/2/18 9:20 AM, Mikhail V wrote:
> [Richard Damon]
>
>> The one major issue with backslashes is that they are a special
>> character in string literals, so you either need to use raw literals a
>> remember the few cases they still act as special characters, or remember
>> to convert them to double back slashes, at a minimum for all the
>> characters that they are special for (easier to double them all).
>> I think it was originally an error to make the backslash followed by a
>> character not defined as special with a backslash as keeping the
>> backslash as a literal as it causes a number of these issues. Yes, it
>> allows you to not need to double it in many cases but that just sets you
>> up for the mistakes that started the thread. It is probably too late to
>> change that behavior now though.
> Yes this would at least make less mistakes.
> I find the whole situation with strings a bit disappointing.
> On the one hand - there were so many string types added, on the other
> hand - there are still many inconveniences. There is an english proverb,
> "it does too much, but still too little". (or something like that)
>
> The initial string syntax -- I think it's direct copy of C strings syntax.
> And it sucks. (path dependency?)
>
> I'd say there should be just two main string types:
> 1. string literal where all special character goes into, say, figure braces 
> {};
> (if only there was a time machine)
> 2. raw strings, including multiline raw strings which should be PEP-8 
> compliant.
>
> So Imo default syntax should be something like:
>
> S = "A:{x41}B:{x42}"
>
> instead of backslashes and Co.

Yes, the backslash notation is just like what C uses (an likely the
source of it). The exact same problem pops up in the C forums too. Not
sure where it got the notation from.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Chris Angelico]

On Tue, Jul 3, 2018 at 2:20 AM, Mikhail V  wrote:
> [Richard Damon]
>
>> The one major issue with backslashes is that they are a special
>> character in string literals, so you either need to use raw literals a
>> remember the few cases they still act as special characters, or remember
>> to convert them to double back slashes, at a minimum for all the
>> characters that they are special for (easier to double them all).
>
>> I think it was originally an error to make the backslash followed by a
>> character not defined as special with a backslash as keeping the
>> backslash as a literal as it causes a number of these issues. Yes, it
>> allows you to not need to double it in many cases but that just sets you
>> up for the mistakes that started the thread. It is probably too late to
>> change that behavior now though.
>
> Yes this would at least make less mistakes.
> I find the whole situation with strings a bit disappointing.
> On the one hand - there were so many string types added, on the other
> hand - there are still many inconveniences. There is an english proverb,
> "it does too much, but still too little". (or something like that)
>
> The initial string syntax -- I think it's direct copy of C strings syntax.
> And it sucks. (path dependency?)
>
> I'd say there should be just two main string types:
> 1. string literal where all special character goes into, say, figure braces 
> {};
> (if only there was a time machine)
> 2. raw strings, including multiline raw strings which should be PEP-8 
> compliant.
>
> So Imo default syntax should be something like:
>
> S = "A:{x41}B:{x42}"
>
> instead of backslashes and Co.

So how do you represent brace characters in a string?

> 2. raw strings, including multiline raw strings which should be PEP-8 
> compliant.

???

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Mikhail V]

[Richard Damon]

> The one major issue with backslashes is that they are a special
> character in string literals, so you either need to use raw literals a
> remember the few cases they still act as special characters, or remember
> to convert them to double back slashes, at a minimum for all the
> characters that they are special for (easier to double them all).

> I think it was originally an error to make the backslash followed by a
> character not defined as special with a backslash as keeping the
> backslash as a literal as it causes a number of these issues. Yes, it
> allows you to not need to double it in many cases but that just sets you
> up for the mistakes that started the thread. It is probably too late to
> change that behavior now though.

Yes this would at least make less mistakes.
I find the whole situation with strings a bit disappointing.
On the one hand - there were so many string types added, on the other
hand - there are still many inconveniences. There is an english proverb,
"it does too much, but still too little". (or something like that)

The initial string syntax -- I think it's direct copy of C strings syntax.
And it sucks. (path dependency?)

I'd say there should be just two main string types:
1. string literal where all special character goes into, say, figure braces {};
(if only there was a time machine)
2. raw strings, including multiline raw strings which should be PEP-8 compliant.

So Imo default syntax should be something like:

S = "A:{x41}B:{x42}"

instead of backslashes and Co.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Karsten Hilbert]

Eryk,

thanks for your to-the-point in-depth posts.

Karsten
-- 
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[eryk sun]

On Sun, Jul 1, 2018 at 4:30 PM, eryk sun  wrote:
> On Sun, Jul 1, 2018 at 8:51 AM, Steven D'Aprano
>
>> spam/eggs
[...]
>> And how would that file be displayed in the Windows GUI file explorer?
>
> I suppose if a file system allowed forward slash in names that
> Explorer would just display it. When I get the time, I'll attach a
> debugger to explorer.exe and hack a forward slash into one of the
> names in a directory listing, i.e. a result from WinAPI FindFirstFile
> (or native NtQueryDirectoryFile).

I checked this by attaching a debugger to an instance of explorer.exe
and opening a folder that has a single file named "spam eggs.txt".
With a breakpoint on NtQueryDirectoryFile, I changed the filename to
"spam/eggs.txt" in the directory listing (an array of
FILE_ID_BOTH_DIR_INFORMATION records). As expected, Explorer just
displays the filename "spam/eggs.txt" if that's what the file system
returns from the NtQueryDirectoryFile system call. However, even if a
file system allowed such a name, it wouldn't be accessible normally.
It requires a \\?\ path, which bypasses canonicalization.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Gregory Ewing]

eryk sun wrote:

Python 2 raw strings are half-baked.


Obviously the "r" actually stand for "rare".

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[eryk sun]

On Sun, Jul 1, 2018 at 4:00 PM, Abdur-Rahmaan Janhangeer
 wrote:
> one common scenario is
>
> C:\Users\
>
> where \U is taken as a unicode litteral

This one is especially annoying in Python 2, since it makes raw
unicode strings useless for common path literals. For example:

>>> ur'C:\Users'
  File "", line 1
SyntaxError: (unicode error) 'rawunicodeescape' codec can't decode
bytes in position 2-3: truncated \u

>>> ur'C:\users'
  File "", line 1
SyntaxError: (unicode error) 'rawunicodeescape' codec can't decode
bytes in position 2-3: truncated \u

Python 2 raw strings are half-baked.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[eryk sun]

On Sun, Jul 1, 2018 at 8:51 AM, Steven D'Aprano
 wrote:
> On Sun, 01 Jul 2018 03:18:23 +, eryk sun wrote:
>
>> Bear in mind that forward slash is just a  name character in NT.
>
> So, using Python, how could you open, write to, and then read from, a
> file with a slash in its name? Say, something like:

By that I meant that NT's base object namespace treats forward slash
as a name character, i.e. object names in the base NT namespace can
include slash. This includes Device objects in the "\Device" directory
and "DOS" device symbolic links to them in "\Global??" or a
subdirectory of "\Sessions\0\DosDevices", as well as other named
objects (Section, Job, Event, Semaphore, etc). The Windows API stores
most of the latter in a single per-session directory that's named
"\BaseNamedOjects" for session 0 (system services) and
"\Sessions\\BaseNamedObjects" for interactive
sessions.

The base registry Key object is named "\Registry". Its parse procedure
(for creating and opening key paths) only reserves backslash, which is
the same as the object namespace itself.

> spam/eggs

You can create a named pipe with that name in the named-pipe file
system. But regular file systems that I've used in Windows reserve
forward slash, even though they don't use it for anything. They'll
just fail the create/open with STATUS_OBJECT_NAME_INVALID. The way to
observe this is to use forward slash in a \\?\ prefixed path. Note
that the prefix must be \\?\, with backslash only. //?/ is basically
the same as //./.

As far as file paths go, "spam/eggs" could be set as a device name,
which from Windows would only be accessible with a \\?\ prefixed path.
For example:

DefineDosDevice(DDD_RAW_TARGET_PATH, 'spam/eggs',
'\\Device\\HarddiskVolume2\\Temp')

f = open(r'\\?\spam/eggs\test.txt', 'w')
h = msvcrt.get_osfhandle(f.fileno())

>>> GetFinalPathNameByHandle(h, VOLUME_NAME_DOS)
'?\\C:\\Temp\\test.txt'

>>> GetFinalPathNameByHandle(h, VOLUME_NAME_NT)
'\\Device\\HarddiskVolume2\\Temp\\test.txt'

> in your home directory. (Is that still C:\\My Documents by default?)

No, the user profile directory is "C:\Users\", i.e. the
%UserProfile% environment variable. In some cases a user may have a
separate home directory configured, which is set in
%HOMEDRIVE%%HOMEPATH%. Commonly no home directory is set, in which
case %HOMEDRIVE%%HOMEPATH% is just the fixed %UserProfile% path.

> And how would that file be displayed in the Windows GUI file explorer?

I suppose if a file system allowed forward slash in names that
Explorer would just display it. When I get the time, I'll attach a
debugger to explorer.exe and hack a forward slash into one of the
names in a directory listing, i.e. a result from WinAPI FindFirstFile
(or native NtQueryDirectoryFile).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Abdur-Rahmaan Janhangeer]

one common scenario is

C:\Users\

where \U is taken as a unicode litteral

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ

>
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File names with slashes [was Re: error in os.chdir]

[Richard Damon]

On 7/1/18 6:49 AM, Mikhail V wrote:
> [Steven D'Aprano]
>
>> (The same applies to Unix/Linux systems too, of course.) But while you're
>> using Python to manipulate files, you should use Python rules, and that
>> is "always use forward slashes".
>>
>> Is that reasonable?
>>
>> Under what circumstances would a user calling open(pathname) in Python
>> need to care about backslashes?
> Cough cough
>
> On Windows a path is e.g.:
> C:\programs\util\
>
> So why should I use forward slashes in a Python literal?
> I don't remember any problem caused by using backslashes in paths in Python -
> are there problems?
> (Apart from the fact that Python dos not have true raw string literals)
>
> So what is reasonable about using forward slashes?
> It happens to me that I need to copy-paste real paths like 100 times
> a day into scripts - do you propose to convert to forward slashes each time?

The one major issue with backslashes is that they are a special
character in string literals, so you either need to use raw literals a
remember the few cases they still act as special characters, or remember
to convert them to double back slashes, at a minimum for all the
characters that they are special for (easier to double them all).

I think it was originally an error to make the backslash followed by a
character not defined as special with a backslash as keeping the
backslash as a literal as it causes a number of these issues. Yes, it
allows you to not need to double it in many cases but that just sets you
up for the mistakes that started the thread. It is probably too late to
change that behavior now though.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[eryk sun]

On Sun, Jul 1, 2018 at 8:50 AM, Steven D'Aprano
 wrote:
> On Sun, 01 Jul 2018 03:18:23 +, eryk sun wrote:
>
>> The Windows API handles this, but not for a path that begins with \\?\.
>
> But what about the *Python* API? The Python open() function, and all the
> high-level os.* and os.path.* functions which take paths as strings are
> supposed to automatically convert forward slashes to backslashes. Aren't
> they?

Python's I/O functions don't modify file paths, except to decode bytes
paths in 3.6+.

os.path.normpath uses backslash in Windows, without involving the OS,
as does pathlib in Python 3. os.path.abspath calls WinAPI
GetFullPathName to get a canonical, fully-qualified path, including
transforming reserved paths, e.g.

>>> os.path.abspath('C:/Temp/conout$.txt')
'.\\conout$'

>>> os.path.abspath('C:/Temp/spam. . .')
'C:\\Temp\\spam'

(Note that the above "conout$" example is for Windows 8 and 10. In
prior versions, "conin$" and "conout$"  aren't reserved in
qualified-relative or absolute paths, only when passed to CreateFile
as unqualified names.)

The \\?\ prefix allows using otherwise reserved names such as "con"
and "spam...".

For versions of Windows prior to 10 or Python prior to 3.6, the \\?\
prefix is also the only way to get past the 260 character limit of
legacy DOS paths to work with paths with up to 32K characters. Python
3.6+ is manifested to support long paths in Windows 10 without
requiring the \\?\ prefix. This can be enabled by setting the
LongPathsEnabled value in
"HKLM\System\CurrentControlSet\Control\FileSystem". Maybe in a future
version this registry setting will be enabled by default.

>> Bear in mind that forward slash is just a  name character in NT.
>
> Interesting...
>
> That's rather like on HFS Plus, where / is an ordinary character allowed
> in file names (although Mac OS goes to extraordinary effort to ensure
> that tools that expect / as a path separator see it, and those which
> expect : as the path separator see it instead).

Many characters are reserved in Windows file systems that aren't
generally reserved in NT, including forward slash. What happens here
is that the Object Manager hands off parsing the rest of a path as
soon as it reaches an object that has a parse procedure. This allows
object types such as Key and Device objects to implement custom
namespaces.

The Configuration Manager's (registry) Key parse procedure limits
names to 256 characters and the tree depth to 512 keys. (Note that
HKCU is a pseudo-handle for "\Registry\User\", so subtract 3 from
the depth.) However it doesn't reserve characters, except for the
native backslash path separator. IIRC it allows NUL in key names, but
that would render them impossible to access in the Windows API, which
uses null-terminated strings.

The I/O Manager's Device parse procedure sends an IRP_MJ_CREATE I/O
request packet either to the first device in the stack (including
lower/upper filter drivers), or to the file-system device stack that's
managing it. The I/O manager doesn't reserve additional characters or
limit name length, but most file systems limit names to 255 characters
(127 for UDF) and reserve backslash (path separator), slash, colon,
pipe, wildcards (*?<>"), and ASCII controls. In contrast, the named
pipe file system (i.e. WinAPI \\.\PIPE or native \Device\NamedPipe) is
extremely lenient. Pipe names can be up to 259 characters and can
include any character except NUL. This file system doesn't support
directories, so pipe names can even include backslash, which is a
common practice.

Except for disk volumes, most devices aren't file systems or managed
by one. However, sometimes they do implement a simple namespace of
hard-coded names. For example, the console device "\Device\ConDrv"
implements a few names such as "Connect", "Input", "Output", "Console"
(con), "CurrentIn" (conin$), and "CurrentOut" (conout$).
-- 
https://mail.python.org/mailman/listinfo/python-list

File names with slashes [was Re: error in os.chdir]

[Mikhail V]

[Steven D'Aprano]

> (The same applies to Unix/Linux systems too, of course.) But while you're
> using Python to manipulate files, you should use Python rules, and that
> is "always use forward slashes".
>
> Is that reasonable?
>
> Under what circumstances would a user calling open(pathname) in Python
> need to care about backslashes?

Cough cough

On Windows a path is e.g.:
C:\programs\util\

So why should I use forward slashes in a Python literal?
I don't remember any problem caused by using backslashes in paths in Python -
are there problems?
(Apart from the fact that Python dos not have true raw string literals)

So what is reasonable about using forward slashes?
It happens to me that I need to copy-paste real paths like 100 times
a day into scripts - do you propose to convert to forward slashes each time?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Peter J. Holzer]

On 2018-07-01 08:50:22 +, Steven D'Aprano wrote:
> On Sun, 01 Jul 2018 03:18:23 +, eryk sun wrote:
> > On Sun, Jul 1, 2018 at 1:44 AM, Steven D'Aprano
> >  wrote:
> >> I guess that if the user is using a path beginning with \\?\ they may
> >> or may not need to use backslashes, but I have no way of testing it,
> >> and I would expect that Python will correctly replace //?/ with
> >> backslashes the same as it does for any other file system path.
> > 
> > The Windows API handles this, but not for a path that begins with \\?\.
> 
> But what about the *Python* API? The Python open() function, and all the 
> high-level os.* and os.path.* functions which take paths as strings are 
> supposed to automatically convert forward slashes to backslashes. Aren't 
> they?

Are they? I don't see anything in the docs that support this (but maybe
my Google Fu is weak).

Rather from the existence and design of os.path I conclude that the
programmer is supposed to construct path names in the local convention
(i.e., you should use os.path.join(dirname, filename) instead of dirname +
"/" + filename, which will ensure that the correct separator is used).

I think the lazy approach using just forward slashes works because
*Windows* treats slashes in filenames like backslashes (most of the
time), not because Python converts them.

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

File names with slashes [was Re: error in os.chdir]

[Steven D'Aprano]

On Sun, 01 Jul 2018 03:18:23 +, eryk sun wrote:

> Bear in mind that forward slash is just a  name character in NT.


So, using Python, how could you open, write to, and then read from, a 
file with a slash in its name? Say, something like:

spam/eggs

in your home directory. (Is that still C:\\My Documents by default?)

And how would that file be displayed in the Windows GUI file explorer?



-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Steven D'Aprano]

On Sun, 01 Jul 2018 03:18:23 +, eryk sun wrote:

> On Sun, Jul 1, 2018 at 1:44 AM, Steven D'Aprano
>  wrote:
>> On Sat, 30 Jun 2018 23:36:40 +, eryk sun wrote:
>>
>>> Only use forward slashes for legacy DOS paths passed to Windows API
>>> functions. Do not use forward slashes for paths in command line
>>> arguments, \\?\ prefixed paths, or registry paths.
>>
>> I don't see why this is relevant, or at least not the "command line
>> arguments" and "registry paths" parts.
> 
> Command-line arguments are relevant to executing a child process, e.g.
> via subprocess.Popen.

Ah, yes they are. But using subprocess is a little more of an advanced 
use than regular file handling. But your point is taken.


>> I guess that if the user is using a path beginning with \\?\ they may
>> or may not need to use backslashes, but I have no way of testing it,
>> and I would expect that Python will correctly replace //?/ with
>> backslashes the same as it does for any other file system path.
> 
> The Windows API handles this, but not for a path that begins with \\?\.

But what about the *Python* API? The Python open() function, and all the 
high-level os.* and os.path.* functions which take paths as strings are 
supposed to automatically convert forward slashes to backslashes. Aren't 
they?


> The intent of this prefix is to bypass DOS-path normalization --
> allowing paths with length up to 32K characters that can use otherwise
> reserved DOS-device names or names ending in spaces or dots.
> 
>> (Besides, Python doesn't have an API for interacting directly with the
>> registry.)
> 
> The standard library has winreg -- or _winreg in 2.x.

Oh, I live and learn.


> Bear in mind that forward slash is just a  name character in NT.

Interesting... 

That's rather like on HFS Plus, where / is an ordinary character allowed 
in file names (although Mac OS goes to extraordinary effort to ensure 
that tools that expect / as a path separator see it, and those which 
expect : as the path separator see it instead).


> We only need to clearly state that the
> Windows API allows using forward slash as the path separator in file
> paths that do not begin with the \\?\ prefix. I prefer to also add a
> warning about command-line arguments, since it isn't an improbable or
> inconsequential problem

Fair enough.


-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Steven D'Aprano]

On Sun, 01 Jul 2018 02:22:41 +, eryk sun wrote:

> I use the native API a lot, so for me registry and file paths are just
> paths. It's only the Windows API that separates the two and only the
> Windows API that allows forward slash as a path separator in file paths.

Not being a Windows user, I don't actually understand how this works.

I gather that there's a low-level "native API" (DOS?) used by the Windows 
OS, and a higher level "Windows API" used by, er, another part of the API.

But I'm not sure how that interacts with Python.

The way I see it, when we say "always use forward slashes for file 
paths", we're talking about Python level file commands like open(), 
os.path.exists, etc.

If you're using Python to directly call Windows external programs, say 
"os.system('dir')", then of course you have to follow the Windows rules. 
(The same applies to Unix/Linux systems too, of course.) But while you're 
using Python to manipulate files, you should use Python rules, and that 
is "always use forward slashes".

Is that reasonable?

Under what circumstances would a user calling open(pathname) in Python 
need to care about backslashes?


-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[eryk sun]

On Sun, Jul 1, 2018 at 1:44 AM, Steven D'Aprano
 wrote:
> On Sat, 30 Jun 2018 23:36:40 +, eryk sun wrote:
>
>> Only use forward slashes for legacy DOS paths passed to Windows API
>> functions. Do not use forward slashes for paths in command line
>> arguments, \\?\ prefixed paths, or registry paths.
>
> I don't see why this is relevant, or at least not the "command line
> arguments" and "registry paths" parts.

Command-line arguments are relevant to executing a child process, e.g.
via subprocess.Popen.

> I guess that if the user is using a path beginning with \\?\ they may or
> may not need to use backslashes, but I have no way of testing it, and I
> would expect that Python will correctly replace //?/ with backslashes the
> same as it does for any other file system path.

The Windows API handles this, but not for a path that begins with
\\?\. The intent of this prefix is to bypass DOS-path normalization --
allowing paths with length up to 32K characters that can use otherwise
reserved DOS-device names or names ending in spaces or dots.

> (Besides, Python doesn't have an API for interacting directly with the
> registry.)

The standard library has winreg -- or _winreg in 2.x.

Bear in mind that forward slash is just a  name character in NT. If a
coder assumes that Windows automatically replaces forward slash with
backslash for registry paths, it could lead to a mistake such as the
following:

>>> hkey = winreg.CreateKey(winreg.HKEY_CURRENT_USER, 'test')
>>> subkeya = winreg.CreateKey(hkey, 'test_a1/test_a2/test_a3')
>>> subkeyb = winreg.CreateKey(hkey, 'test_b1\\test_b2\\test_b3')
>>> winreg.EnumKey(hkey, 0)
'test_a1/test_a2/test_a3'
>>> winreg.EnumKey(hkey, 1)
'test_b1'

In the test_a case, instead of a tree of keys
"test_a1\test_a2\test_a3", we get a single key named
"test_a1/test_a2/test_a3". The test_b case creates the intended tree.

I grant that having to explicitly say that forward slash isn't
supported in registry paths is certainly beyond what's required, as is
having to state that forward slash isn't supported in the native API
(e.g. NtCreateFile, NtOpenFile). We only need to clearly state that
the Windows API allows using forward slash as the path separator in
file paths that do not begin with the \\?\ prefix. I prefer to also
add a warning about command-line arguments, since it isn't an
improbable or inconsequential problem, and it's so easy to state
upfront rather than letting someone waste hours unravelling a cryptic
failure, given the path separator isn't something one is likely to
consider as the source of the problem.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Chris Angelico]

On Sun, Jul 1, 2018 at 12:43 PM, eryk sun  wrote:
> On Sun, Jul 1, 2018 at 2:28 AM, Chris Angelico  wrote:
>> On Sun, Jul 1, 2018 at 12:22 PM, eryk sun  wrote:
 So what if, internally, that's done by converting them to backslashes?
 No Python program needs to care. In fact, there are other conversions,
 too - the underlying file system is most likely using UTF-16 paths,
 but your Python program needn't use UTF-16. No, it simply uses a plain
 old string literal - a Unicode string. The OP need not be concerned
 about any of these fiddlinesses, unless in some way they actually
 affect Python.
>>>
>>> A few minor changes in in wording are both technically correct and
>>> don't make the text any more difficult to understand. You're the one
>>> blowing this up beyond proportion and making a mountain out of a small
>>> technical-correction mole hill. Please just let it pass.
>>
>> If you can show me a way in which a Python program actually needs to
>> care about any of these distinctions when dealing with files on
>> Windows, then I'll admit that the technical correction was needed. You
>> won't be shelling out to the "dir" command, for instance.
>
> Python has no built-in support for Windows file security, so a Python
> program may decided to use icacls.exe, takeown.exe, or xcopy.exe (copy
> owner/ACL). It's best in general to normalize paths when integrating
> an external program -- unless you've already checked that the
> program(s) you're running support forward slash without a problem.
> Checking this in each case isn't worth the trouble when it's so simple
> to call normpath(). Doing a quick test isn't sufficient. Programs that
> fail in this case do so in different ways and not always consistently.
> Using forward slash may work in some cases (e.g. passing a explicit
> file vs passing a wildcard pattern), which may give someone a false
> perception that the program intentionally supports forward slash as a
> path separator, when it's only working by accident of implementation.

Cool. Okay, so for people who don't know about pywin32, it's important
to properly wrap up arguments to some of the external programs you
might need. I'll grant you that. Internally, use forward slashes; then
when you need to shell out to something, you process the arguments
into something that will be accepted.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[eryk sun]

On Sun, Jul 1, 2018 at 2:28 AM, Chris Angelico  wrote:
> On Sun, Jul 1, 2018 at 12:22 PM, eryk sun  wrote:
>>> So what if, internally, that's done by converting them to backslashes?
>>> No Python program needs to care. In fact, there are other conversions,
>>> too - the underlying file system is most likely using UTF-16 paths,
>>> but your Python program needn't use UTF-16. No, it simply uses a plain
>>> old string literal - a Unicode string. The OP need not be concerned
>>> about any of these fiddlinesses, unless in some way they actually
>>> affect Python.
>>
>> A few minor changes in in wording are both technically correct and
>> don't make the text any more difficult to understand. You're the one
>> blowing this up beyond proportion and making a mountain out of a small
>> technical-correction mole hill. Please just let it pass.
>
> If you can show me a way in which a Python program actually needs to
> care about any of these distinctions when dealing with files on
> Windows, then I'll admit that the technical correction was needed. You
> won't be shelling out to the "dir" command, for instance.

Python has no built-in support for Windows file security, so a Python
program may decided to use icacls.exe, takeown.exe, or xcopy.exe (copy
owner/ACL). It's best in general to normalize paths when integrating
an external program -- unless you've already checked that the
program(s) you're running support forward slash without a problem.
Checking this in each case isn't worth the trouble when it's so simple
to call normpath(). Doing a quick test isn't sufficient. Programs that
fail in this case do so in different ways and not always consistently.
Using forward slash may work in some cases (e.g. passing a explicit
file vs passing a wildcard pattern), which may give someone a false
perception that the program intentionally supports forward slash as a
path separator, when it's only working by accident of implementation.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Chris Angelico]

On Sun, Jul 1, 2018 at 12:22 PM, eryk sun  wrote:
>> So what if, internally, that's done by converting them to backslashes?
>> No Python program needs to care. In fact, there are other conversions,
>> too - the underlying file system is most likely using UTF-16 paths,
>> but your Python program needn't use UTF-16. No, it simply uses a plain
>> old string literal - a Unicode string. The OP need not be concerned
>> about any of these fiddlinesses, unless in some way they actually
>> affect Python.
>
> A few minor changes in in wording are both technically correct and
> don't make the text any more difficult to understand. You're the one
> blowing this up beyond proportion and making a mountain out of a small
> technical-correction mole hill. Please just let it pass.

If you can show me a way in which a Python program actually needs to
care about any of these distinctions when dealing with files on
Windows, then I'll admit that the technical correction was needed. You
won't be shelling out to the "dir" command, for instance.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[eryk sun]

On Sun, Jul 1, 2018 at 1:58 AM, Chris Angelico  wrote:
> On Sun, Jul 1, 2018 at 10:20 AM, eryk sun  wrote:
>> On Sat, Jun 30, 2018 at 11:42 PM, Chris Angelico  wrote:
>>> "Legacy" implies that it's the old standard that is now deprecated,
>>
>> I did not mean to imply that DOS paths are deprecated. That's not what
>> legacy means to me.
>
> Unless you are Humpty Dumpty, you can't just say "that's not what it
> means to me". Sorry. People WILL see "legacy" as an implication that
> "only the old stuff goes that way". You are responding to a thread in
> which someone used a perfectly ordinary file path, and you said that
> only legacy paths can be done with slashes. At best, you are
> technically correct and functionally deceptive.

That's fine, and I understand how I could have been misconstrued. I
clarified what I meant. I did not intend to be deceptive.

>>> Registry paths aren't file paths,
>>
>> You said to use forward slash for "all paths".
>
> *facepalm* Okay. I said "all paths", but I did not refer to XPath
> (although that also uses the slash character), nor registry paths, nor
> the footpath that runs parallel to the road outside my house. You're
> absolutely right, there are no forward slashes in the path that leads
> to my door. My bad. I'm sorry that the word "all" has to be
> interpreted within the context of the thread that it was used in.
> Whoops.

I also clarified why to me your "all paths" statement needs to be
qualified to the domain. I use the native API a lot, so for me
registry and file paths are just paths. It's only the Windows API that
separates the two and only the Windows API that allows forward slash
as a path separator in file paths.

>>> command line arguments are interpreted by the target program (and in many
>>> MANY cases, forward slashes are absolutely fine),
>>
>> You can't generalize this. In many cases it's not ok. Most
>> command-line utilities that use ulib make a simply assumption when
>> parsing the command line that forward slash is used for options
>> (switches). It's best to normalize paths in a command line via
>> os.path.normpath.
>
> Ahh but I *can* generalize it. In many cases, forward slashes ARE
> fine, and I can prove it by listing off large numbers of programs that
> are perfectly happy to receive file paths with slashes in them.
> Surprise surprise, most of those programs are going to be taking those
> paths and handing them straight to the OS. In other words: THE OS
> SUPPORTS SLASHES. Does this surprise anyone?

There are many cases for common command-line utilities that do not
support paths with forward slash -- even if you disambiguate by
putting the path in double quotes. In the following examples, I have a
file named "C:\Temp\test.txt" and an existing directory named
"C:\Temp\dest".

Internal CMD commands:

C:\>dir /b "C:/Temp/test.txt"
File Not Found

C:\>type "C:/Temp/test.txt"
The system cannot find the file specified.

C:\>copy "C:/Temp/test.txt" "C:/Temp/dest"
The system cannot find the file specified.
0 file(s) copied.

C:\>del "C:/Temp/test.txt"
The system cannot find the path specified.

External utility commands:

C:\>find "spam" "C:/Temp/*.txt"
File not found - C:TEST.TXT

C:\>findstr "spam" "C:/Temp/*.txt"
FINDSTR: Cannot open C:test.txt

C:\>fc "C:/Temp/test.txt" "C:/Temp/*.txt"
FC: cannot open C:/TEMP/TEST.TXT - No such file or folder

C:\>forfiles /p "C:/Temp" /m *.txt
ERROR: The directory name is invalid.

C:\>where /r "C:/Temp" test.txt
ERROR: Invalid directory specified.

C:\>icacls "C:/Temp/*.txt"
C:test.txt: The system cannot find the file specified.
Successfully processed 0 files; Failed processing 1 files

C:\>takeown /f "C:/Temp/test.txt"
ERROR: File or Directory not found.

C:\>xcopy.exe "C:/Temp/test.txt" "C:/Temp/dest"
0 File(s) copied

In the following case I create "C:\Temp\dest\test.txt" to be replaced
using replace.exe, which silently fails to replace the file:

C:\>replace "C:/Temp/test.txt" "C:/Temp/dest"

C:\>fc C:\Temp\test.txt C:\Temp\dest\test.txt
Comparing files C:\TEMP\test.txt and C:\TEMP\DEST\TEST.TXT
* C:\TEMP\test.txt
spam
* C:\TEMP\DEST\TEST.TXT
eggs
*

> So what if, internally, that's done by converting them to backslashes?
> No Python program needs to care. In fact, there are other conversions,
> too - the underlying file system is most likely using UTF-16 paths,
> but your Python program needn't use UTF-16. No, it simply uses a plain
> old string literal - a Unicode string. The OP need not be concerned
> about any of these fiddlinesses, unless in some way they actually
> affect Python.

A few minor changes in in wording are both technically correct and
don't make the text any more difficult to understand. You're the one
blowing this up beyond proportion and making a mountain out of a small
technical-correction mole hill. Please just let it pass.
-- 

Re: error in os.chdir

[Chris Angelico]

On Sun, Jul 1, 2018 at 10:20 AM, eryk sun  wrote:
> On Sat, Jun 30, 2018 at 11:42 PM, Chris Angelico  wrote:
>> "Legacy" implies that it's the old standard that is now deprecated,
>
> I did not mean to imply that DOS paths are deprecated. That's not what
> legacy means to me.

Unless you are Humpty Dumpty, you can't just say "that's not what it
means to me". Sorry. People WILL see "legacy" as an implication that
"only the old stuff goes that way". You are responding to a thread in
which someone used a perfectly ordinary file path, and you said that
only legacy paths can be done with slashes. At best, you are
technically correct and functionally deceptive.

>> Registry paths aren't file paths,
>
> You said to use forward slash for "all paths".

*facepalm* Okay. I said "all paths", but I did not refer to XPath
(although that also uses the slash character), nor registry paths, nor
the footpath that runs parallel to the road outside my house. You're
absolutely right, there are no forward slashes in the path that leads
to my door. My bad. I'm sorry that the word "all" has to be
interpreted within the context of the thread that it was used in.
Whoops.

>> command line arguments are interpreted by the target program (and in many
>> MANY cases, forward slashes are absolutely fine),
>
> You can't generalize this. In many cases it's not ok. Most
> command-line utilities that use ulib make a simply assumption when
> parsing the command line that forward slash is used for options
> (switches). It's best to normalize paths in a command line via
> os.path.normpath.

Ahh but I *can* generalize it. In many cases, forward slashes ARE
fine, and I can prove it by listing off large numbers of programs that
are perfectly happy to receive file paths with slashes in them.
Surprise surprise, most of those programs are going to be taking those
paths and handing them straight to the OS. In other words: THE OS
SUPPORTS SLASHES. Does this surprise anyone?

So what if, internally, that's done by converting them to backslashes?
No Python program needs to care. In fact, there are other conversions,
too - the underlying file system is most likely using UTF-16 paths,
but your Python program needn't use UTF-16. No, it simply uses a plain
old string literal - a Unicode string. The OP need not be concerned
about any of these fiddlinesses, unless in some way they actually
affect Python.

Congratulations on being technically correct (maybe) and utterly unhelpful.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Steven D'Aprano]

On Sat, 30 Jun 2018 23:36:40 +, eryk sun wrote:

> Only use forward slashes for legacy DOS paths passed to Windows API
> functions. Do not use forward slashes for paths in command line
> arguments, \\?\ prefixed paths, or registry paths.

I don't see why this is relevant, or at least not the "command line 
arguments" and "registry paths" parts.

We're talking about using file names in Python, not the Windows command 
line (Power Shell?), and not registry paths.

I guess that if the user is using a path beginning with \\?\ they may or 
may not need to use backslashes, but I have no way of testing it, and I 
would expect that Python will correctly replace //?/ with backslashes the 
same as it does for any other file system path.

I know that Chris said "for all paths" but in context that's to be 
understood as *file system paths*. We wouldn't interpret him as meaning 
this:

import package/subpackage/module

instead of using dots, so I think registry paths is irrelevant.

(Besides, Python doesn't have an API for interacting directly with the 
registry.)




-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[eryk sun]

On Sat, Jun 30, 2018 at 11:42 PM, Chris Angelico  wrote:
> On Sun, Jul 1, 2018 at 9:36 AM, eryk sun  wrote:
>> On Sat, Jun 30, 2018 at 11:21 AM, Chris Angelico  wrote:
>>> On Sat, Jun 30, 2018 at 9:05 PM, Sharan Basappa
>>>  wrote:

 0
 down vote
 favorite

 I need to change directory to my local working directory in windows and 
 then open a file for processing.
 Its just a 3 lines code, as below:
 import csv
 import os
 os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
 The error is as follows:
 WindowsError: [Error 123] The filename, directory name, or volume label 
 syntax is incorrect: 'D:\Projects\Initiatives\machine 
 learning\programs\x07ssertion'
 Notice x07 character that has replaced character x07.
 I have a similar code but that goes through fine:
 import csv
 import os
 os.chdir('D:\Projects\Initiatives\machine learning\programs')
>>>
>>> Use forward slashes instead of backslashes for all paths.
>>>
>>> os.chdir('D:/Projects/Initiatives/machine learning/programs')
>>
>> Only use forward slashes for legacy DOS paths passed to Windows API
>> functions. Do not use forward slashes for paths in command line
>> arguments, \\?\ prefixed paths, or registry paths.
>
> "Legacy" implies that it's the old standard that is now deprecated,

I did not mean to imply that DOS paths are deprecated. That's not what
legacy means to me. I only meant that they're inherited from an old
system. The native API does not use forward slash as a path separator.
It's just a name character in NT (except in most file-system
implementations, which reserve forward slash as an invalid filename
character to avoid confusion). This doesn't affect device names. You
can call DefineDosDevice or native NtCreateSymbolicLinkObject to
create a device name with a slash in it.

The Windows file API supports forward slash as a path separator, which
it implements by rewriting the path via ntdll library functions such
as RtlDosPathNameToNtPathName_U_WithStatus. Paths prefixed with \\?\
bypass this normalization step, so they cannot use forward slash as
the path separator.

> Registry paths aren't file paths,

You said to use forward slash for "all paths".

Also, from the POV of the NT API, registry paths and file paths are
all just paths, such as "\Registry\Machine\Software" and
"\Device\HarddiskVolume2\Program Files". The executive's Object
Manager parses the path up to an object with a ParseProcedure (e.g.
"Registry" or "HarddiskVolume2"), which in turn parses the rest of the
path. Registry paths in the Windows API are closer to native NT paths,
since they don't support forward slash as a path separator and
implement relative paths NT style, i.e. relative to a Key handle. The
Windows API doesn't expose handle-relative paths for file paths, but
it uses this feature implicitly by keeping a handle open for the
process working directory.

> command line arguments are interpreted by the target program (and in many
> MANY cases, forward slashes are absolutely fine),

You can't generalize this. In many cases it's not ok. Most
command-line utilities that use ulib make a simply assumption when
parsing the command line that forward slash is used for options
(switches). It's best to normalize paths in a command line via
os.path.normpath.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Terry Reedy]

On 6/30/2018 7:36 PM, eryk sun wrote:

On Sat, Jun 30, 2018 at 11:21 AM, Chris Angelico  wrote:

On Sat, Jun 30, 2018 at 9:05 PM, Sharan Basappa
 wrote:


0
down vote
favorite

I need to change directory to my local working directory in windows and then 
open a file for processing.
Its just a 3 lines code, as below:
import csv
import os
os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
The error is as follows:
WindowsError: [Error 123] The filename, directory name, or volume label syntax 
is incorrect: 'D:\Projects\Initiatives\machine learning\programs\x07ssertion'
Notice x07 character that has replaced character x07.
I have a similar code but that goes through fine:
import csv
import os
os.chdir('D:\Projects\Initiatives\machine learning\programs')


Use forward slashes instead of backslashes for all paths.

os.chdir('D:/Projects/Initiatives/machine learning/programs')


Only use forward slashes for legacy DOS paths passed to Windows API
functions. Do not use forward slashes for paths in command line
arguments, \\?\ prefixed paths, or registry paths.


To the best of my memeory, forward slashes work in arguments.
They do not work in paths to an executable.

F:\>cd dev/3x works

f:\dev\3x> ../pull  prints
'..' is not recognized as an internal or external command,
operable program or batch file.
because /pull is seen as an option switch.  ..\pull does (because 
..\pull.bat exists).





--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Chris Angelico]

On Sun, Jul 1, 2018 at 9:36 AM, eryk sun  wrote:
> On Sat, Jun 30, 2018 at 11:21 AM, Chris Angelico  wrote:
>> On Sat, Jun 30, 2018 at 9:05 PM, Sharan Basappa
>>  wrote:
>>>
>>> 0
>>> down vote
>>> favorite
>>>
>>> I need to change directory to my local working directory in windows and 
>>> then open a file for processing.
>>> Its just a 3 lines code, as below:
>>> import csv
>>> import os
>>> os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
>>> The error is as follows:
>>> WindowsError: [Error 123] The filename, directory name, or volume label 
>>> syntax is incorrect: 'D:\Projects\Initiatives\machine 
>>> learning\programs\x07ssertion'
>>> Notice x07 character that has replaced character x07.
>>> I have a similar code but that goes through fine:
>>> import csv
>>> import os
>>> os.chdir('D:\Projects\Initiatives\machine learning\programs')
>>
>> Use forward slashes instead of backslashes for all paths.
>>
>> os.chdir('D:/Projects/Initiatives/machine learning/programs')
>
> Only use forward slashes for legacy DOS paths passed to Windows API
> functions. Do not use forward slashes for paths in command line
> arguments, \\?\ prefixed paths, or registry paths.

"Legacy" implies that it's the old standard that is now deprecated,
but the truth is that MOST path names are going to work fine with
forward slashes. Registry paths aren't file paths, command line
arguments are interpreted by the target program (and in many MANY
cases, forward slashes are absolutely fine), and so it's just the \\?\
paths that will need to be special. So, let me rephrase what you said:

Forward slashes work for all Windows file paths, except those that are
prefixed with \\?\.

Is that correct?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[eryk sun]

On Sat, Jun 30, 2018 at 11:21 AM, Chris Angelico  wrote:
> On Sat, Jun 30, 2018 at 9:05 PM, Sharan Basappa
>  wrote:
>>
>> 0
>> down vote
>> favorite
>>
>> I need to change directory to my local working directory in windows and then 
>> open a file for processing.
>> Its just a 3 lines code, as below:
>> import csv
>> import os
>> os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
>> The error is as follows:
>> WindowsError: [Error 123] The filename, directory name, or volume label 
>> syntax is incorrect: 'D:\Projects\Initiatives\machine 
>> learning\programs\x07ssertion'
>> Notice x07 character that has replaced character x07.
>> I have a similar code but that goes through fine:
>> import csv
>> import os
>> os.chdir('D:\Projects\Initiatives\machine learning\programs')
>
> Use forward slashes instead of backslashes for all paths.
>
> os.chdir('D:/Projects/Initiatives/machine learning/programs')

Only use forward slashes for legacy DOS paths passed to Windows API
functions. Do not use forward slashes for paths in command line
arguments, \\?\ prefixed paths, or registry paths.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Jim Lee]

On 06/30/18 07:34, Sharan Basappa wrote:


sorry. I mean why my code worked in one case but did not in the other one.

This worked - os.chdir('D:\Projects\Initiatives\machine learning\programs')

This did not work - os.chdir('D:\Projects\Initiatives\machine 
learning\programs\assertion')

only difference is, I added an additional directory in the problematic case.

In Python (and many other languages), certain non-printable characters 
are represented in strings by "escape sequences".  Those escape 
sequences begin with a backslash "\".  In your particular case, there 
are five (possible) escape sequences: "\P", "\I", "\m", "\p", and "\a".  
All but the last one have no meaning to Python, so they are treated 
literally.  The last one, however, translates to "BEL", or 0x07, which 
is a non-printable code that was once sent to teletypes to ring the 
bell, alerting the operator to an incoming message, or to signal the 
completion of a long-running job.


There are three ways to fix your code.  One is to replace the 
backslashes with forward slashes.  Python will translate them internally 
to the correct path separators for your operating system.


Another way is to replace the backslashes with os.sep, which is a 
portable representation of the native path separator.


A third way is to escape all literal backslashes by doubling them - "\\" 
instead of "\".  However, this will make your code Windows-only (but it 
already is, so that may not matter to you).


-Jim


--
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Karsten Hilbert]

On Sat, Jun 30, 2018 at 07:34:46AM -0700, Sharan Basappa wrote:

> sorry. I mean why my code worked in one case but did not in the other one.
> 
> This worked - os.chdir('D:\Projects\Initiatives\machine learning\programs') 
> 
> This did not work - os.chdir('D:\Projects\Initiatives\machine 
> learning\programs\assertion')
> 
> only difference is, I added an additional directory in the problematic case.

The answer lies in the (inadvertant) use of escaping.

Which is the very reason _why_ forward slashes were favoured
over backward ones.

Karsten
-- 
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Sharan Basappa]

On Saturday, 30 June 2018 18:55:53 UTC+5:30, Karsten Hilbert  wrote:
> On Sat, Jun 30, 2018 at 05:46:59AM -0700, Sharan Basappa wrote:
> 
> > > >> The quick fix:
> > > >> 
> > > >> put an r in front of the directory string: r'...'
> > > 
> > > Please don't do that. It's the wrong solution -- all you are doing is 
> > > postponing failure. It will *seem* to work, until one day you will write 
> > > something like this:
> > > 
> > > directory = r'D:\directory\'
> > > 
> > > and you will get a mysterious failure. Chris gave you the right solution: 
> > > use forward slashes instead of backslashes for all paths.
> > 
> > alright. I will do that but still I don't have an answer why I got the 
> > error in the first place.
> 
> For that you'll have to read up on strings and escaping.
> 
>   https://docs.python.org/2/tutorial/introduction.html#strings
> 
> Karsten
> -- 
> GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B

sorry. I mean why my code worked in one case but did not in the other one.

This worked - os.chdir('D:\Projects\Initiatives\machine learning\programs') 

This did not work - os.chdir('D:\Projects\Initiatives\machine 
learning\programs\assertion')

only difference is, I added an additional directory in the problematic case.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Karsten Hilbert]

On Sat, Jun 30, 2018 at 05:46:59AM -0700, Sharan Basappa wrote:

> > >> The quick fix:
> > >> 
> > >> put an r in front of the directory string: r'...'
> > 
> > Please don't do that. It's the wrong solution -- all you are doing is 
> > postponing failure. It will *seem* to work, until one day you will write 
> > something like this:
> > 
> > directory = r'D:\directory\'
> > 
> > and you will get a mysterious failure. Chris gave you the right solution: 
> > use forward slashes instead of backslashes for all paths.
> 
> alright. I will do that but still I don't have an answer why I got the error 
> in the first place.

For that you'll have to read up on strings and escaping.

https://docs.python.org/2/tutorial/introduction.html#strings

Karsten
-- 
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Sharan Basappa]

On Saturday, 30 June 2018 18:05:02 UTC+5:30, Steven D'Aprano  wrote:
> On Sat, 30 Jun 2018 04:47:44 -0700, Sharan Basappa wrote:
> 
> >> The quick fix:
> >> 
> >> put an r in front of the directory string: r'...'
> >
> > thanks. That works
> 
> Please don't do that. It's the wrong solution -- all you are doing is 
> postponing failure. It will *seem* to work, until one day you will write 
> something like this:
> 
> directory = r'D:\directory\'
> 
> and you will get a mysterious failure. Chris gave you the right solution: 
> use forward slashes instead of backslashes for all paths.
> 
> os.chdir('D:/Projects/Initiatives/machine learning/programs')
> 
> 
> 
> 
> -- 
> Steven D'Aprano
> "Ever since I learned about confirmation bias, I've been seeing
> it everywhere." -- Jon Ronson

alright. I will do that but still I don't have an answer why I got the error in 
the first place.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Steven D'Aprano]

On Sat, 30 Jun 2018 04:47:44 -0700, Sharan Basappa wrote:

>> The quick fix:
>> 
>> put an r in front of the directory string: r'...'
>
> thanks. That works

Please don't do that. It's the wrong solution -- all you are doing is 
postponing failure. It will *seem* to work, until one day you will write 
something like this:

directory = r'D:\directory\'

and you will get a mysterious failure. Chris gave you the right solution: 
use forward slashes instead of backslashes for all paths.

os.chdir('D:/Projects/Initiatives/machine learning/programs')




-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Abdur-Rahmaan Janhangeer]

for user input, a replace might be the solution, using / in internal code

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ

Use forward slashes instead of backslashes for all paths.
>
> os.chdir('D:/Projects/Initiatives/machine learning/programs')
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Sharan Basappa]

On Saturday, 30 June 2018 16:51:53 UTC+5:30, Karsten Hilbert  wrote:
> On Sat, Jun 30, 2018 at 04:05:22AM -0700, Sharan Basappa wrote:
> 
> > I need to change directory to my local working directory in windows and 
> > then open a file for processing.
> > Its just a 3 lines code, as below:
> > import csv
> > import os
> > os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
> > The error is as follows:
> > WindowsError: [Error 123] The filename, directory name, or volume label 
> > syntax is incorrect: 'D:\Projects\Initiatives\machine 
> > learning\programs\x07ssertion'
> > Notice x07 character that has replaced character x07.
> > I have a similar code but that goes through fine:
> > import csv
> > import os
> > os.chdir('D:\Projects\Initiatives\machine learning\programs')
> > 
> > with open('example.csv') as csvfile:
> > readCSV = csv.reader(csvfile, delimiter=',')
> > The only difference is directory assertion in the problematic code.
> > I have tried single quoting, double quoting etc. for the chdir directive 
> > but nothing helps. I have also tried escaping as \assertion but that is not 
> > the issue
> 
> The quick fix:
> 
> put an r in front of the directory string: r'...'
> 
> Karsten
> -- 
> GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B

thanks. That works
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Chris Angelico]

On Sat, Jun 30, 2018 at 9:05 PM, Sharan Basappa
 wrote:
>
> 0
> down vote
> favorite
>
> I need to change directory to my local working directory in windows and then 
> open a file for processing.
> Its just a 3 lines code, as below:
> import csv
> import os
> os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
> The error is as follows:
> WindowsError: [Error 123] The filename, directory name, or volume label 
> syntax is incorrect: 'D:\Projects\Initiatives\machine 
> learning\programs\x07ssertion'
> Notice x07 character that has replaced character x07.
> I have a similar code but that goes through fine:
> import csv
> import os
> os.chdir('D:\Projects\Initiatives\machine learning\programs')

Use forward slashes instead of backslashes for all paths.

os.chdir('D:/Projects/Initiatives/machine learning/programs')

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error in os.chdir

[Karsten Hilbert]

On Sat, Jun 30, 2018 at 04:05:22AM -0700, Sharan Basappa wrote:

> I need to change directory to my local working directory in windows and then 
> open a file for processing.
> Its just a 3 lines code, as below:
> import csv
> import os
> os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
> The error is as follows:
> WindowsError: [Error 123] The filename, directory name, or volume label 
> syntax is incorrect: 'D:\Projects\Initiatives\machine 
> learning\programs\x07ssertion'
> Notice x07 character that has replaced character x07.
> I have a similar code but that goes through fine:
> import csv
> import os
> os.chdir('D:\Projects\Initiatives\machine learning\programs')
> 
> with open('example.csv') as csvfile:
> readCSV = csv.reader(csvfile, delimiter=',')
> The only difference is directory assertion in the problematic code.
> I have tried single quoting, double quoting etc. for the chdir directive but 
> nothing helps. I have also tried escaping as \assertion but that is not the 
> issue

The quick fix:

put an r in front of the directory string: r'...'

Karsten
-- 
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

error in os.chdir

[Sharan Basappa]

0
down vote
favorite

I need to change directory to my local working directory in windows and then 
open a file for processing.
Its just a 3 lines code, as below:
import csv
import os
os.chdir('D:\Projects\Initiatives\machine learning\programs\assertion')
The error is as follows:
WindowsError: [Error 123] The filename, directory name, or volume label syntax 
is incorrect: 'D:\Projects\Initiatives\machine learning\programs\x07ssertion'
Notice x07 character that has replaced character x07.
I have a similar code but that goes through fine:
import csv
import os
os.chdir('D:\Projects\Initiatives\machine learning\programs')

with open('example.csv') as csvfile:
readCSV = csv.reader(csvfile, delimiter=',')
The only difference is directory assertion in the problematic code.
I have tried single quoting, double quoting etc. for the chdir directive but 
nothing helps. I have also tried escaping as \assertion but that is not the 
issue
-- 
https://mail.python.org/mailman/listinfo/python-list