Re: [Django] #10254: The use of 'FileField' Upload file missing Chinese characters (non-ascii)

[Django]

#10254: The use of 'FileField' Upload file missing Chinese characters 
(non-ascii)
---+
  Reporter:  isoem...@163. com | Owner:  kmtracey   
 
Status:  assigned  | Milestone:  1.1
 
 Component:  File uploads/storage  |   Version:  1.0
 
Resolution:|  Keywords:  FileField  
non-ascii
 Stage:  Accepted  | Has_patch:  0  
 
Needs_docs:  0 |   Needs_tests:  0  
 
Needs_better_patch:  0 |  
---+
Changes (by kmtracey):

  * owner:  nobody => kmtracey
  * status:  new => assigned

Comment:

 OK, I will make the regex change.  Looks like there is also some part of a
 test in file_uploads that is attempting to verify that upload with unicode
 chars works, but it only tests the value seen in the POST data, it doesn't
 verify the actual file field after save still has the unicode characters
 in its name.  I'll fix up that test to follow through with the save and
 thus verify the whole process is working.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #10254: The use of 'FileField' Upload file missing Chinese characters (non-ascii)

[Django]

#10254: The use of 'FileField' Upload file missing Chinese characters 
(non-ascii)
---+
  Reporter:  isoem...@163. com | Owner:  nobody 
 
Status:  new   | Milestone:  1.1
 
 Component:  File uploads/storage  |   Version:  1.0
 
Resolution:|  Keywords:  FileField  
non-ascii
 Stage:  Accepted  | Has_patch:  0  
 
Needs_docs:  0 |   Needs_tests:  0  
 
Needs_better_patch:  0 |  
---+
Comment (by mtredinnick):

 That patch looks fine to me, Karen. I (and apparently everybody else with
 more motivation to find this, since they would have been using non-ASCII
 filenames) overlooked this in the unicode changes. That's why it wasn't
 changed then.

 I can't think of any security problems it will introduce, as directory
 traversal still isn't possible with that change (for any of the `os.sep`
 values on Mac (even older ones), Windows, Linux, )

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #10254: The use of 'FileField' Upload file missing Chinese characters (non-ascii)

[Django]

#10254: The use of 'FileField' Upload file missing Chinese characters 
(non-ascii)
---+
  Reporter:  isoem...@163. com | Owner:  nobody 
 
Status:  new   | Milestone:  1.1
 
 Component:  File uploads/storage  |   Version:  1.0
 
Resolution:|  Keywords:  FileField  
non-ascii
 Stage:  Accepted  | Has_patch:  0  
 
Needs_docs:  0 |   Needs_tests:  0  
 
Needs_better_patch:  0 |  
---+
Comment (by kmtracey):

 Also reported here: http://groups.google.com/group/django-
 users/browse_thread/thread/337848b9d16f79de#

 so I added the patch as an attachment so as to have something to point
 this user to for a possible fix.  (Note the patch as-is isn't something
 we'd want to check in, as it doesn't update the comment to match the
 changed code.)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #10254: The use of 'FileField' Upload file missing Chinese characters (non-ascii)

[Django]

#10254: The use of 'FileField' Upload file missing Chinese characters 
(non-ascii)
---+
  Reporter:  isoem...@163. com | Owner:  nobody 
 
Status:  new   | Milestone:  1.1
 
 Component:  File uploads/storage  |   Version:  1.0
 
Resolution:|  Keywords:  FileField  
non-ascii
 Stage:  Accepted  | Has_patch:  0  
 
Needs_docs:  0 |   Needs_tests:  0  
 
Needs_better_patch:  0 |  
---+
Changes (by kmtracey):

  * milestone:  => 1.1

Comment:

 Can anyone with more history than I have comment on the proposed change to
 get_valid_filename?  The very restrictive version that strips out all but
 A-Z, a-z, dot, dash, and underscore has been there since day 1 as far as
 the public Django repository goes, so I can't get any sense for the reason
 why it was made so restrictive.

 I don't see the harm in allowing unicode alphanumerics, and I get the
 impression this was thought to have already been fixed -- see #6009, where
 tests were added that attempt to verify files with non-ASCII chars can be
 uploaded...only the tests don't go so far as to actually save the file to
 the file system, which is when the non-ASCII chars are being stripped.
 I've tested uploading files with non-ASCII chars on Ubuntu and Windows,
 and they both work fine.  The only potential problem I see is that an
 English Windows box doesn't necessarily display really exotic (e.g.
 Japanese and Chinese) characters in file names correctly in either a
 command prompt or the file explorer, so it allows them but doesn't really
 fully support them.  I'd expect there are versions of Windows that handle
 this better but all I have is plain English Windows...and even there all
 you get are files that may be a little difficult to deal with since you
 cannot type their names, but you can still get auto-complete to do it for
 you, or select them via a GUI, etc.

 At any rate it seems we should be able to resolve this one way or the
 other for 1.1.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #10254: The use of 'FileField' Upload file missing Chinese characters (non-ascii)

[Django]

#10254: The use of 'FileField' Upload file missing Chinese characters 
(non-ascii)
---+
  Reporter:  isoem...@163. com | Owner:  nobody 
 
Status:  new   | Milestone: 
 
 Component:  File uploads/storage  |   Version:  1.0
 
Resolution:|  Keywords:  FileField  
non-ascii
 Stage:  Accepted  | Has_patch:  0  
 
Needs_docs:  0 |   Needs_tests:  0  
 
Needs_better_patch:  0 |  
---+
Comment (by gulliver):

 I changed get_valid_filename(s) of utils/text.py,

 the last line of get_valid_filename(s) is became
 return re.sub(r'(?u)[^-\w_.]', '', s)
 So it worked.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #10254: The use of 'FileField' Upload file missing Chinese characters (non-ascii)

[Django]

#10254: The use of 'FileField' Upload file missing Chinese characters 
(non-ascii)
---+
  Reporter:  isoem...@163. com | Owner:  nobody 
 
Status:  new   | Milestone: 
 
 Component:  File uploads/storage  |   Version:  1.0
 
Resolution:|  Keywords:  FileField  
non-ascii
 Stage:  Accepted  | Has_patch:  0  
 
Needs_docs:  0 |   Needs_tests:  0  
 
Needs_better_patch:  0 |  
---+
Changes (by kmtracey):

  * needs_better_patch:  => 0
  * stage:  Unreviewed => Accepted
  * needs_tests:  => 0
  * needs_docs:  => 0

Comment:

 This has been reported before, see #3119 and #6009, and apparently thought
 to have been fixed, but it isn't fixed.  (Also, near as I can tell this is
 not a recent regression, this part of the issues identified in those
 earlier tickets just was never addressed, so far as I can see).  Earlier
 there were two problems: an exception on upload of a file with non-ASCII
 chars in the name, and stripping of non-ASCII chars from the filename.
 The 1st seems to have been fixed, the 2nd is still in place.

 django/utils/text.py get_valid_filename still strips non-ASCII chars from
 the filename, and this routine is called by core/files/storage.py to
 generate a "valid" file name when a !FileField is saved.  Though r7987
 added a test to ensure that uploading files with non-ASCII names "works",
 the part of the test that checks that the name is preserved simply checks
 the file name in cleaned_data, and this stripping of non-ASCII chars is
 done long after form validation.  So while the file name remains "correct"
 in what can be seen in the form's cleaned_data, that name is stripped of
 its non-ASCII chars (and perhaps spaces are turned into underscores and
 additional underscores are appended to the name to avoid overwriting an
 existing file of the same name).  So the name you can see in cleaned_data
 isn't necessarily the name the file will be stored under.

 I don't know the right way to fix this, but the existing
 get_valid_filename in django/utils/text.py certainly seems over-
 restrictive in terms of what is an allowed character in a file name.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

[Django] #10254: The use of 'FileField' Upload file missing Chinese characters (non-ascii)

[Django]

#10254: The use of 'FileField' Upload file missing Chinese characters 
(non-ascii)
--+-
 Reporter:  isoem...@163. com |   Owner:  nobody
   Status:  new   |   Milestone:
Component:  File uploads/storage  | Version:  1.0   
 Keywords:  FileField  non-ascii  |   Stage:  Unreviewed
Has_patch:  0 |  
--+-
 file : /home/jane/铅笔画.xcf[[BR]]

 use of 'FileField' Upload file,[[BR]]

 get "Image/.xcf" lost (non-ascii)characters part

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---