Problem reading with bz2.BZ2File(). Bug?
When comparing two files which should be equal the last line is different: The first file is a bzip2 compressed file and is read with bz2.BZ2File() The second file is the same file uncompressed and read with open() The first file named file.txt.bz2 is uncompressed with: $ bunzip2 -k file.txt.bz2 To compare I use this script: ### import bz2 f1 = bz2.BZ2File(r'file.txt.bz2', 'r') f2 = open(r'file.txt', 'r') lines = 0 while True: line1 = f1.readline() line2 = f2.readline() if line1 == '': break lines += 1 if line1 != line2: print 'line number:', lines print repr(line1) print repr(line2) f1.close() f2.close() ## The offending file is 5.5 MB. Sorry, i could not reproduce this problem with a smaller file. http://fahstats.com/img/file.txt.bz2 Regards, Clodoaldo Pinto Neto -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem reading with bz2.BZ2File(). Bug?
Fredrik Lundh wrote: Clodoaldo Pinto Neto wrote: The offending file is 5.5 MB. Sorry, i could not reproduce this problem with a smaller file. but surely you can post the repr() of the last two lines? This is the output: $ python bzp.py line number: 588317 '\x07' '' Clodoaldo -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem reading with bz2.BZ2File(). Bug?
Leo Kislov wrote: Confirmed on windows with 2.4 and 2.5: C:\p\Python24\python.exe bzp.py line number: 588317 '\x1e' '' C:\p\Python25\python.exe bzp.py line number: 588317 '\x1e' '' Looks like one byte of garbage is appended at the end of file. Please file a bug report. Bug number 1597011 Clodoaldo -- http://mail.python.org/mailman/listinfo/python-list
Re: Output from subprocess.Popen()
Fredrik Lundh wrote: Clodoaldo Pinto Neto wrote: But I still don't understand what is happening. The manual says that when shell=True the executable argument specifies which shell to use: no, it says that when shell=True, it runs the command through the default shell. that is, it hands it over to the shell for execution, pretty much as if you'd typed it in yourself. If shell=True, the executable argument specifies which shell to use. Taken from 6.8.1 v2.4: The executable argument specifies the program to execute. It is very seldom needed: Usually, the program to execute is defined by the args argument. If shell=True, the executable argument specifies which shell to use. On Unix, the default shell is /bin/sh. On Windows, the default shell is specified by the COMSPEC environment variable. -- http://mail.python.org/mailman/listinfo/python-list
Re: Output from subprocess.Popen()
Now we have 3 different outputs from 3 people to the command: f = subprocess.Popen(set | grep IFS, shell=True, stdout=subprocess.PIPE) f.stdout.readlines() From me on FC5: [BASH_EXECUTION_STRING='set | grep IFS'\n, IFS=' \t\n] From Fredrik Lundh on unknown OS: [IFS=$' \\t\\n'\n] From Sion Arrowsmith on Debian 1:3.3.5-12: [IFS=' \t\n] I found this problem while developing a CGI shell: http://code.google.com/p/cgpy-shell/ Although the cgi-shell is working quite well, this issue is a show stopper for that project. I need a stable/reliable output from Popen(). It looks like each bash or sh version outputs things differently, with different escaping. It is not a problem if there is only one or more than one line in the returned list as i will always get the last one. The problem is the escaping. -- http://mail.python.org/mailman/listinfo/python-list
Re: Output from subprocess.Popen()
Fredrik Lundh wrote:
[EMAIL PROTECTED] wrote:
I can't see any obvious way to ask subprocess to use a shell other than
the default.
-c ?
f = Popen([/bin/bash, -c, set|grep IFS], stdout=PIPE)
f.stdout.read()
IFS=$' \\t\\n'\n
f = Popen([/bin/sh, -c, set|grep IFS], stdout=PIPE)
f.stdout.read()
IFS=' \t\n
It solves my problem:
f = sub.Popen(['/bin/sh', '-c', 'set|grep IFS'], stdout=sub.PIPE)
f.stdout.read()
BASH_EXECUTION_STRING='set|grep IFS'\nIFS=' \t\n
f = sub.Popen(['/bin/bash', '-c', 'set|grep IFS'], stdout=sub.PIPE)
f.stdout.read()
BASH_EXECUTION_STRING='set|grep IFS'\nIFS=$' \\t\\n'\n
But I still don't understand what is happening. The manual says that
when shell=True the executable argument specifies which shell to use:
f = sub.Popen('set|grep IFS', shell=True, executable='/bin/sh',
stdout=sub.PIPE)
f.stdout.read()
BASH_EXECUTION_STRING='set|grep IFS'\nIFS=' \t\n
f = sub.Popen('set|grep IFS', shell=True, executable='/bin/bash',
stdout=sub.PIPE)
f.stdout.read()
BASH_EXECUTION_STRING='set|grep IFS'\nIFS=' \t\n
To make my confusion bigger, in Fedora sh is just a link to bash:
$ ll /bin/*sh
-rwxr-xr-x 1 root root 720888 Feb 11 2006 /bin/bash
lrwxrwxrwx 1 root root 4 Aug 28 22:53 /bin/csh - tcsh
-rwxr-xr-x 1 root root 1175496 Aug 17 13:19 /bin/ksh
lrwxrwxrwx 1 root root 4 Feb 24 2006 /bin/sh - bash
-rwxr-xr-x 1 root root 349312 Aug 17 15:20 /bin/tcsh
-rwxr-xr-x 1 root root 514668 Feb 12 2006 /bin/zsh
--
http://mail.python.org/mailman/listinfo/python-list
Output from subprocess.Popen()
Output from the shell:
[EMAIL PROTECTED] teste]$ set | grep IFS
IFS=$' \t\n'
Output from subprocess.Popen():
import subprocess as sub
p = sub.Popen('set | grep IFS', shell=True, stdout=sub.PIPE)
p.stdout.readlines()[1]
IFS=' \t\n
Both outputs for comparison:
IFS=$' \t\n'
IFS=' \t\n
The subprocess.Popen() output is missing the $ and the last '
How to get the raw shell output from subprocess.Popen()?
Regards, Clodoaldo Pinto Neto
--
http://mail.python.org/mailman/listinfo/python-list
Re: Output from subprocess.Popen()
Fredrik Lundh wrote: this works for me: f = subprocess.Popen(set | grep IFS, shell=True, stdout=subprocess.PIPE) f.stdout.readlines() [IFS=$' \\t\\n'\n] what does the above return on your machine? f = subprocess.Popen(set | grep IFS, shell=True, stdout=subprocess.PIPE) f.stdout.readlines() [BASH_EXECUTION_STRING='set | grep IFS'\n, IFS=' \t\n] I'm on FC5 -- http://mail.python.org/mailman/listinfo/python-list
Re: CGI Tutorial
[EMAIL PROTECTED] wrote:
Clodoaldo Pinto Neto wrote:
print 'pThe submited name was ' + name + '/p'
Bzzt! Script injection security hole. See cgi.escape and use it (or a
similar function) for *all* text - HTML output.
open('files/' + fileitem.filename, 'w')
BZZT. filesystem overwriting security hole, possibly escalatable to
code execution. clue: fileitem.filename= '../../something.py'
Do you think os.path.basename() is good enough?
#!/usr/bin/env python
import cgi, os.path
form = cgi.FieldStorage()
fileitem = form['file']
fn = fileitem.filename
fnb = os.path.basename(fn)
print \
Content-Type: text/plain\n
filename = %s
basename = %s
% (fn, fnb)
[EMAIL PROTECTED] ~]$ nc teste.s0 80
POST /cgi-bin/dir_traversal.py HTTP/1.1
Host: teste.s0
Content-Type: multipart/form-data;
boundary=---170451527316340742161395972977
Content-Length: 226
-170451527316340742161395972977
Content-Disposition: form-data; name=file; filename=../test.txt
Content-Type: text/plain
file text
-170451527316340742161395972977--
HTTP/1.1 200 OK
Date: Fri, 06 Oct 2006 20:48:58 GMT
Server: Apache/2.2.2 (Fedora)
Content-Length: 48
Content-Type: text/plain; charset=UTF-8
filename = ../test.txt
basename = test.txt
Regards, Clodoaldo
--
http://mail.python.org/mailman/listinfo/python-list
CGI Tutorial
I'm just building a Python CGI Tutorial and would appreciate any feedback from the many experts in this list. Regards, Clodoaldo Pinto Neto -- http://mail.python.org/mailman/listinfo/python-list
Re: CGI Tutorial
Clodoaldo Pinto Neto wrote: I'm just building a Python CGI Tutorial and would appreciate any feedback from the many experts in this list. http://webpython.codepoint.net -- http://mail.python.org/mailman/listinfo/python-list
Re: CGI Tutorial
Clodoaldo Pinto Neto wrote: I'm just building a Python CGI Tutorial and would appreciate any feedback from the many experts in this list. http://webpython.codepoint.net -- http://mail.python.org/mailman/listinfo/python-list
Re: CGI Tutorial
2006/10/4, Tim Chase [EMAIL PROTECTED]: I'm just building a Python CGI Tutorial and would appreciate any feedback from the many experts in this list. http://webpython.codepoint.net Thanks! :) My first note would be regarding http://webpython.codepoint.net/shell_commands The code is very dangerous...allowing any ol' schmoe to run arbitrary code on your server. At the barest of minimums, I'd plaster the code with warnings that this is a Very Dangerous Thing(tm) to do. I though the danger was so obvious that i didn't bother. Now i have issued a warning. Similarly, regarding http://webpython.codepoint.net/debugging you might want to caution that this will/can display potentially sensitive information (passwords, internal file-structure, etc), and thus should only be used while debugging, and turned off in any sort of production code. Yes, another warning was issued. Thanks for your help. Clodoaldo Pinto Neto -- http://mail.python.org/mailman/listinfo/python-list
