Re: Odd/Weird errors with FTPLib
On Sep 15, 1:32 am, MRAB wrote:
> Bakes wrote:
> > On Sep 13, 11:47 pm, MRAB wrote:
> >> Bakes wrote:
> >>> On 13 Sep, 22:41, Chris Rebert wrote:
> On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
> > I am using a simple python script to download my logfiles. This is on
> > a while loop, the logfile grows rapidly, so it is necessary for python
> > to start downloading the new script as soon as it has finished the
> > old.
> > It works fine (for about 20 minutes), then crashes. I have removed a
> > couple of excepts, and have narrowed the error down to a 'error_perm:
> > 550 logfile.log: The data is invalid.' error.
> > Does anyone know what the problem might be regarding this, and what I
> > might do to fix it?
> Including an actual code snippet and the full error traceback would help
> a lot.
> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
> error code 550 translates to:
> "Requested action not taken. File unavailable (e.g., file not found,
> no access)."
> Does the logfile get rotated or something, thus causing it to briefly
> not exist?
> It might also help if you explain how your logfile system works.
> Cheers,
> Chris
> --http://blog.rebertia.com
> >>> It's a cod4 gameserver logfile, being downloaded for a python bot to
> >>> parse.
> >>> The logfile is downloaded using this try/except while loop.
> >>> while True:
> >>> try:
> >>> if ftp == False:
> >>> self.debug('FTP connection not active, attempting to
> >>> (re)connect')
> >>> ftp = self.ftpconnect()
> >>> size=os.path.getsize('games_mp.log')
> >>> ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig
> >>> ['path']), handleDownload, rest=size)
> >>> if self.console._paused:
> >>> self.console.unpause()
> >>> except:
> >>> print error
> >>> self.debug('Lost connection to server, pausing until updated
> >>> properly, Sleeping 10 seconds')
> >>> self.console.pause()
> >>> try:
> >>> ftp.close()
> >>> self.debug('FTP Connection Closed')
> >>> except:
> >>> self.debug('FTP does not appear to be open, so not
> >>> closed')
> >>> ftp = False
> >>> time.sleep(10)
> >>> I can only assume that occasionally, the logfile is being written to
> >>> by the gameserver at the same time that it's downloading.
> >>> If this was the case, do you think a try: download except: sleep
> >>> 900msec then download loop would work?
> >> Bare excepts are almost always a bad idea because they'll catch _all_
> >> exceptions, both those you expect could happen and those you don't.
> >> Catch only those you expect.
>
> >> For example, if the file 'games_mp.log' doesn't exist then
> >> os.path.getsize('games_mp.log') will raise an exception, and if you
> >> forgot to import the os module then that will raise a NameError
> >> exception.
>
> >> Anyway, I can't see how you leave the loop; I'd expect something like a
> >> 'break' statement.
>
> >> And as a matter of style, I'd prefer None to False to indicate when
> >> there's no FTP connection (and "if not ftp" instead of "if ftp ==
> >> False").
>
> > I removed the try/except and saw when it failed.
>
> > I'll change those things, games_mp.log is guaranteed to be there (file
> > made in another script), os is imported correctly.
>
> > So, what do you think the error could be?
>
> How does control leave the while loop?
>
> If you're running on Windows and you're opening the file 'games_mp.log'
> with mode 'w' then there'll be the issue of the line-endings. If the log
> file on the server uses '\n' for the line endings and you're using
> '\r\n' then os.path.getsize('games_mp.log') will return a larger size
> then you expect. In that case, could ftp.retrbinary() be complaining
> because it's given you the entire file and then you're trying to
> download from an offset that's beyond the end?
>
> For example, suppose the file on the server contains just "foo\n" (4
> bytes):
>
> 1. You open a file locally in text mode ('w').
>
> 2. You download the entire file, "foo\n".
>
> 3. You write out the data, but because you've opened in text mode it
> writes "foo\r\n" to the file. The local file size is now 5 bytes.
>
> 4. The loop means that you then try to download from offset 5, which is
> beyond the end of the file on the server.
>
> 5. Error?
It opens it in append mode, and if it errors with that message, the
ending is different. I have come across that problem before, and it
has been fixed (download in binary is the fix, mostly).
--
http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
Bakes wrote:
On Sep 13, 11:47 pm, MRAB wrote:
Bakes wrote:
On 13 Sep, 22:41, Chris Rebert wrote:
On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
I am using a simple python script to download my logfiles. This is on
a while loop, the logfile grows rapidly, so it is necessary for python
to start downloading the new script as soon as it has finished the
old.
It works fine (for about 20 minutes), then crashes. I have removed a
couple of excepts, and have narrowed the error down to a 'error_perm:
550 logfile.log: The data is invalid.' error.
Does anyone know what the problem might be regarding this, and what I
might do to fix it?
Including an actual code snippet and the full error traceback would help a lot.
According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
error code 550 translates to:
"Requested action not taken. File unavailable (e.g., file not found,
no access)."
Does the logfile get rotated or something, thus causing it to briefly not exist?
It might also help if you explain how your logfile system works.
Cheers,
Chris
--http://blog.rebertia.com
It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.
The logfile is downloaded using this try/except while loop.
while True:
try:
if ftp == False:
self.debug('FTP connection not active, attempting to
(re)connect')
ftp = self.ftpconnect()
size=os.path.getsize('games_mp.log')
ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']),
handleDownload, rest=size)
if self.console._paused:
self.console.unpause()
except:
print error
self.debug('Lost connection to server, pausing until updated
properly, Sleeping 10 seconds')
self.console.pause()
try:
ftp.close()
self.debug('FTP Connection Closed')
except:
self.debug('FTP does not appear to be open, so not closed')
ftp = False
time.sleep(10)
I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?
Bare excepts are almost always a bad idea because they'll catch _all_
exceptions, both those you expect could happen and those you don't.
Catch only those you expect.
For example, if the file 'games_mp.log' doesn't exist then
os.path.getsize('games_mp.log') will raise an exception, and if you
forgot to import the os module then that will raise a NameError
exception.
Anyway, I can't see how you leave the loop; I'd expect something like a
'break' statement.
And as a matter of style, I'd prefer None to False to indicate when
there's no FTP connection (and "if not ftp" instead of "if ftp ==
False").
I removed the try/except and saw when it failed.
I'll change those things, games_mp.log is guaranteed to be there (file
made in another script), os is imported correctly.
So, what do you think the error could be?
How does control leave the while loop?
If you're running on Windows and you're opening the file 'games_mp.log'
with mode 'w' then there'll be the issue of the line-endings. If the log
file on the server uses '\n' for the line endings and you're using
'\r\n' then os.path.getsize('games_mp.log') will return a larger size
then you expect. In that case, could ftp.retrbinary() be complaining
because it's given you the entire file and then you're trying to
download from an offset that's beyond the end?
For example, suppose the file on the server contains just "foo\n" (4
bytes):
1. You open a file locally in text mode ('w').
2. You download the entire file, "foo\n".
3. You write out the data, but because you've opened in text mode it
writes "foo\r\n" to the file. The local file size is now 5 bytes.
4. The loop means that you then try to download from offset 5, which is
beyond the end of the file on the server.
5. Error?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
On Sep 13, 11:47 pm, MRAB wrote:
> Bakes wrote:
> > On 13 Sep, 22:41, Chris Rebert wrote:
> >> On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
> >>> I am using a simple python script to download my logfiles. This is on
> >>> a while loop, the logfile grows rapidly, so it is necessary for python
> >>> to start downloading the new script as soon as it has finished the
> >>> old.
> >>> It works fine (for about 20 minutes), then crashes. I have removed a
> >>> couple of excepts, and have narrowed the error down to a 'error_perm:
> >>> 550 logfile.log: The data is invalid.' error.
> >>> Does anyone know what the problem might be regarding this, and what I
> >>> might do to fix it?
> >> Including an actual code snippet and the full error traceback would help a
> >> lot.
>
> >> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
> >> error code 550 translates to:
> >> "Requested action not taken. File unavailable (e.g., file not found,
> >> no access)."
>
> >> Does the logfile get rotated or something, thus causing it to briefly not
> >> exist?
>
> >> It might also help if you explain how your logfile system works.
>
> >> Cheers,
> >> Chris
> >> --http://blog.rebertia.com
>
> > It's a cod4 gameserver logfile, being downloaded for a python bot to
> > parse.
>
> > The logfile is downloaded using this try/except while loop.
>
> > while True:
> > try:
> > if ftp == False:
> > self.debug('FTP connection not active, attempting to
> > (re)connect')
> > ftp = self.ftpconnect()
> > size=os.path.getsize('games_mp.log')
> > ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig
> > ['path']), handleDownload, rest=size)
> > if self.console._paused:
> > self.console.unpause()
> > except:
> > print error
> > self.debug('Lost connection to server, pausing until updated
> > properly, Sleeping 10 seconds')
> > self.console.pause()
> > try:
> > ftp.close()
> > self.debug('FTP Connection Closed')
> > except:
> > self.debug('FTP does not appear to be open, so not closed')
> > ftp = False
> > time.sleep(10)
>
> > I can only assume that occasionally, the logfile is being written to
> > by the gameserver at the same time that it's downloading.
> > If this was the case, do you think a try: download except: sleep
> > 900msec then download loop would work?
>
> Bare excepts are almost always a bad idea because they'll catch _all_
> exceptions, both those you expect could happen and those you don't.
> Catch only those you expect.
>
> For example, if the file 'games_mp.log' doesn't exist then
> os.path.getsize('games_mp.log') will raise an exception, and if you
> forgot to import the os module then that will raise a NameError
> exception.
>
> Anyway, I can't see how you leave the loop; I'd expect something like a
> 'break' statement.
>
> And as a matter of style, I'd prefer None to False to indicate when
> there's no FTP connection (and "if not ftp" instead of "if ftp ==
> False").
I removed the try/except and saw when it failed.
I'll change those things, games_mp.log is guaranteed to be there (file
made in another script), os is imported correctly.
So, what do you think the error could be?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
Bakes wrote:
On 13 Sep, 22:41, Chris Rebert wrote:
On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
I am using a simple python script to download my logfiles. This is on
a while loop, the logfile grows rapidly, so it is necessary for python
to start downloading the new script as soon as it has finished the
old.
It works fine (for about 20 minutes), then crashes. I have removed a
couple of excepts, and have narrowed the error down to a 'error_perm:
550 logfile.log: The data is invalid.' error.
Does anyone know what the problem might be regarding this, and what I
might do to fix it?
Including an actual code snippet and the full error traceback would help a lot.
According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
error code 550 translates to:
"Requested action not taken. File unavailable (e.g., file not found,
no access)."
Does the logfile get rotated or something, thus causing it to briefly not exist?
It might also help if you explain how your logfile system works.
Cheers,
Chris
--http://blog.rebertia.com
It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.
The logfile is downloaded using this try/except while loop.
while True:
try:
if ftp == False:
self.debug('FTP connection not active, attempting to
(re)connect')
ftp = self.ftpconnect()
size=os.path.getsize('games_mp.log')
ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']),
handleDownload, rest=size)
if self.console._paused:
self.console.unpause()
except:
print error
self.debug('Lost connection to server, pausing until updated
properly, Sleeping 10 seconds')
self.console.pause()
try:
ftp.close()
self.debug('FTP Connection Closed')
except:
self.debug('FTP does not appear to be open, so not closed')
ftp = False
time.sleep(10)
I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?
Bare excepts are almost always a bad idea because they'll catch _all_
exceptions, both those you expect could happen and those you don't.
Catch only those you expect.
For example, if the file 'games_mp.log' doesn't exist then
os.path.getsize('games_mp.log') will raise an exception, and if you
forgot to import the os module then that will raise a NameError
exception.
Anyway, I can't see how you leave the loop; I'd expect something like a
'break' statement.
And as a matter of style, I'd prefer None to False to indicate when
there's no FTP connection (and "if not ftp" instead of "if ftp ==
False").
--
http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
On 13 Sep, 22:41, Chris Rebert wrote:
> On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote:
> > I am using a simple python script to download my logfiles. This is on
> > a while loop, the logfile grows rapidly, so it is necessary for python
> > to start downloading the new script as soon as it has finished the
> > old.
>
> > It works fine (for about 20 minutes), then crashes. I have removed a
> > couple of excepts, and have narrowed the error down to a 'error_perm:
> > 550 logfile.log: The data is invalid.' error.
>
> > Does anyone know what the problem might be regarding this, and what I
> > might do to fix it?
>
> Including an actual code snippet and the full error traceback would help a
> lot.
>
> According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
> error code 550 translates to:
> "Requested action not taken. File unavailable (e.g., file not found,
> no access)."
>
> Does the logfile get rotated or something, thus causing it to briefly not
> exist?
>
> It might also help if you explain how your logfile system works.
>
> Cheers,
> Chris
> --http://blog.rebertia.com
It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.
The logfile is downloaded using this try/except while loop.
while True:
try:
if ftp == False:
self.debug('FTP connection not active, attempting to
(re)connect')
ftp = self.ftpconnect()
size=os.path.getsize('games_mp.log')
ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig
['path']), handleDownload, rest=size)
if self.console._paused:
self.console.unpause()
except:
print error
self.debug('Lost connection to server, pausing until
updated properly, Sleeping 10 seconds')
self.console.pause()
try:
ftp.close()
self.debug('FTP Connection Closed')
except:
self.debug('FTP does not appear to be open, so not
closed')
ftp = False
time.sleep(10)
I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Odd/Weird errors with FTPLib
On Sun, Sep 13, 2009 at 2:34 PM, Bakes wrote: > I am using a simple python script to download my logfiles. This is on > a while loop, the logfile grows rapidly, so it is necessary for python > to start downloading the new script as soon as it has finished the > old. > > It works fine (for about 20 minutes), then crashes. I have removed a > couple of excepts, and have narrowed the error down to a 'error_perm: > 550 logfile.log: The data is invalid.' error. > > Does anyone know what the problem might be regarding this, and what I > might do to fix it? Including an actual code snippet and the full error traceback would help a lot. According to http://en.wikipedia.org/wiki/List_of_FTP_server_return_codes , error code 550 translates to: "Requested action not taken. File unavailable (e.g., file not found, no access)." Does the logfile get rotated or something, thus causing it to briefly not exist? It might also help if you explain how your logfile system works. Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Odd/Weird errors with FTPLib
I am using a simple python script to download my logfiles. This is on a while loop, the logfile grows rapidly, so it is necessary for python to start downloading the new script as soon as it has finished the old. It works fine (for about 20 minutes), then crashes. I have removed a couple of excepts, and have narrowed the error down to a 'error_perm: 550 logfile.log: The data is invalid.' error. Does anyone know what the problem might be regarding this, and what I might do to fix it? It's really irritating to have it break every so often, and at the moment, I have no chance in which to fix the error. -- http://mail.python.org/mailman/listinfo/python-list
