Re: how to handle network failures

[Aahz]

In article <7bee5155-f1a1-44aa-90f6-eaabb3617...@v6g2000prd.googlegroups.com>,
harryos   wrote:
>
>class DataGrabber(threading.Thread):
>def __init__(self,url):
>threading.Thread.__init__(self)
>self.url=url
>def run(self):
>data=self.get_page_data()
>process_data(data)
>
>def get_page_data():
>try:
>f=urllib.urlopen(self.url)
>data=f.read(1024)
>except IOError:
>#wait for some time and try again
>time.sleep(120)
>data=self.get_page_data()
>return data

Use urllib2 so that you can set a timeout (Python 2.6+).
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

"If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur."  --Red Adair
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to handle network failures

[Jorgen Grahn]

On Fri, 2010-10-08, harryos wrote:
> hi
> I  am trying to write a DataGrabber which reads some data from given
> url..I made DataGrabber as a Thread and want to wait for some interval
> of time in case there is a network failure that prevents read().
> I am not very sure how to implement this
>
> class DataGrabber(threading.Thread):
> def __init__(self,url):
> threading.Thread.__init__(self)
> self.url=url
> def run(self):
> data=self.get_page_data()
> process_data(data)
>
> def get_page_data():
> try:
> f=urllib.urlopen(self.url)
> data=f.read(1024)
> except IOError:
> #wait for some time and try again
> time.sleep(120)
> data=self.get_page_data()
> return data
>
> Is this the way to  implement the part where the thread waits and
> reads the  data again? Will this handle network failures?Can somebody
> please help?

You are using TCP sockets. When you get an error on one of those, the
TCP connection is dead (except for a few special cases like EAGAIN,
EINTR).

But you also risk *not* getting told and hanging forever, or anyway
for far longer than your application is likely to want to wait. For
example if the peer host is suddenly disconnected from the network --
TCP will keep trying, in case a connection suddenly reappears much
later.

Try provoking that situation and see what happens.

/Jorgen

-- 
  // Jorgen GrahnO  o   .
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to handle network failures

[Lawrence D'Oliveiro]

In message , Diez B. Roggisch wrote:

> for n in xrange(max_number_of_retries):
> try:
> f=urllib.urlopen(self.url)
> data = f.read()
> break # exist the loop if all
> except IOError:
> pass

Is it worth delaying before retrying? In case of a transient routing error, 
that kind of thing.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to handle network failures

[Diez B. Roggisch]

harryos  writes:

> hi
> I  am trying to write a DataGrabber which reads some data from given
> url..I made DataGrabber as a Thread and want to wait for some interval
> of time in case there is a network failure that prevents read().
> I am not very sure how to implement this
>
> class DataGrabber(threading.Thread):
> def __init__(self,url):
> threading.Thread.__init__(self)
> self.url=url
> def run(self):
> data=self.get_page_data()
> process_data(data)
>
> def get_page_data():
> try:
> f=urllib.urlopen(self.url)
> data=f.read(1024)
> except IOError:
> #wait for some time and try again
> time.sleep(120)
> data=self.get_page_data()
> return data
>
> Is this the way to  implement the part where the thread waits and
> reads the  data again? Will this handle network failures?Can somebody
> please help?

This only works if your page is always 1024 bytes long. Which I
doubt. So don't pass the 1024 to read.

Also, you need a loop to re-read the data. Like this:


for n in xrange(max_number_of_retries):
try:
f=urllib.urlopen(self.url)
data = f.read()
break # exist the loop if all
except IOError:
pass


self.process_data(data)


Diez
-- 
http://mail.python.org/mailman/listinfo/python-list

how to handle network failures

[harryos]

hi
I  am trying to write a DataGrabber which reads some data from given
url..I made DataGrabber as a Thread and want to wait for some interval
of time in case there is a network failure that prevents read().
I am not very sure how to implement this

class DataGrabber(threading.Thread):
def __init__(self,url):
threading.Thread.__init__(self)
self.url=url
def run(self):
data=self.get_page_data()
process_data(data)

def get_page_data():
try:
f=urllib.urlopen(self.url)
data=f.read(1024)
except IOError:
#wait for some time and try again
time.sleep(120)
data=self.get_page_data()
return data

Is this the way to  implement the part where the thread waits and
reads the  data again? Will this handle network failures?Can somebody
please help?

thanks
harry


-- 
http://mail.python.org/mailman/listinfo/python-list