Re: Question RE urllib

[Tobiah]

On 12/17/2013 08:10 AM, Larry Martell wrote:

On Tue, Dec 17, 2013 at 10:26 AM, Jeff James mailto:j...@jeffljames.com>> wrot


So I'm using the following script to check our sites to make sure they are 
all up and some of them are reporting they are
"down" when, in fact, they are actually up.   These sites do not require a 
logon in order for the home page to come up.  Could
this be due to some port being blocked internally ?  Only one of the sites reporting 
as down is "https" but all are internal
sites.  Is there some other component I should be including in the script ? 
 There are about 30 or 40 sites that I have listed
in all.  I just use those in the following script as examples.   Thanks

import urllib

sites = ["http://www.amazon.com/";, 
"https://internalsite.com/intranet.html";, etc.]

for site in sites:
try:
urllib.urlopen(site)
print site + " "
except Exception, e:
print site + " is down"
--
https://mail.python.org/mailman/listinfo/python-list


I've never used urllib, although I've done a fair amount of network 
programming at lower levels.

Are you sure the report of "down" isn't simply a time out due to the server 
being busier than you expect when you hit it?

-Bill

After adding the line suggested by Larry, I was able to determine that the URLs 
reporting as "down" were actually sites
requiring authentication in order to provide site content, so adding that 
line to the handler was at least enlightening in that
respect.  Thanks Larry.

Glad to help. Here is some info on authenticating with urllib:

http://docs.python.org/2.7/howto/urllib2.html#id6





It must be a network problem, cuz your code works fine:

:w !python
http://www.amazon.com/
http://google.com
http://tobiah.org
http://notavalidurl.com
http://superreallyforsurenotavalidurlnokidding.com is down

Tobiah
--
https://mail.python.org/mailman/listinfo/python-list

Re: Question RE urllib

[Larry Martell]

On Tue, Dec 17, 2013 at 10:26 AM, Jeff James  wrot
>
>  So I'm using the following script to check our sites to make sure they
> are all up and some of them are reporting they are "down" when, in fact,
> they are actually up.   These sites do not require a logon in order for the
> home page to come up.  Could this be due to some port being blocked
> internally ?  Only one of the sites reporting as down is "https" but all
> are internal sites.  Is there some other component I should be including in
> the script ?  There are about 30 or 40 sites that I have listed in all.  I
> just use those in the following script as examples.   Thanks
>
> import urllib
>
> sites = ["http://www.amazon.com/";, "https://internalsite.com/intranet.html";,
> etc.]
>
> for site in sites:
> try:
> urllib.urlopen(site)
> print site + " "
> except Exception, e:
> print site + " is down"
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
> I've never used urllib, although I've done a fair amount of network
> programming at lower levels.
>
> Are you sure the report of "down" isn't simply a time out due to the
> server being busier than you expect when you hit it?
>
> -Bill
>
> After adding the line suggested by Larry, I was able to determine that the
> URLs reporting as "down" were actually sites requiring authentication in
> order to provide site content, so adding that line to the handler was at
> least enlightening in that  respect.  Thanks Larry.
>
Glad to help. Here is some info on authenticating with urllib:

http://docs.python.org/2.7/howto/urllib2.html#id6

>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Question RE urllib

[Jeff James]

So I'm using the following script to check our sites to make sure they are
all up and some of them are reporting they are "down" when, in fact, they
are actually up.   These sites do not require a logon in order for the home
page to come up.  Could this be due to some port being blocked internally ?
 Only one of the sites reporting as down is "https" but all are internal
sites.  Is there some other component I should be including in the script ?
 There are about 30 or 40 sites that I have listed in all.  I just use
those in the following script as examples.   Thanks

import urllib

sites = ["http://www.amazon.com/";, "https://internalsite.com/intranet.html";,
etc.]

for site in sites:
try:
urllib.urlopen(site)
print site + " "
except Exception, e:
print site + " is down"
-- 
https://mail.python.org/mailman/listinfo/python-list


I've never used urllib, although I've done a fair amount of network
programming at lower levels.

Are you sure the report of "down" isn't simply a time out due to the server
being busier than you expect when you hit it?

-Bill

After adding the line suggested by Larry, I was able to determine that the
URLs reporting as "down" were actually sites requiring authentication in
order to provide site content, so adding that line to the handler was at
least enlightening in that respect.  Thanks Larry.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question RE urllib

[William Ray Wing]

On Dec 16, 2013, at 6:40 AM, Jeff James  wrote:

> So I'm using the following script to check our sites to make sure they are 
> all up and some of them are reporting they are "down" when, in fact, they are 
> actually up.   These sites do not require a logon in order for the home page 
> to come up.  Could this be due to some port being blocked internally ?  Only 
> one of the sites reporting as down is "https" but all are internal sites.  Is 
> there some other component I should be including in the script ?  There are 
> about 30 or 40 sites that I have listed in all.  I just use those in the 
> following script as examples.   Thanks
> 
> import urllib
> 
> sites = ["http://www.amazon.com/";, "https://internalsite.com/intranet.html";, 
> etc.]
> 
> for site in sites:
> try:
> urllib.urlopen(site)
> print site + " "
> except Exception, e:
> print site + " is down"
> -- 
> https://mail.python.org/mailman/listinfo/python-list

I've never used urllib, although I've done a fair amount of network programming 
at lower levels.

Are you sure the report of "down" isn't simply a time out due to the server 
being busier than you expect when you hit it?

-Bill-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question RE urllib

[John Gordon]

In  Jeff James 
 writes:

> --f46d04479f936227ee04edac31bd
> Content-Type: text/plain; charset=ISO-8859-1

> Sorry to be a pain here, guys, as I'm also a newbie at this as well.

> Where, exactly in the script would I place the " print str(e)  " ?

except Exception, e:
print site + " is down"
print str(e)

-- 
John Gordon Imagine what it must be like for a real medical doctor to
gor...@panix.comwatch 'House', or a real serial killer to watch 'Dexter'.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question Re urllib (Resolved)

[Jeff James]

This worked perfectly.   Thank You


Where, exactly in the script would I place the " print str(e)  " ?

The line after the print site + " is down" line.



Original Post :



I'm not really receiving an "exception" other than those three sites, out
of the 30 or so I have listed, are the only sites which show "is down" at
the end of that line specifying the site.

Where " # " has been substituted for our domain name

https://my..com/intranet.html
* is
down*
http://#.main..com/psso/pssignsso.asp?dbname=FSPRD90
* is down*
http://sharepoint..com/regions/west/PHX_NSC/default.aspx
* is down*




> So I'm using the following script to check our sites to make sure they are
> all up and some of them are reporting they are "down" when, in fact, they
> are actually up.   These sites do not require a logon in order for the
home
> page to come up.  Could this be due to some port being blocked internally
?
> Only one of the sites reporting as down is "https" but all are internal
> sites.  Is there some other component I should be including in the script
?
> There are about 30 or 40 sites that I have listed in all.  I just use
those
> in the following script as examples.   Thanks
>
> import urllib
>
> sites = ["http://www.amazon.com/";, "https://internalsite.com/intranet.html
",
> etc.]
>
> for site in sites:
> try:
> urllib.urlopen(site)
> print site + " "
> except Exception, e:
> print site + " is down"

In the handler print out the exception you are getting.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question RE urllib

[Larry Martell]

On Mon, Dec 16, 2013 at 2:55 PM, Jeff James  wrote:
> Sorry to be a pain here, guys, as I'm also a newbie at this as well.
>
> Where, exactly in the script would I place the " print str(e)  " ?

The line after the print site + " is down" line.

>
> Thanks
>
> Original message :
>
>> I'm not really receiving an "exception" other than those three sites, out
>> of the 30 or so I have listed, are the only sites which show "is down" at
>> the end of that line specifying the site.
>>
>> Where " # " has been substituted for our domain name
>>
>> https://my..com/intranet.html is down
>> http://#.main..com/psso/pssignsso.asp?dbname=FSPRD90 is down
>> http://sharepoint..com/regions/west/PHX_NSC/default.aspx is down
>>
>>
>>
>> > So I'm using the following script to check our sites to make sure they
>> > are
>> > all up and some of them are reporting they are "down" when, in fact,
>> > they
>> > are actually up.   These sites do not require a logon in order for the
>> > home
>>
>> > page to come up.  Could this be due to some port being blocked
>> > internally ?
>> > Only one of the sites reporting as down is "https" but all are internal
>> > sites.  Is there some other component I should be including in the
>> > script ?
>> > There are about 30 or 40 sites that I have listed in all.  I just use
>> > those
>> > in the following script as examples.   Thanks
>> >
>> > import urllib
>> >
>> > sites = ["http://www.amazon.com/";,
>> > "https://internalsite.com/intranet.html";,
>> > etc.]
>> >
>> > for site in sites:
>> > try:
>> > urllib.urlopen(site)
>> > print site + " "
>> > except Exception, e:
>> > print site + " is down"
>>
>> In the handler print out the exception you are getting.
>
>
> If your exception handler is being called then you are indeed getting an
> exception. Add:
>
> print str(e)
>
> to the handler
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Question RE urllib

[Jeff James]

Sorry to be a pain here, guys, as I'm also a newbie at this as well.

Where, exactly in the script would I place the " print str(e)  " ?

Thanks

Original message :

I'm not really receiving an "exception" other than those three sites, out
> of the 30 or so I have listed, are the only sites which show "is down" at
> the end of that line specifying the site.
>
> Where " # " has been substituted for our domain name
>
> https://my..com/intranet.html* 
> is
> down*
> http://#.main..com/psso/pssignsso.asp?dbname=FSPRD90
> * is down*
> http://sharepoint..com/regions/west/PHX_NSC/default.aspx
> * is down*
>
>
>
> > So I'm using the following script to check our sites to make sure they
> are
> > all up and some of them are reporting they are "down" when, in fact, they
> > are actually up.   These sites do not require a logon in order for the
> home
> > page to come up.  Could this be due to some port being blocked
> internally ?
> > Only one of the sites reporting as down is "https" but all are internal
> > sites.  Is there some other component I should be including in the
> script ?
> > There are about 30 or 40 sites that I have listed in all.  I just use
> those
> > in the following script as examples.   Thanks
> >
> > import urllib
> >
> > sites = ["http://www.amazon.com/";, "
> https://internalsite.com/intranet.html";,
> > etc.]
> >
> > for site in sites:
> > try:
> > urllib.urlopen(site)
> > print site + " "
> > except Exception, e:
> > print site + " is down"
>
> In the handler print out the exception you are getting.
>

If your exception handler is being called then you are indeed getting an
exception. Add:

print str(e)

to the handler
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question Re urllib (Jeff James)

[Larry Martell]

On Monday, December 16, 2013, Jeff James wrote:

> I'm not really receiving an "exception" other than those three sites, out
> of the 30 or so I have listed, are the only sites which show "is down" at
> the end of that line specifying the site.
>
>
> Where " # " has been substituted for our domain name
>
> https://my..com/intranet.html* is down*
> http://#.main..com/psso/pssignsso.asp?dbname=FSPRD90
> * is down *
> http://sharepoint..com/regions/west/PHX_NSC/default.aspx
> * is down*
>
>
>
> Cc: python-list@python.org  'python-list@python.org');>
> Date: Mon, 16 Dec 2013 06:54:48 -0500
> Subject: Re: Question RE urllib
> On Mon, Dec 16, 2013 at 6:40 AM, Jeff James 
> >
> wrote:
> > So I'm using the following script to check our sites to make sure they
> are
> > all up and some of them are reporting they are "down" when, in fact, they
> > are actually up.   These sites do not require a logon in order for the
> home
> > page to come up.  Could this be due to some port being blocked
> internally ?
> > Only one of the sites reporting as down is "https" but all are internal
> > sites.  Is there some other component I should be including in the
> script ?
> > There are about 30 or 40 sites that I have listed in all.  I just use
> those
> > in the following script as examples.   Thanks
> >
> > import urllib
> >
> > sites = ["http://www.amazon.com/";, "
> https://internalsite.com/intranet.html";,
> > etc.]
> >
> > for site in sites:
> > try:
> > urllib.urlopen(site)
> > print site + " "
> > except Exception, e:
> > print site + " is down"
>
> In the handler print out the exception you are getting.
>

If your exception handler is being called then you are indeed getting an
exception. Add:

print str(e)

to the handler
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question Re urllib (Jeff James)

[Jeff James]

I'm not really receiving an "exception" other than those three sites, out
of the 30 or so I have listed, are the only sites which show "is down" at
the end of that line specifying the site.

Where " # " has been substituted for our domain name

https://my..com/intranet.html* is down*
http://#.main..com/psso/pssignsso.asp?dbname=FSPRD90
* is down*
http://sharepoint..com/regions/west/PHX_NSC/default.aspx
* is down*



Cc: python-list@python.org
Date: Mon, 16 Dec 2013 06:54:48 -0500
Subject: Re: Question RE urllib
On Mon, Dec 16, 2013 at 6:40 AM, Jeff James  wrote:
> So I'm using the following script to check our sites to make sure they are
> all up and some of them are reporting they are "down" when, in fact, they
> are actually up.   These sites do not require a logon in order for the
home
> page to come up.  Could this be due to some port being blocked internally
?
> Only one of the sites reporting as down is "https" but all are internal
> sites.  Is there some other component I should be including in the script
?
> There are about 30 or 40 sites that I have listed in all.  I just use
those
> in the following script as examples.   Thanks
>
> import urllib
>
> sites = ["http://www.amazon.com/";, "https://internalsite.com/intranet.html
",
> etc.]
>
> for site in sites:
> try:
> urllib.urlopen(site)
> print site + " "
> except Exception, e:
> print site + " is down"

In the handler print out the exception you are getting.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question RE urllib

[Tim Chase]

On 2013-12-16 04:40, Jeff James wrote:
> These sites do not require a logon in order for the home
> page to come up.  Could this be due to some port being blocked
> internally ? Only one of the sites reporting as down is "https" but
> all are internal sites.  Is there some other component I should be
> including in the script ?

>From your description, I'm unsure whether it's *only* the HTTPS sites
that are having issues, or a subset of your URL list that includes
both HTTP and HTTPS (where those sites happen to be internal).

Either way, my first suspicion is that you have some sort of proxy
server in your network that web-browsers use.  The urlopen() accepts a
parameter to list proxy server information, which you should be able
to glean from your browser settings.

If your web-browser isn't using a proxy, the sites might be doing
unsavory things like filtering by user-agent.  Dumping the contents
of the exception (and including it here) might help diagnose.

-tkc



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question RE urllib

[Larry Martell]

On Mon, Dec 16, 2013 at 6:40 AM, Jeff James  wrote:
> So I'm using the following script to check our sites to make sure they are
> all up and some of them are reporting they are "down" when, in fact, they
> are actually up.   These sites do not require a logon in order for the home
> page to come up.  Could this be due to some port being blocked internally ?
> Only one of the sites reporting as down is "https" but all are internal
> sites.  Is there some other component I should be including in the script ?
> There are about 30 or 40 sites that I have listed in all.  I just use those
> in the following script as examples.   Thanks
>
> import urllib
>
> sites = ["http://www.amazon.com/";, "https://internalsite.com/intranet.html";,
> etc.]
>
> for site in sites:
> try:
> urllib.urlopen(site)
> print site + " "
> except Exception, e:
> print site + " is down"

In the handler print out the exception you are getting.
-- 
https://mail.python.org/mailman/listinfo/python-list

Question RE urllib

[Jeff James]

So I'm using the following script to check our sites to make sure they are
all up and some of them are reporting they are "down" when, in fact, they
are actually up.   These sites do not require a logon in order for the home
page to come up.  Could this be due to some port being blocked internally ?
 Only one of the sites reporting as down is "https" but all are internal
sites.  Is there some other component I should be including in the script ?
 There are about 30 or 40 sites that I have listed in all.  I just use
those in the following script as examples.   Thanks

import urllib

sites = ["http://www.amazon.com/";, "https://internalsite.com/intranet.html";,
etc.]

for site in sites:
try:
urllib.urlopen(site)
print site + " "
except Exception, e:
print site + " is down"
-- 
https://mail.python.org/mailman/listinfo/python-list