Re: Web servers
Virgil Stokes wrote: Any suggestions on using Python to connect to Web servers (e.g. to access financial time series data)? --V. Stokes You can open a web page for reading with urllib2 module. You can parse html with beautiful soup, or if it's clean xhtml, with the xml module. But parsing raw html is error prone, and subject to change as the web designer reorganizes things. So many web servers also have a protocol intended for data (as opposed to intended for a browser). This is specific to each service, however. If you want to get started in your reading, you could google for "web services", which is one approach using SOAP & WSDL. Note also that most servers have restrictions on the data you access this way. They may or may not be enforceable, but if you access a lot of data from a server, you may be too big a drain on its resources, if it's configured for browser access only. DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Web servers
On Mon, Nov 16, 2009 at 11:17 AM, Virgil Stokes wrote: > Any suggestions on using Python to connect to Web servers (e.g. to access > financial time series data)? In what format? Using what protocol? (*Insert other basic questions that need answering in order to answer your question here*) Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Web servers
Any suggestions on using Python to connect to Web servers (e.g. to access financial time series data)? --V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
Fwd: Twisted vs. CherryPy vs. ??? for light-weight web servers
I just want to be able to write simple scripts to serve xml data and don't want the headache of administrating an apache server. I want to collect some data from some of our production servers and share them with a sharepoint website. On Wed, Sep 24, 2008 at 12:29 PM, Jean-Paul Calderone <[EMAIL PROTECTED]>wrote: > On Tue, 23 Sep 2008 21:22:08 -0500, Michael Mabin <[EMAIL PROTECTED]> > wrote: > >> Is there any consensus on what the best lightweight web-server is? Or >> rather would Twisted be a better choice to choose as a framework that >> allows >> me to serve html or xml data for light webservices. Or is CherryPy just as >> good? >> >> > You haven't described the problem you want to solve in very much detail. I > can't tell, for example, why I shouldn't recommend that you use Apache > instead > of CherryPy or Twisted or anything else. Apache has a huge user community, > lots of documentation, and lots of developers fixing its bugs and making it > work well. What are you trying to do that would make Apache a bad choice? > > Jean-Paul > > -- > http://mail.python.org/mailman/listinfo/python-list > -- | _ | * | _ | | _ | _ | * | | * | * | * | -- | _ | * | _ | | _ | _ | * | | * | * | * | -- http://mail.python.org/mailman/listinfo/python-list
Re: Twisted vs. CherryPy vs. ??? for light-weight web servers
On Tue, 23 Sep 2008 21:22:08 -0500, Michael Mabin <[EMAIL PROTECTED]> wrote: Is there any consensus on what the best lightweight web-server is? Or rather would Twisted be a better choice to choose as a framework that allows me to serve html or xml data for light webservices. Or is CherryPy just as good? You haven't described the problem you want to solve in very much detail. I can't tell, for example, why I shouldn't recommend that you use Apache instead of CherryPy or Twisted or anything else. Apache has a huge user community, lots of documentation, and lots of developers fixing its bugs and making it work well. What are you trying to do that would make Apache a bad choice? Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Twisted vs. CherryPy vs. ??? for light-weight web servers
Is there any consensus on what the best lightweight web-server is? Or rather would Twisted be a better choice to choose as a framework that allows me to serve html or xml data for light webservices. Or is CherryPy just as good? -- | _ | * | _ | | _ | _ | * | | * | * | * | -- http://mail.python.org/mailman/listinfo/python-list
Re: Monitoring SSHd and web servers?
Gilles Ganault <[EMAIL PROTECTED]> wrote:
>I'd like to monitor connections to a remote SSH and web server. Does
>someone have some code handy that would try to connect every 5mn, and
>print an error if the script can't connect?
This script has been pretty reliable for us for the past few years,
much more reliable than the expect script it replaced.
#!/usr/bin/python
import sys
import pexpect
quiet = open("/dev/null", "rw")
sys.stdin = quiet
sys.stdout = quiet
sys.stderr = quiet
smtp = pexpect.spawn("telnet " + sys.argv[1] + " 25")
smtp.expect('220', timeout=120)
smtp.sendline("quit\n^]q")
Pacman
--
http://mail.python.org/mailman/listinfo/python-list
Re: Monitoring SSHd and web servers?
I would recommend using a tried-and-true solution for making sure your uptime of various services is maximized (if that's what your goal is). Running a local "daemon-monitoring daemon" is one option--monit does a good job. Checking services over the network, as nagios does well, is another solution. Jonathan Gardner wrote: > On Mar 13, 11:32 pm, Gilles Ganault <[EMAIL PROTECTED]> wrote: > >> I'd like to monitor connections to a remote SSH and web server. Does >> someone have some code handy that would try to connect every 5mn, and >> print an error if the script can't connect? >> > > from time import sleep > while True: > # Try to connect. May want to spawn a subprocess running a simple > shell script > # Handle success / failure appropriately > sleep 5*60 # Sleep for 5 minutes > > What you monitor is up to you. At a basic level, you can see if the > server is accepting connections. At a higher level, see if you can get > a page or see if you can login as a specific person. At a higher > level, you may want to check what is on the page or what happens when > you log in. It's all up to you. > > > -- Shane Geiger IT Director National Council on Economic Education [EMAIL PROTECTED] | 402-438-8958 | http://www.ncee.net Leading the Campaign for Economic and Financial Literacy -- http://mail.python.org/mailman/listinfo/python-list
Re: Monitoring SSHd and web servers?
On Mar 13, 11:32 pm, Gilles Ganault <[EMAIL PROTECTED]> wrote: > I'd like to monitor connections to a remote SSH and web server. Does > someone have some code handy that would try to connect every 5mn, and > print an error if the script can't connect? from time import sleep while True: # Try to connect. May want to spawn a subprocess running a simple shell script # Handle success / failure appropriately sleep 5*60 # Sleep for 5 minutes What you monitor is up to you. At a basic level, you can see if the server is accepting connections. At a higher level, see if you can get a page or see if you can login as a specific person. At a higher level, you may want to check what is on the page or what happens when you log in. It's all up to you. -- http://mail.python.org/mailman/listinfo/python-list
Monitoring SSHd and web servers?
Hello I'd like to monitor connections to a remote SSH and web server. Does someone have some code handy that would try to connect every 5mn, and print an error if the script can't connect? Thank you. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Web Servers and Page Retrievers
"Collin Stocks" <[EMAIL PROTECTED]> wrote:
> --=_Part_19087_21002019.1176329323968
> I tried it, and when checking it using a proxy, saw that it
> didn't really work, at least in the version that I have (urllib
> v1.17 and urllib2 v2.5). It just added that header onto the end,
> therefore making there two User-Agent headers, each with
> different values. I might add that my script IS able to retrieve
> search pages from Google, whereas both urllibs are FORBIDDEN with
> the headers that they use.
>
I don't know enough about either library to argue about it, but here
is what I get following the Dive Into Python example(but hitting
google for a search):
>>> import urllib2
>>> opener=urllib2.build_opener()
>>> request=urllib2.Request('http://www.google.com/search?
q=tesla+battery')
>>> request.add_header('User-Agent','OpenAnything/1.0
+http://diveintopython.org/')
>>> data=opener.open(request).read()
>>> data
'tesla battery - Google Search<
[snip rest of results page]
This is with python 2.5 on windows.
max
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python Web Servers and Page Retrievers
And yes, I do have two email addresses that I use for Python-List On 4/11/07, Collin Stocks <[EMAIL PROTECTED]> wrote: I tried it, and when checking it using a proxy, saw that it didn't really work, at least in the version that I have (urllib v1.17 and urllib2 v2.5). It just added that header onto the end, therefore making there two User-Agent headers, each with different values. I might add that my script IS able to retrieve search pages from Google, whereas both urllibs are FORBIDDEN with the headers that they use. On 4/8/07, Max Erickson <[EMAIL PROTECTED]> wrote: > > Subscriber123 <[EMAIL PROTECTED]> wrote: > > urllib, or urllib2 for advanced users. For example, you can > > easily set your own headers when retrieving and serving pages, > > such as the User-Agent header which you cannot set in either > > urllib or urllib2. > > Sure you can. See: > > http://www.diveintopython.org/http_web_services/user_agent.html > > (though the behavior was changed for python 2.3 to make setting the > user agent work better) > > > max > > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Web Servers and Page Retrievers
I tried it, and when checking it using a proxy, saw that it didn't really work, at least in the version that I have (urllib v1.17 and urllib2 v2.5). It just added that header onto the end, therefore making there two User-Agent headers, each with different values. I might add that my script IS able to retrieve search pages from Google, whereas both urllibs are FORBIDDEN with the headers that they use. On 4/8/07, Max Erickson <[EMAIL PROTECTED]> wrote: Subscriber123 <[EMAIL PROTECTED]> wrote: > urllib, or urllib2 for advanced users. For example, you can > easily set your own headers when retrieving and serving pages, > such as the User-Agent header which you cannot set in either > urllib or urllib2. Sure you can. See: http://www.diveintopython.org/http_web_services/user_agent.html (though the behavior was changed for python 2.3 to make setting the user agent work better) max -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Web Servers and Page Retrievers
Subscriber123 <[EMAIL PROTECTED]> wrote: > urllib, or urllib2 for advanced users. For example, you can > easily set your own headers when retrieving and serving pages, > such as the User-Agent header which you cannot set in either > urllib or urllib2. Sure you can. See: http://www.diveintopython.org/http_web_services/user_agent.html (though the behavior was changed for python 2.3 to make setting the user agent work better) max -- http://mail.python.org/mailman/listinfo/python-list
Python Web Servers and Page Retrievers
I wrote most of the following script, useful for retrieving pages from the
web and serving web pages. Since it is so low level, it is much more
customizable than simpleHTTPserver, cgiHTTPserver, urllib, or urllib2 for
advanced users. For example, you can easily set your own headers when
retrieving and serving pages, such as the User-Agent header which you cannot
set in either urllib or urllib2.
(sorry for not putting in any comments!)
By the way, I just threw this together quickly, and haven't really had time
to test retrieve() very much. Please let me know if it is buggy.
I guess I should also write a dictToQuery() function. Oh well.
import socket
host,port='',80
sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
sock.bind((host,port))
sock.listen(1)
def serve(function=lambda *args:(args[2],200,'OK',{},'')):
"""\
def serve(function(method,filename,httpversion,headers,get,post))
Serves one request, calling function() with the above
parameters. function() must return (httpversion,code,
accepted,headers,content) in that order. If you don't
pass a function, then
function=lambda *args:(args[2],200,'OK',{},'')
"""
csock,caddr=sock.accept()
rfile=csock.makefile('r',0)
wfile=csock.makefile('w',0)
# Protocol exchange - read request
headers={}
line=rfile.readline().strip()
split1=line.find(' ')
method,remainder=line[:split1].strip(),line[split1+1:].strip()
split2=remainder.find(' ')
filename,httpversion=remainder[:split2].strip(),remainder[split2+1:].strip()
while 1:
line=rfile.readline().strip()
print line
if line=='':
break
else:
split=line.find(':')
key,value=line[:split],line[split+1:]
headers[key.strip()]=value.strip()
try:
post=rfile.read(int(headers['Content-Length']))
except:
post=''
get=queryToDict(filename)
post=queryToDict(post)
loc=filename.find("?")
if loc>-1:
filename=filename[:loc]
print "get:",`get`
print "post:",`post`
httpversion,code,accepted,headers,content=function(method,filename,httpversion,headers,get,post)
wfile.write("%s %s %s\n"%(httpversion,code,accepted))
for header in list(headers):
wfile.write("%s: %s\n"%(header,headers[header]))
wfile.write("\n%s\n"%content)
wfile.close()
csock.close()
def
retrieve(host,port=80,method='GET',filename='/',httpversion='HTTP/1.0',headers={},post=''):
"""\
Retrieves one web page from:
http://host:port/filename
with the headers
"""
sock.connect((host,port))
rfile=sock.makefile('r',0)
wfile=sock.makefile('w',0)
wfile.write("%s %s %s\n"%(method,filename,httpversion))
for header in list(headers):
wfile.write("%s: %s\n"%(header,headers[header]))
wfile.write('\n')
wfile.write("%s\n"%post)
headers={}
line=rfile.readline().strip()
split1=line.find(' ')
httpversion,remainder=line[:split1].strip(),line[split1+1:].strip()
split2=remainder.find(' ')
code,accepted=remainder[:split2].strip(),remainder[split2+1:].strip()
while 1:
line=rfile.readline().strip()
if line=='':
break
else:
split=line.find(':')
key,value=line[:split],line[split+1:]
headers[key.strip()]=value.strip()
return httpversion,code,accepted,headers,rfile
def queryToDict(query):
if '?' in query:
query=query[query.index('?')+1:]
kvpairs=query.split("&")
ret={}
for kvpair in kvpairs:
if '=' in kvpair:
loc=kvpair.index('=')
key,value=kvpair[:loc],kvpair[loc+1:]
ret[key]=value
return ret
if __name__=='__main__':
i=0
while True:
i+=1
print "\nserve #%d:"%i
serve(lambda
*args:(args[2],200,'OK',{'Content-Type':'text/html'},'Go Away!'))
--
http://mail.python.org/mailman/listinfo/python-list
