Re: Parsing/Crawler Questions - solution

2009-03-08 Thread lkcl
list-bounces+bedouglas=earthlink@python.org > > [mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf > Oflkcl > Sent: Saturday, March 07, 2009 2:34 AM > To: python-l...@python.org > Subject: Re: Parsing/Crawler Questions - solution > > On Mar 7, 12:19

RE: Parsing/Crawler Questions - solution

2009-03-07 Thread bruce
Behalf Of lkcl Sent: Saturday, March 07, 2009 2:34 AM To: python-list@python.org Subject: Re: Parsing/Crawler Questions - solution On Mar 7, 12:19 am, rounderwe...@gmail.com wrote: > So, it sounds like your update means that it is related to a specific > url. > > I'm curious about

Re: Parsing/Crawler Questions - solution

2009-03-07 Thread lkcl
On Mar 7, 12:19 am, rounderwe...@gmail.com wrote: > So, it sounds like your update means that it is related to a specific > url. > > I'm curious about this issue myself. I've often wondered how one > could properly crawl anAJAX-ish site when you're not sure how quickly > the data will be returned

Re: Parsing/Crawler Questions - solution

2009-03-07 Thread Lie Ryan
bruce wrote: john... again the problem i'm facing really has nothing to do with a specific url... the app i have for the usc site works... but for any number of reasons... you might get different results when running the app.. -the server could be screwed up.. -data might be cached -data mi

Re: Parsing/Crawler Questions - solution

2009-03-06 Thread rounderweget
; -Original Message- > From: python-list-bounces+bedouglas=earthlink@python.org > > [mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf > Of John Nagle > Sent: Thursday, March 05, 2009 10:54 AM > To: python-l...@python.org > Subject: Re: Parsin

RE: Parsing/Crawler Questions - solution

2009-03-05 Thread bruce
@python.org]on Behalf Of John Nagle Sent: Thursday, March 05, 2009 10:54 AM To: python-list@python.org Subject: Re: Parsing/Crawler Questions - solution Philip Semanchuk wrote: > On Mar 5, 2009, at 12:31 PM, bruce wrote: > >> hi.. >> >> the url i'm focusing on is irrele

RE: Parsing/Crawler Questions - solution

2009-03-05 Thread bruce
t's on the site... ain't screen scraping fun!!! -Original Message- From: python-list-bounces+bedouglas=earthlink@python.org [mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf Of John Nagle Sent: Thursday, March 05, 2009 10:54 AM To: python-list@python.org Sub

Re: Parsing/Crawler Questions - solution

2009-03-05 Thread John Nagle
Philip Semanchuk wrote: On Mar 5, 2009, at 12:31 PM, bruce wrote: hi.. the url i'm focusing on is irrelevant to the issue i'm trying to solve at this time. Not if we're to understand the situation you're trying to describe. From what I can tell, you're saying that the target site displays d

Re: Parsing/Crawler Questions..

2009-03-05 Thread Philip Semanchuk
anks -Original Message- From: python-list-bounces+bedouglas=earthlink@python.org [mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf Of John Nagle Sent: Thursday, March 05, 2009 8:38 AM To: python-list@python.org Subject: Re: Parsing/Crawler Questions.. br

RE: Parsing/Crawler Questions..

2009-03-05 Thread bruce
on.org]on Behalf Of John Nagle Sent: Thursday, March 05, 2009 8:38 AM To: python-list@python.org Subject: Re: Parsing/Crawler Questions.. bruce wrote: > hi john.. > > You're missing the issue, so a little clarification... > > I've got a number of test parsers that point to a g

Re: Parsing/Crawler Questions..

2009-03-05 Thread John Nagle
bruce wrote: hi john.. You're missing the issue, so a little clarification... I've got a number of test parsers that point to a given classlist site.. the scripts work. the issue that one faces is that you never "know" if you've gotten all of the items/links that you're looking for based on th

RE: Parsing/Crawler Questions..

2009-03-05 Thread bruce
parsing functions... Thanks.. -Original Message- From: python-list-bounces+bedouglas=earthlink@python.org [mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf Of John Nagle Sent: Wednesday, March 04, 2009 10:23 PM To: python-list@python.org Subject: Re: Parsing/

Re: Parsing/Crawler Questions..

2009-03-04 Thread John Nagle
bruce wrote: hi phillip... thanks for taking a sec to reply... i'm solid on the test app i've created.. but as an example.. i have a parse for usc (southern cal) and it exrtacts the courselist/class schedule... my issue was that i realized the multiple runs of the app was giving differentt resu

RE: Parsing/Crawler Questions..

2009-03-04 Thread bruce
.@python.org [mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf Of Philip Semanchuk Sent: Wednesday, March 04, 2009 6:15 PM To: python-list (General) Subject: Re: Parsing/Crawler Questions.. On Mar 4, 2009, at 4:44 PM, bruce wrote: > Hi... > > Sorry that this is a bit of

Re: Parsing/Crawler Questions..

2009-03-04 Thread Philip Semanchuk
On Mar 4, 2009, at 4:44 PM, bruce wrote: Hi... Sorry that this is a bit off track. Ok, maybe way off track! But I don't have anyone to bounce this off of.. I'm working on a crawling project, crawling a college website, to extract course/class information. I've built a quick test app in pyt

Re: Parsing/Crawler Questions..

2009-03-04 Thread MRAB
bruce wrote: Hi... Sorry that this is a bit off track. Ok, maybe way off track! But I don't have anyone to bounce this off of.. I'm working on a crawling project, crawling a college website, to extract course/class information. I've built a quick test app in python to crawl the site. I crawl a

Parsing/Crawler Questions..

2009-03-04 Thread bruce
Hi... Sorry that this is a bit off track. Ok, maybe way off track! But I don't have anyone to bounce this off of.. I'm working on a crawling project, crawling a college website, to extract course/class information. I've built a quick test app in python to crawl the site. I crawl at the top level