On Mon, Jun 9, 2008 at 2:56 PM, Philip Stephens <[EMAIL PROTECTED]> wrote:
> Hello wget users and developers, > > I am attempting to retrieve public information from BostonResourceNet ( > http://www.bostonresourcenet.org/index.cfm ). There are four drop-down > boxes that allow you to filter the database. After retrieving the results of > a particular query, I can select a non-profit, for instance, and copy the > link, then pass that long link back into the browser and retrieve the > company's information. Of course, I could just click the link and get the > info as well. My point is that I can pass the long link (as seen in the wget > query below) directly to Firefox and retrieve the data. However, when I try > to pass the long link to wget, I get the following: (Note that the second > return to the command line was not initiated by me). Is wget an appropriate > tool for this kind of operation? I plan to script this, replacing variables > in the string that is passed to wget. > > Thanks. > > > [EMAIL PROTECTED]:~/Documents/projects/Boston/psdiss/code$ wget -dv > http://www.bostonresourcenet.org/detail.cfm?StartRow=2&EndRow=99&SearchType=detailed&VarZip=&VarMiles=0&keyword=&ServID=17&OrgTypeID=&LangID=&SpecialID=&LocationID= > Setting --verbose (verbose) to 1 > DEBUG output created by Wget 1.11.2 on linux-gnu. > > --2008-06-09 14:08:08-- > http://www.bostonresourcenet.org/detail.cfm?StartRow=2 > Resolving www.bostonresourcenet.org... [1] 12857 > [2] 12858 > [3] 12859 > [4] 12860 > [5] 12861 > [6] 12862 > [7] 12863 > [8] 12864 > [9] 12865 > [10] 12866 > [2] Done EndRow=99 > [3] Done SearchType=detailed > [4] Done VarZip= > [5] Done VarMiles=0 > [6] Done keyword= > [7] Done ServID=17 > [8] Done OrgTypeID= > [9] Done LangID= > [10]+ Done SpecialID= > [EMAIL PROTECTED]:~/Documents/projects/Boston/psdiss/code$ 69.25.69.36 > Caching www.bostonresourcenet.org => 69.25.69.36 > Connecting to www.bostonresourcenet.org|69.25.69.36|:80... connected. > Created socket 3. > Releasing 0x08090e48 (new refcount 1). > > ---request begin--- > GET /detail.cfm?StartRow=2 HTTP/1.0 > User-Agent: Wget/1.11.2 > Accept: */* > Host: www.bostonresourcenet.org > Connection: Keep-Alive > > ---request end--- > HTTP request sent, awaiting response... > ---response begin--- > HTTP/1.1 500 Internal Server Error > Connection: close > Date: Mon, 09 Jun 2008 21:08:09 GMT > Server: Microsoft-IIS/6.0 > X-Powered-By: ASP.NET > server-error: true > Content-Type: text/html; charset=UTF-8 > Set-Cookie: CFID=765597;expires=Wed, 02-Jun-2038 21:00:15 GMT;path=/ > Set-Cookie: CFTOKEN=93009364;expires=Wed, 02-Jun-2038 21:00:15 GMT;path=/ > > ---response end--- > 500 Internal Server Error > > Stored cookie www.bostonresourcenet.org -1 (ANY) / <session> <insecure> > [expiry none] CFID 765597 > > Stored cookie www.bostonresourcenet.org -1 (ANY) / <session> <insecure> > [expiry none] CFTOKEN 93009364 > Closed fd 3 > 2008-06-09 14:08:09 ERROR 500: Internal Server Error. > > I should also mention that at this point, the process hangs and must be killed with CTRl-C. -- Philip Stephens Doctoral Student Joint Geography Program San Diego State University UC Santa Barbara
