RE: [PHP-DB] grabbing data from other peoples sites
I guess my real question is how to parse the output from someone else's site. Setting up the lookup procedure from data in my database and filling their form with the address is the easy part. How do I make their form submit, parse the output, store it, and then reuse it in another function? I have never tried something like this before. - Matthew -Original Message- From: Martin Norland [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 16, 2005 10:09 AM To: php-db@lists.php.net Subject: Re: [PHP-DB] grabbing data from other peoples sites Perry, Matthew (Fire Marshal's Office) wrote: [snip] What this has to do with PHP Is there a way to write some sort of PHP script that accesses to the county auditors website, automatically enters the info into their form, grabs whatever information is outputted, imports the data into our SQL Server, and repeats the process for all addresses in our database? I will of course research the legality of this before we implement the script. I just wanted to know if this was even feasible before we meet about this. I don't even know where we should start. Sure, you're just issuing POST requests with data from your database, and parsing the output - and entering that extracted data back into your database again. A better question, however, would be how are you planning to handle this once you are ready to run it - are you just storing the address at each corner of your 'covered regions'? You'll need some kind of fuzzy lookup to be able to do all this. Maybe a better solution would be to just link directly to the county auditors website with instructions, allowing the users to do their queries as they're needed? Even if you have every current covered address on file, people may mistype things - new buildings are created - roads are renamed, etc. Someone should contact maps.google.com and see about adding this sort of information to that :) Cheers, -- - Martin Norland, Sys Admin / Database / Web Developer, International Outreach x3257 The opinion(s) contained within this email do not necessarily represent those of St. Jude Children's Research Hospital. -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] grabbing data from other peoples sites
Perry, Matthew (Fire Marshal's Office) wrote:
I guess my real question is how to parse the output from someone else's
site. Setting up the lookup procedure from data in my database and filling
their form with the address is the easy part. How do I make their form
submit, parse the output, store it, and then reuse it in another function?
I have never tried something like this before.
- Matthew
[snip]
just fyi - you should test this on your own site until it's working as
you expect, then start poking at them - it's only proper, and you can
debug better that way. You may want to see about setting a referrer as
well, they may check for it. Lastly - if they require a cookie of some
sort, you'll have to figure that bit out :) Should be doable, just need
two requests/etc.
example modified from php manual user contrib notes for fsockopen ( joe
at edwardsconsultants dot com (10-Aug-2003 08:56) original example )
//
oh - and it's untested by yours truly.
//
$proto = http;
$host = foobarfoobar.com;
$port = 80; // or 443 for https
$path = /where/on/their/site/lookup_backend.php;
$our_data = array ( address = 55 main street );
$poststring = ;
// build our post
foreach ($our_data as $key = $val) {
$poststring .= urlencode($key) . = . urlencode($val) . ;
}
// strip off trailing ampersand
if ($poststring != ) { $poststring = substr($poststring, 0, -1); }
// done with setup - now let's move on to the action
$fp = fsockopen($proto.'://'.$host, $port, $errno, $errstr, $timeout = 30);
if (!fp) {
echo $errstr ($errno)\n;
} else {
// send request
fputs($fp, POST $path HTTP/1.1\r\n);
fputs($fp, Host: $host\r\n);
fputs($fp, Content-type: application/x-www-form-urlencoded\r\n);
fputs($fp, Content-length: .strlen($poststring).\r\n);
fputs($fp, Connection: close\r\n\r\n);
fputs($fp, $poststring . \r\n\r\n);
// get response
while(!feof($fp)) {
*/
todo - do something other than fgets with this.
it would be better to get the whole response then close
fp, then move on to doing what you need - instead of
doing your database stuff right away.
*/
echo fgets($fp, 4096);
}
//close fp - we are done with it
fclose($fp);
}
Cheers,
--
- Martin Norland, Sys Admin / Database / Web Developer, International
Outreach x3257
The opinion(s) contained within this email do not necessarily represent
those of St. Jude Children's Research Hospital.
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] grabbing data from other peoples sites
Incidentally, as with many other messages that arrive at this list - if
you happen to be using a certain *wonderful* email client that thinks it
knows better than the sender, click the Extra line breaks in this
message were removed. To restore, click here yellow info box - to
avoid the migraine of trying to read unsplit code.
Cheers,
Martin Norland wrote:
Perry, Matthew (Fire Marshal's Office) wrote:
I guess my real question is how to parse the output from someone else's
site. Setting up the lookup procedure from data in my database and
filling
their form with the address is the easy part. How do I make their form
submit, parse the output, store it, and then reuse it in another
function?
I have never tried something like this before.
- Matthew
[snip]
just fyi - you should test this on your own site until it's working as
you expect, then start poking at them - it's only proper, and you can
debug better that way. You may want to see about setting a referrer as
well, they may check for it. Lastly - if they require a cookie of some
sort, you'll have to figure that bit out :) Should be doable, just need
two requests/etc.
example modified from php manual user contrib notes for fsockopen ( joe
at edwardsconsultants dot com (10-Aug-2003 08:56) original example )
//
oh - and it's untested by yours truly.
//
$proto = http;
$host = foobarfoobar.com;
$port = 80; // or 443 for https
$path = /where/on/their/site/lookup_backend.php;
$our_data = array ( address = 55 main street );
$poststring = ;
// build our post
foreach ($our_data as $key = $val) {
$poststring .= urlencode($key) . = . urlencode($val) . ;
}
// strip off trailing ampersand
if ($poststring != ) { $poststring = substr($poststring, 0, -1); }
// done with setup - now let's move on to the action
$fp = fsockopen($proto.'://'.$host, $port, $errno, $errstr, $timeout = 30);
if (!fp) {
echo $errstr ($errno)\n;
} else {
// send request
fputs($fp, POST $path HTTP/1.1\r\n);
fputs($fp, Host: $host\r\n);
fputs($fp, Content-type: application/x-www-form-urlencoded\r\n);
fputs($fp, Content-length: .strlen($poststring).\r\n);
fputs($fp, Connection: close\r\n\r\n);
fputs($fp, $poststring . \r\n\r\n);
// get response
while(!feof($fp)) {
*/
todo - do something other than fgets with this.
it would be better to get the whole response then close
fp, then move on to doing what you need - instead of
doing your database stuff right away.
*/
echo fgets($fp, 4096);
}
//close fp - we are done with it
fclose($fp);
}
Cheers,
--
- Martin Norland, Sys Admin / Database / Web Developer, International
Outreach x3257
The opinion(s) contained within this email do not necessarily represent
those of St. Jude Children's Research Hospital.
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DB] grabbing data from other peoples sites
I have searched the archives as it this question has almost assuredly been asked countless times. Unfortunately I have no idea what sort of keyword or parameter is applicable for this topic. History Here is a brief history of the problem we are trying to solve. Feel free to skip this section if you only want to see what this has to do with PHP. We need to correlate residents of our county to their local fire departments. We receive about 10 calls a day asking this very question and someone in our office has to literally pick up a map and see find where the little lines for the fire department end. The process takes about 10 minutes where it should only take seconds. We want citizens of Harris County to enter this information into our web forms instead of calling us. Our county tax auditors have already done this through a project that costed millions of taxpayers dollars. The information is protected and available under the Open Records Act of 2000. Unfortunately the Open Records Act does NOT require any government agency to provide their information in electronic form or some sort of database dump. We have requested ALL data about EVERYTHING they have concerning ANY fire department at ANY location (which is our right and privilege). The handed us a huge stack of paper (not something we can import) with a big invoice (both of which we happily declined). The county auditor's website uses ASP and MS SQL Server to allow people to enter their address and find out this information. Unfortunately the information only makes sense to people who understand the coding scheme! What this has to do with PHP Is there a way to write some sort of PHP script that accesses to the county auditors website, automatically enters the info into their form, grabs whatever information is outputted, imports the data into our SQL Server, and repeats the process for all addresses in our database? I will of course research the legality of this before we implement the script. I just wanted to know if this was even feasible before we meet about this. I don't even know where we should start. - Matthew
Re: [PHP-DB] grabbing data from other peoples sites
Perry, Matthew (Fire Marshal's Office) wrote: [snip] What this has to do with PHP Is there a way to write some sort of PHP script that accesses to the county auditors website, automatically enters the info into their form, grabs whatever information is outputted, imports the data into our SQL Server, and repeats the process for all addresses in our database? I will of course research the legality of this before we implement the script. I just wanted to know if this was even feasible before we meet about this. I don't even know where we should start. Sure, you're just issuing POST requests with data from your database, and parsing the output - and entering that extracted data back into your database again. A better question, however, would be how are you planning to handle this once you are ready to run it - are you just storing the address at each corner of your 'covered regions'? You'll need some kind of fuzzy lookup to be able to do all this. Maybe a better solution would be to just link directly to the county auditors website with instructions, allowing the users to do their queries as they're needed? Even if you have every current covered address on file, people may mistype things - new buildings are created - roads are renamed, etc. Someone should contact maps.google.com and see about adding this sort of information to that :) Cheers, -- - Martin Norland, Sys Admin / Database / Web Developer, International Outreach x3257 The opinion(s) contained within this email do not necessarily represent those of St. Jude Children's Research Hospital. -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
