Re: [PHP] PHP Data Mining/Data Scraping
On Wed, May 23, 2007 5:25 am, Shannon Whitty wrote: THanks, I will have far to much data to append to a GET request so a POST is the best option I think. Also consider RSS, REST, RPC and SOAP as long-term solutions with increasing overhead/functionality. These have the advantage of having standards and tons of toolkits around instead of a custom-brewed client-server hack you may come up with. This all presumes you have control over the success outputting server... If not, you're stuck with scraping what they send, unless they offer RSS, REST, RPC or SOAP. -- Some people have a gift link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
On Wed, May 23, 2007 5:37 am, Shannon Whitty wrote: THanks, That's what I thought... I really only need to check for one value = success - anything else should redirect back to the form entry page and display whatever result has been returned from the remote site. cUrl should be able to differentiate between a connection and an application failure shouldn't it? The way I see it: 1: If I can connect to port80 THEN continue ELSE goto:6 2: If I get a response THEN continue ELSE goto:6 3: If response contains result tags THEN continue ELSE goto:6 4: If result tags contains success THEN goto:5 ELSE goto:7 5: Display success and finish 6: Display connection error and resort to email and finish 7: Reload form and display error returned within result tags This looks right, though I won't swear to it with all the goto:s... I'm more of an if/else kinda guy... :-) -- Some people have a gift link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
THanks,
I will have far to much data to append to a GET request so a POST is the
best option I think.
Myron Turner [EMAIL PROTECTED] wrote in message
news:[EMAIL PROTECTED]
On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote:
I'm looking for a piece of software or coding that will let me post a
form
to another URL, accept the response, search it for a specific
success
string and then let me continue processing the rest of my program.
http://php.net/curl
I want to accept queries on behalf of my supplier, forward it to them
behind
the scenes, accept their response and display it within my website.
Has anyone had any experience with this? Is there a simple, basic
utility
to let me do this?
I was kind of hoping I could avoid developing it myself.
As I understand this, you want to create a web page of your own which
accepts requests for customers who are going to order products from your
supplier. You want to have a form on your page which accepts their
requests, then forward the form data on to your supplier's web site, where
presumably it will be processed. Then you want to retrieve the response
from your supplier's page, and display the result on your own web page.
You suggest that the response string for success is relatively stable
and that this string is this what you want to search for in the response.
This doesn't sound like a very complicated problem. You can do this
either using Ajax or not. The basic solution is the same. You have a
script on the server which accepts the form data from your page and
re-sends it to the supplier's site. If your supplier's site accepts form
data using GET, then you can simply create a url with the form data
attached in a query string:
http://my.supplier.com?fdata_1=data1fdata_2=data2
Send this url to your suppler using file_get_contents:
$return_string =
file_get_contents(http://my.supplier.com?fdata_1=data1fdata_2=data2;);
This will return the html file as a string which you can then parse with
preg_match() for the 'success' string.
The problem is more involved if your supplier doesn't accept GET but only
accepts POST. Then you have to use either curl or fsockopen to post your
data. I've tested the following fockopen script and it worked for me:
?php
$fp = fsockopen(my.supplier.com, 80, $errno, $errstr, 30);
if (!$fp) {
echo $errstr ($errno)br /\n;
} else {
$out = POST http://my.supplier.com/form_page.html / HTTP/1.1\r\n;
$out .= Host: my.supplier.com\r\n;
$post = form_data_1=data_1formdata_2=data_2;
$len = strlen($post);
$post .= \r\n;
$out .=Content-Length: $len\r\n; $out .= Connection:
Close\r\n\r\n;
$out .= $post;
fwrite($fp, $out);
$result= ;
while (!feof($fp)) {
$result .= fgets($fp, 128);
}
fclose($fp);
echo $result;
}
?
You have to adhere to the above sequence. The posted data comes last and
it is preceded by a content-length header which tells the receiving server
how long the posted data is. The returned result is the html page
returned from your posted request.
--
_
Myron Turner
http://www.room535.org
http://www.bstatzero.org
http://www.mturner.org/XML_PullParser/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
THanks,
I will have far to much data to append to a GET request so a POST is the
best option I think.
Myron Turner [EMAIL PROTECTED] wrote in message
news:[EMAIL PROTECTED]
On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote:
I'm looking for a piece of software or coding that will let me post a
form
to another URL, accept the response, search it for a specific
success
string and then let me continue processing the rest of my program.
http://php.net/curl
I want to accept queries on behalf of my supplier, forward it to them
behind
the scenes, accept their response and display it within my website.
Has anyone had any experience with this? Is there a simple, basic
utility
to let me do this?
I was kind of hoping I could avoid developing it myself.
As I understand this, you want to create a web page of your own which
accepts requests for customers who are going to order products from your
supplier. You want to have a form on your page which accepts their
requests, then forward the form data on to your supplier's web site, where
presumably it will be processed. Then you want to retrieve the response
from your supplier's page, and display the result on your own web page.
You suggest that the response string for success is relatively stable
and that this string is this what you want to search for in the response.
This doesn't sound like a very complicated problem. You can do this
either using Ajax or not. The basic solution is the same. You have a
script on the server which accepts the form data from your page and
re-sends it to the supplier's site. If your supplier's site accepts form
data using GET, then you can simply create a url with the form data
attached in a query string:
http://my.supplier.com?fdata_1=data1fdata_2=data2
Send this url to your suppler using file_get_contents:
$return_string =
file_get_contents(http://my.supplier.com?fdata_1=data1fdata_2=data2;);
This will return the html file as a string which you can then parse with
preg_match() for the 'success' string.
The problem is more involved if your supplier doesn't accept GET but only
accepts POST. Then you have to use either curl or fsockopen to post your
data. I've tested the following fockopen script and it worked for me:
?php
$fp = fsockopen(my.supplier.com, 80, $errno, $errstr, 30);
if (!$fp) {
echo $errstr ($errno)br /\n;
} else {
$out = POST http://my.supplier.com/form_page.html / HTTP/1.1\r\n;
$out .= Host: my.supplier.com\r\n;
$post = form_data_1=data_1formdata_2=data_2;
$len = strlen($post);
$post .= \r\n;
$out .=Content-Length: $len\r\n; $out .= Connection:
Close\r\n\r\n;
$out .= $post;
fwrite($fp, $out);
$result= ;
while (!feof($fp)) {
$result .= fgets($fp, 128);
}
fclose($fp);
echo $result;
}
?
You have to adhere to the above sequence. The posted data comes last and
it is preceded by a content-length header which tells the receiving server
how long the posted data is. The returned result is the html page
returned from your posted request.
--
_
Myron Turner
http://www.room535.org
http://www.bstatzero.org
http://www.mturner.org/XML_PullParser/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
THanks,
I will have far to much data to append to a GET request so a POST is the
best option I think.
Myron Turner [EMAIL PROTECTED] wrote in message
news:[EMAIL PROTECTED]
On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote:
I'm looking for a piece of software or coding that will let me post a
form
to another URL, accept the response, search it for a specific
success
string and then let me continue processing the rest of my program.
http://php.net/curl
I want to accept queries on behalf of my supplier, forward it to them
behind
the scenes, accept their response and display it within my website.
Has anyone had any experience with this? Is there a simple, basic
utility
to let me do this?
I was kind of hoping I could avoid developing it myself.
As I understand this, you want to create a web page of your own which
accepts requests for customers who are going to order products from your
supplier. You want to have a form on your page which accepts their
requests, then forward the form data on to your supplier's web site, where
presumably it will be processed. Then you want to retrieve the response
from your supplier's page, and display the result on your own web page.
You suggest that the response string for success is relatively stable
and that this string is this what you want to search for in the response.
This doesn't sound like a very complicated problem. You can do this
either using Ajax or not. The basic solution is the same. You have a
script on the server which accepts the form data from your page and
re-sends it to the supplier's site. If your supplier's site accepts form
data using GET, then you can simply create a url with the form data
attached in a query string:
http://my.supplier.com?fdata_1=data1fdata_2=data2
Send this url to your suppler using file_get_contents:
$return_string =
file_get_contents(http://my.supplier.com?fdata_1=data1fdata_2=data2;);
This will return the html file as a string which you can then parse with
preg_match() for the 'success' string.
The problem is more involved if your supplier doesn't accept GET but only
accepts POST. Then you have to use either curl or fsockopen to post your
data. I've tested the following fockopen script and it worked for me:
?php
$fp = fsockopen(my.supplier.com, 80, $errno, $errstr, 30);
if (!$fp) {
echo $errstr ($errno)br /\n;
} else {
$out = POST http://my.supplier.com/form_page.html / HTTP/1.1\r\n;
$out .= Host: my.supplier.com\r\n;
$post = form_data_1=data_1formdata_2=data_2;
$len = strlen($post);
$post .= \r\n;
$out .=Content-Length: $len\r\n; $out .= Connection:
Close\r\n\r\n;
$out .= $post;
fwrite($fp, $out);
$result= ;
while (!feof($fp)) {
$result .= fgets($fp, 128);
}
fclose($fp);
echo $result;
}
?
You have to adhere to the above sequence. The posted data comes last and
it is preceded by a content-length header which tells the receiving server
how long the posted data is. The returned result is the html page
returned from your posted request.
--
_
Myron Turner
http://www.room535.org
http://www.bstatzero.org
http://www.mturner.org/XML_PullParser/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
THanks, That's what I thought... I really only need to check for one value = success - anything else should redirect back to the form entry page and display whatever result has been returned from the remote site. cUrl should be able to differentiate between a connection and an application failure shouldn't it? The way I see it: 1: If I can connect to port80 THEN continue ELSE goto:6 2: If I get a response THEN continue ELSE goto:6 3: If response contains result tags THEN continue ELSE goto:6 4: If result tags contains success THEN goto:5 ELSE goto:7 5: Display success and finish 6: Display connection error and resort to email and finish 7: Reload form and display error returned within result tags thanks Shannon Richard Lynch [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote: I'm looking for a piece of software or coding that will let me post a form to another URL, accept the response, search it for a specific success string and then let me continue processing the rest of my program. http://php.net/curl I want to accept queries on behalf of my supplier, forward it to them behind the scenes, accept their response and display it within my website. Has anyone had any experience with this? Is there a simple, basic utility to let me do this? I was kind of hoping I could avoid developing it myself. Search for PHP curl examples online, and you should find the code simple enough to copy/paste and alter to taste... It won't be quite as easy as install forum X but it shouldn't kill you either... The tricky bit is to figure out what to do when your result from the supplier is not success nor failure but their site has gone down and you've got some weird answer you've never seen before... Or when they alter their web application and then yours breaks because of it... You'll end up taking a simple 5-line program and adding about 50 lines of what if error handling if you do this right... Or leave it at 5 lines and pray nothing goes wrong :-) -- Some people have a gift link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote:
I'm looking for a piece of software or coding that will let me post a
form
to another URL, accept the response, search it for a specific
success
string and then let me continue processing the rest of my program.
http://php.net/curl
I want to accept queries on behalf of my supplier, forward it to them
behind
the scenes, accept their response and display it within my website.
Has anyone had any experience with this? Is there a simple, basic
utility
to let me do this?
I was kind of hoping I could avoid developing it myself.
As I understand this, you want to create a web page of your own which
accepts requests for customers who are going to order products from your
supplier. You want to have a form on your page which accepts their
requests, then forward the form data on to your supplier's web site,
where presumably it will be processed. Then you want to retrieve the
response from your supplier's page, and display the result on your own
web page. You suggest that the response string for success is
relatively stable and that this string is this what you want to search
for in the response.
This doesn't sound like a very complicated problem. You can do this
either using Ajax or not. The basic solution is the same. You have a
script on the server which accepts the form data from your page and
re-sends it to the supplier's site. If your supplier's site accepts
form data using GET, then you can simply create a url with the form data
attached in a query string:
http://my.supplier.com?fdata_1=data1fdata_2=data2
Send this url to your suppler using file_get_contents:
$return_string =
file_get_contents(http://my.supplier.com?fdata_1=data1fdata_2=data2;);
This will return the html file as a string which you can then parse with
preg_match() for the 'success' string.
The problem is more involved if your supplier doesn't accept GET but
only accepts POST. Then you have to use either curl or fsockopen to
post your data. I've tested the following fockopen script and it
worked for me:
?php
$fp = fsockopen(my.supplier.com, 80, $errno, $errstr, 30);
if (!$fp) {
echo $errstr ($errno)br /\n;
} else {
$out = POST http://my.supplier.com/form_page.html / HTTP/1.1\r\n;
$out .= Host: my.supplier.com\r\n;
$post = form_data_1=data_1formdata_2=data_2;
$len = strlen($post);
$post .= \r\n;
$out .=Content-Length: $len\r\n;
$out .= Connection: Close\r\n\r\n;
$out .= $post;
fwrite($fp, $out);
$result= ;
while (!feof($fp)) {
$result .= fgets($fp, 128);
}
fclose($fp);
echo $result;
}
?
You have to adhere to the above sequence. The posted data comes last
and it is preceded by a content-length header which tells the receiving
server how long the posted data is. The returned result is the html
page returned from your posted request.
--
_
Myron Turner
http://www.room535.org
http://www.bstatzero.org
http://www.mturner.org/XML_PullParser/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
On 5/22/07, Myron Turner [EMAIL PROTECTED] wrote:
On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote:
I'm looking for a piece of software or coding that will let me post a
form
to another URL, accept the response, search it for a specific
success
string and then let me continue processing the rest of my program.
http://php.net/curl
I want to accept queries on behalf of my supplier, forward it to them
behind
the scenes, accept their response and display it within my website.
Has anyone had any experience with this? Is there a simple, basic
utility
to let me do this?
I was kind of hoping I could avoid developing it myself.
As I understand this, you want to create a web page of your own which
accepts requests for customers who are going to order products from your
supplier. You want to have a form on your page which accepts their
requests, then forward the form data on to your supplier's web site,
where presumably it will be processed. Then you want to retrieve the
response from your supplier's page, and display the result on your own
web page. You suggest that the response string for success is
relatively stable and that this string is this what you want to search
for in the response.
This doesn't sound like a very complicated problem. You can do this
either using Ajax or not. The basic solution is the same. You have a
script on the server which accepts the form data from your page and
re-sends it to the supplier's site. If your supplier's site accepts
form data using GET, then you can simply create a url with the form data
attached in a query string:
http://my.supplier.com?fdata_1=data1fdata_2=data2
Send this url to your suppler using file_get_contents:
$return_string =
file_get_contents(http://my.supplier.com?fdata_1=data1fdata_2=data2;);
This will return the html file as a string which you can then parse with
preg_match() for the 'success' string.
The problem is more involved if your supplier doesn't accept GET but
only accepts POST. Then you have to use either curl or fsockopen to
post your data. I've tested the following fockopen script and it
worked for me:
?php
$fp = fsockopen(my.supplier.com, 80, $errno, $errstr, 30);
if (!$fp) {
echo $errstr ($errno)br /\n;
} else {
$out = POST http://my.supplier.com/form_page.html / HTTP/1.1\r\n;
$out .= Host: my.supplier.com\r\n;
$post = form_data_1=data_1formdata_2=data_2;
$len = strlen($post);
$post .= \r\n;
$out .=Content-Length: $len\r\n;
$out .= Connection: Close\r\n\r\n;
$out .= $post;
fwrite($fp, $out);
$result= ;
while (!feof($fp)) {
$result .= fgets($fp, 128);
}
fclose($fp);
echo $result;
}
?
You have to adhere to the above sequence. The posted data comes last
and it is preceded by a content-length header which tells the receiving
server how long the posted data is. The returned result is the html
page returned from your posted request.
It's a nice script, but you're way better off using cURL, you can
simply pass a PHP array as POST form data.
Tijnema
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
Tijnema wrote:
On 5/22/07, Myron Turner [EMAIL PROTECTED] wrote:
On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote:
I'm looking for a piece of software or coding that will let me post a
form
to another URL, accept the response, search it for a specific
success
string and then let me continue processing the rest of my program.
http://php.net/curl
I want to accept queries on behalf of my supplier, forward it to them
behind
the scenes, accept their response and display it within my website.
Has anyone had any experience with this? Is there a simple, basic
utility
to let me do this?
I was kind of hoping I could avoid developing it myself.
As I understand this, you want to create a web page of your own which
accepts requests for customers who are going to order products from your
supplier. You want to have a form on your page which accepts their
requests, then forward the form data on to your supplier's web site,
where presumably it will be processed. Then you want to retrieve the
response from your supplier's page, and display the result on your own
web page. You suggest that the response string for success is
relatively stable and that this string is this what you want to search
for in the response.
This doesn't sound like a very complicated problem. You can do this
either using Ajax or not. The basic solution is the same. You have a
script on the server which accepts the form data from your page and
re-sends it to the supplier's site. If your supplier's site accepts
form data using GET, then you can simply create a url with the form data
attached in a query string:
http://my.supplier.com?fdata_1=data1fdata_2=data2
Send this url to your suppler using file_get_contents:
$return_string =
file_get_contents(http://my.supplier.com?fdata_1=data1fdata_2=data2;);
This will return the html file as a string which you can then parse with
preg_match() for the 'success' string.
The problem is more involved if your supplier doesn't accept GET but
only accepts POST. Then you have to use either curl or fsockopen to
post your data. I've tested the following fockopen script and it
worked for me:
?php
$fp = fsockopen(my.supplier.com, 80, $errno, $errstr, 30);
if (!$fp) {
echo $errstr ($errno)br /\n;
} else {
$out = POST http://my.supplier.com/form_page.html / HTTP/1.1\r\n;
$out .= Host: my.supplier.com\r\n;
$post = form_data_1=data_1formdata_2=data_2;
$len = strlen($post);
$post .= \r\n;
$out .=Content-Length: $len\r\n;
$out .= Connection: Close\r\n\r\n;
$out .= $post;
fwrite($fp, $out);
$result= ;
while (!feof($fp)) {
$result .= fgets($fp, 128);
}
fclose($fp);
echo $result;
}
?
You have to adhere to the above sequence. The posted data comes last
and it is preceded by a content-length header which tells the receiving
server how long the posted data is. The returned result is the html
page returned from your posted request.
It's a nice script, but you're way better off using cURL, you can
simply pass a PHP array as POST form data.
Tijnema
Thanks. That's good to know. My experience with these kinds of things
is with Perl, with which I've done an awful lot of screen-scraping. But
I haven't had any actual practical experience with cURL. I've looked at
it, but that's all. Also, I was under the impression that cURL is an
extension which is not always installed, whereas fsockopen() is a
built-in. There is one server I sometimes use, a shared server, where it
isn't installed. The other servers I use are all independent machines.
--
_
Myron Turner
http://www.room535.org
http://www.bstatzero.org
http://www.mturner.org/XML_PullParser/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote: I'm looking for a piece of software or coding that will let me post a form to another URL, accept the response, search it for a specific success string and then let me continue processing the rest of my program. http://php.net/curl I want to accept queries on behalf of my supplier, forward it to them behind the scenes, accept their response and display it within my website. Has anyone had any experience with this? Is there a simple, basic utility to let me do this? I was kind of hoping I could avoid developing it myself. Search for PHP curl examples online, and you should find the code simple enough to copy/paste and alter to taste... It won't be quite as easy as install forum X but it shouldn't kill you either... The tricky bit is to figure out what to do when your result from the supplier is not success nor failure but their site has gone down and you've got some weird answer you've never seen before... Or when they alter their web application and then yours breaks because of it... You'll end up taking a simple 5-line program and adding about 50 lines of what if error handling if you do this right... Or leave it at 5 lines and pray nothing goes wrong :-) -- Some people have a gift link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Data Mining/Data Scraping
Shannon Whitty wrote: Hi, I'm looking for a piece of software or coding that will let me post a form to another URL, accept the response, search it for a specific success string and then let me continue processing the rest of my program. I want to accept queries on behalf of my supplier, forward it to them behind the scenes, accept their response and display it within my website. Has anyone had any experience with this? Is there a simple, basic utility to let me do this? I was kind of hoping I could avoid developing it myself. http://php.net/curl -Stut -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
