Re: Getting data from external URL
Steve Reppucci [EMAIL PROTECTED] writes: Just a word of warning: LWP::Simple doesn't follow redirects (at least, the last I checked, not sure if it's been changed in the 3 or 4 months since I've last used it...), If it does not follow redirects then that is a bug. Do you have a test case? Not much has changed in the last 3 or 4 months either. Regards, Gisle
Re: Getting data from external URL
Hmmm
Looking at _trivial_http_get:
if ($code =~ /^30[1237]/ $buf =~ /\012Location:\s*(\S+)/) {
# redirect
So it certainly seems like it's *trying to handle it.
As I recall (it was a late night when I had an application that wasn't
working), I had single stepped down into the guts of LWP::Simple and
realized that it was returning a failure indicator when encountering a 302
status. I had assumed that this was intended behavior, but now that I
look at the pod of what we've currently got installed (1.32), it sure
seems like it should work.
I'll look into this a bit to see if I can recreate it, but for now, let's
chalk it up to either (1) something that's been fixed since the version
that I was using at the time, or (2) I'm just out of my head.
More likely the latter...
Sorry for the confusion.
Steve
On 29 Aug 2000, Gisle Aas wrote:
Steve Reppucci [EMAIL PROTECTED] writes:
Just a word of warning: LWP::Simple doesn't follow redirects (at least,
the last I checked, not sure if it's been changed in the 3 or 4
months since I've last used it...),
If it does not follow redirects then that is a bug. Do you have a
test case?
Not much has changed in the last 3 or 4 months either.
Regards,
Gisle
=-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-=
Steve Reppucci [EMAIL PROTECTED] |
Logical Choice Software http://logsoft.com/ |
508/958-0183 Be Open |
Re: Getting data from external URL
Just a word of warning: LWP::Simple doesn't follow redirects (at least,
the last I checked, not sure if it's been changed in the 3 or 4
months since I've last used it...), so you need to be certain that you're
using it in a context where you're fetching something that won't return a
redirect.
HTH...
On Sat, 26 Aug 2000, Stas Bekman wrote:
On Sat, 26 Aug 2000, Rodney Broom wrote:
OK, lots of banter...
Hey V, if you are on a *NIX system, then this is a fast way:
open U, "lynx -source www.some.url.dom |";
$data = join '', U;
There, you're finished. Admittedly, this isn't terribly efficiant, but it works
just fine and has short devel time.
This one is much more efficient and requires even less coding:
use LWP::Simple;
$content = get("http://www.sn.no/")
And it doesn't require you to be on any particular OS, as far as I know.
see perldoc LWP::Simple and as advised by many others LWP::UserAgent for
more advanced uses.
=-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-=
Steve Reppucci [EMAIL PROTECTED] |
Logical Choice Software http://logsoft.com/ |
508/958-0183 Be Open |
Re: Getting data from external URL
On Mon, 28 Aug 2000, Steve Reppucci wrote:
Just a word of warning: LWP::Simple doesn't follow redirects (at least,
the last I checked, not sure if it's been changed in the 3 or 4
months since I've last used it...), so you need to be certain that you're
using it in a context where you're fetching something that won't return a
redirect.
That's why I've mentioned this :)
LWP::UserAgent for more advanced uses.
HTH...
On Sat, 26 Aug 2000, Stas Bekman wrote:
On Sat, 26 Aug 2000, Rodney Broom wrote:
OK, lots of banter...
Hey V, if you are on a *NIX system, then this is a fast way:
open U, "lynx -source www.some.url.dom |";
$data = join '', U;
There, you're finished. Admittedly, this isn't terribly efficiant, but it works
just fine and has short devel time.
This one is much more efficient and requires even less coding:
use LWP::Simple;
$content = get("http://www.sn.no/")
And it doesn't require you to be on any particular OS, as far as I know.
see perldoc LWP::Simple and as advised by many others LWP::UserAgent for
more advanced uses.
=-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-=
Steve Reppucci [EMAIL PROTECTED] |
Logical Choice Software http://logsoft.com/ |
508/958-0183 Be Open |
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com
http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Getting data from external URL
I just wrote a function yesterday that uses IO::Socket to interact with an
outside CGI program. It appears to work the way I want but this is my first
foray into writing TCP client code.
It's basically this:
sub tcp_client {
my ( $rhost, $rport, $query_string ) = @_;
my $socket = IO::Socket::INET-new( PerrAddr = $rhost,
PeerPort = $rport,
Proto= "tcp",
Type = SOCK_STREAM)
or die "Couldn't connect to $rhost:$rport : $!\n";
print $socket "GET /programname?$query_string\n";
my $response;
while ( $socket ) { $response .= $_ }
close($socket);
return \$answer;
}
--Jeff
At 06:58 AM 8/28/00 -0400, Steve Reppucci wrote:
Just a word of warning: LWP::Simple doesn't follow redirects (at least,
the last I checked, not sure if it's been changed in the 3 or 4
months since I've last used it...), so you need to be certain that you're
using it in a context where you're fetching something that won't return a
redirect.
HTH...
On Sat, 26 Aug 2000, Stas Bekman wrote:
On Sat, 26 Aug 2000, Rodney Broom wrote:
OK, lots of banter...
Hey V, if you are on a *NIX system, then this is a fast way:
open U, "lynx -source www.some.url.dom |";
$data = join '', U;
There, you're finished. Admittedly, this isn't terribly efficiant,
but it works
just fine and has short devel time.
This one is much more efficient and requires even less coding:
use LWP::Simple;
$content = get("http://www.sn.no/")
And it doesn't require you to be on any particular OS, as far as I know.
see perldoc LWP::Simple and as advised by many others LWP::UserAgent for
more advanced uses.
=-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-=
Steve Reppucci [EMAIL PROTECTED] |
Logical Choice Software http://logsoft.com/ |
508/958-0183 Be Open |
Jeff Beard
__
Web:www.cyberxape.com
Email: jeff at cyberxape dot com
Location: Boulder, Colorado, USA
Re: Getting data from external URL
On Mon, 28 Aug 2000, Jeff Beard wrote:
I just wrote a function yesterday that uses IO::Socket to interact with an
outside CGI program. It appears to work the way I want but this is my first
foray into writing TCP client code.
It's basically this:
sub tcp_client {
my ( $rhost, $rport, $query_string ) = @_;
my $socket = IO::Socket::INET-new( PerrAddr = $rhost,
PeerPort = $rport,
Proto= "tcp",
Type = SOCK_STREAM)
or die "Couldn't connect to $rhost:$rport : $!\n";
print $socket "GET /programname?$query_string\n";
At the very least send a HTTP/1.x part and a Host: header. Otherwise
virtual hosts are hosed.
--
Matt/
Fastnet Software Ltd. High Performance Web Specialists
Providing mod_perl, XML, Sybase and Oracle solutions
Email for training and consultancy availability.
http://sergeant.org | AxKit: http://axkit.org
Re: Getting data from external URL
SB This one is much more efficient and requires even less coding:
SB use LWP::Simple;
SB $content = get("http://www.sn.no/")
Even better, thanks Stas.
Re: Getting data from external URL
Hello,
Thanks everyone for such a simple solutions.
I think I will use followig method as it seems much simpler.
Thanks
Vijay
- Original Message -
From: "Rodney Broom" [EMAIL PROTECTED]
To: "Stas Bekman" [EMAIL PROTECTED]
Cc: "Vijay" [EMAIL PROTECTED]; "mod_perl Maillinglist" [EMAIL PROTECTED]
Sent: Saturday, August 26, 2000 10:24 AM
Subject: Re: Getting data from external URL
SB This one is much more efficient and requires even less coding:
SB use LWP::Simple;
SB $content = get("http://www.sn.no/")
Even better, thanks Stas.
RE: Getting data from external URL
Hello Vijay,
You can make simple HTTP GET or POST requests
using the LWP and HTTP modules. I recommend
that you read the ActivePerl help files.
I have attached the one you really need but
it probably won't make sense unless you read
up on LWP.
# A small sample follows:
use LWP::UserAgent;
# use HTTP::Request::Common qw(POST);
$ua = new LWP::UserAgent;
$ua-agent("Netscape 3.0");
$page_url =
'http://www.1freestuff.com/cgi-bin/topsites/topsites.cgi?puzzler';
$response = get($page_url); # Do a GET.
# print $response-as_string;
# Get the NEW form number:
$content = $response-content;
$content =~ m/(value=")(\d.*)(")/ig;
$number = $2;
print "The magic number is: " . $number;
sub get {
my $page_url = $_[0];
my $request = new HTTP::Request(GET, "$page_url");
my $response = $ua-request($request);
return $response;
};
-Original Message-
From: Vijay [mailto:[EMAIL PROTECTED]]
Sent: Friday, August 25, 2000 11:15 PM
To: [EMAIL PROTECTED]; mod_perl Maillinglist
Subject: Getting data from external URL
Hello,
I want to get data from an external url in my perl program (either thru
Embperl Execute or directly from perl). What I need is like this.
There is a URL which gives some information in text format. I want to get
that into a variable or file using perl and using my own html templates, I
want to show that data.
If anyone has done something like this, please let me know.
Thanks
Vijay
Title: HTTP::Request::Common - Construct common HTTP::Request objects
HTTP::Request::Common - Construct common
HTTP::Request objects
NAME
SUPPORTED
PLATFORMS
SYNOPSIS
DESCRIPTION
SEE
ALSO
COPYRIGHT
NAME
HTTP::Request::Common - Construct common HTTP::Request objects
SUPPORTED PLATFORMS
Linux
Solaris
Windows
SYNOPSIS use HTTP::Request::Common;
$ua = LWP::UserAgent-new;
$ua-request(GET 'http://www.sn.no/');
$ua-request(POST 'http://somewhere/foo', [foo = bar, bar = foo]);
DESCRIPTION
This module provide functions that return newly created HTTP::Request
objects. These functions are usually more convenient to use than the standard
HTTP::Request constructor for these common requests. The following functions are
provided.
GET
$url, Header = Value,...
The GET() function returns a HTTP::Request object initialized
with the GET method and the specified URL. Without additional arguments it is
exactly equivalent to the following call HTTP::Request-new(GET = $url)
but is less cluttered. It also reads better when used together with the
LWP::UserAgent-request() method: my $ua = new LWP::UserAgent;
my $res = $ua-request(GET 'http://www.sn.no/')
if ($res-is_success) { ...
You can also initialize header values in the request by specifying some
key/value pairs as optional arguments. For instance: $ua-request(GET 'http://www.sn.no/',
If_Match = 'foo',
From = '[EMAIL PROTECTED]',
);
A header key called 'Content' is special and when seen the value will
initialize the content part of the request instead of setting a header.
HEAD $url,
[Header = Value,...]
Like GET() but the method in the request is HEAD.
PUT $url, [Header
= Value,...]
Like GET() but the method in the request is PUT.
POST $url,
[$form_ref], [Header = Value,...]
This works mostly like GET() with POST as the method, but
this function also takes a second optional array or hash reference parameter
($form_ref). This argument can be used to pass key/value pairs for the form
content. By default we will initialize a request using the
application/x-www-form-urlencoded content type. This means that
you can emulate a HTML form POSTing like this: POST 'http://www.perl.org/survey.cgi',
[ name = 'Gisle Aas',
email = '[EMAIL PROTECTED]',
gender = 'M',
born = '1964',
perc = '3%',
];
This will create a HTTP::Request object that looks like this: POST http://www.perl.org/survey.cgi
Content-Length: 66
Content-Type: application/x-www-form-urlencoded name=Gisle%20Aasemail=gisle%40aas.nogender=Mborn=1964perc=3%25
The POST method also supports the multipart/form-data content
used for Form-based File Upload as specified in RFC 1867. You trigger
this content format by specifying a content type of 'form-data'
as one of the request headers. If one of the values in the $form_ref is an
array reference, then it is treated as a file part specification with the
following interpretation: [ $file, $filename, Header = Value... ]
The first value in the array ($file) is the name of a file to open. This
file will be read and its content placed in the request. The routine will
croak if the file can't be opened. Use an undef
as $file value if you want to specify the content directly. The $filename is
the filename to report in the request. If this value is undefined, then
Re: Getting data from external URL
OK, lots of banter... Hey V, if you are on a *NIX system, then this is a fast way: open U, "lynx -source www.some.url.dom |"; $data = join '', U; There, you're finished. Admittedly, this isn't terribly efficiant, but it works just fine and has short devel time. Rodney Broom
Re: Getting data from external URL
On Sat, 26 Aug 2000, Rodney Broom wrote:
OK, lots of banter...
Hey V, if you are on a *NIX system, then this is a fast way:
open U, "lynx -source www.some.url.dom |";
$data = join '', U;
There, you're finished. Admittedly, this isn't terribly efficiant, but it works
just fine and has short devel time.
This one is much more efficient and requires even less coding:
use LWP::Simple;
$content = get("http://www.sn.no/")
And it doesn't require you to be on any particular OS, as far as I know.
see perldoc LWP::Simple and as advised by many others LWP::UserAgent for
more advanced uses.
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com
http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Getting data from external URL
Hello, I want to get data from an external url in my perl program (either thru Embperl Execute or directly from perl). What I need is like this. There is a URL which gives some information in text format. I want to get that into a variable or file using perl and using my own html templates, I want to show that data. If anyone has done something like this, please let me know. HTML::Embperl::ProxyInput maybe (ab)used to fetch the data from the remote host Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
Re: Getting data from external URL
If you're in a modperl enviroment, and don't use CGI.pm, try using Apache::Request (I'm not a fan of using Apache.pm for too much). Has anyone ever considered making a wrapper for all the modules which get back data from the request (roll Apache::Request, Apache::Cookie, etc, into one)? Seems trivial. Yann Vijay wrote: Hello, I want to get data from an external url in my perl program (either thru Embperl Execute or directly from perl). What I need is like this. There is a URL which gives some information in text format. I want to get that into a variable or file using perl and using my own html templates, I want to show that data. If anyone has done something like this, please let me know. Thanks Vijay -- Yann Ramin [EMAIL PROTECTED] Atrus Trivalie Productions www.redshift.com/~yramin Monterey High ITwww.montereyhigh.com ICQ 46805627 AIM oddatrus Marina, CA IRM Developer Network Toaster Developer SNTS Developer KLevel Developer (yes, this .signature is way too big) "All cats die. Socrates is dead. Therefore Socrates is a cat." - The Logician THE STORY OF CREATION In the beginning there was data. The data was without form and null, and darkness was upon the face of the console; and the Spirit of IBM was moving over the face of the market. And DEC said, "Let there be registers"; and there were registers. And DEC saw that they carried; and DEC seperated the data from the instructions. DEC called the data Stack, and the instructions they called Code. And there was evening and there was a maorning, one interrupt... -- Rico Tudor William Safire's Rules for Writers: Remembe
Re: Getting data from external URL
Apache::Request simply maps the Apache object into an Apache::Request object (I know that sounds like double-speak, but it's late), and adds some extra methods. I don't think that's what Vijay's original question is all about. What's needed is an HTTP::Request object. Look at LWP::UserAgent which brings together HTTP::Request and HTTP::Response objects and you also should grab HTTP::Headers. Like I said, it's late. I think that whole collection ball of wax is in libwww-perl available on CPAN. Other than the parsing and redisplaying that Vijay asked about, that's pretty much what a proxy server does, and there's a pretty detailed example or two in the Eagle book. As far as the parsing that response data, a mixture of good ole perl technique and HTML::Parser should be sufficient. -- Rob --On 08/25/00 21:33:14 -0700 Yann Ramin [EMAIL PROTECTED] wrote: If you're in a modperl enviroment, and don't use CGI.pm, try using Apache::Request (I'm not a fan of using Apache.pm for too much). Has anyone ever considered making a wrapper for all the modules which get back data from the request (roll Apache::Request, Apache::Cookie, etc, into one)? Seems trivial. Yann Vijay wrote: Hello, I want to get data from an external url in my perl program (either thru Embperl Execute or directly from perl). What I need is like this. There is a URL which gives some information in text format. I want to get that into a variable or file using perl and using my own html templates, I want to show that data. If anyone has done something like this, please let me know. Thanks Vijay -- Yann Ramin[EMAIL PROTECTED] Atrus Trivalie Productionswww.redshift.com/~yramin Monterey High IT www.montereyhigh.com ICQ 46805627 AIM oddatrus Marina, CA IRM Developer Network Toaster Developer SNTS Developer KLevel Developer (yes, this .signature is way too big) "All cats die. Socrates is dead. Therefore Socrates is a cat." - The Logician THE STORY OF CREATION In the beginning there was data. The data was without form and null, and darkness was upon the face of the console; and the Spirit of IBM was moving over the face of the market. And DEC said, "Let there be registers"; and there were registers. And DEC saw that they carried; and DEC seperated the data from the instructions. DEC called the data Stack, and the instructions they called Code. And there was evening and there was a maorning, one interrupt... -- Rico Tudor William Safire's Rules for Writers: Remembe _ _ _ _ __ _ _ _ _ /\_\_\_\_\/\_\ /\_\_\_\_\_\ /\/_/_/_/_/ /\/_/ \/_/_/_/_/_/ QUIDQUID LATINE DICTUM SIT, /\/_/__\/_/ __/\/_//\/_/ PROFUNDUM VIDITUR /\/_/_/_/_/ /\_\ /\/_//\/_/ /\/_/ \/_/ /\/_/_/\/_//\/_/ (Whatever is said in Latin \/_/ \/_/ \/_/_/_/_/ \/_/ appears profound) Rob Tanner McMinnville, Oregon [EMAIL PROTECTED]
