Re: binary responses via LWP
How are you processing $content?
I tried a modifed version of your code (see below) and I was able to view
the graphic.
If you are getting \012's stripped out, it sounds like Perl is treating
$content as a text file not as a binary. Depending on your OS, Perl will
strip out LF (\012), CR (\013), or do nothing when working with what it
considers text.
Here's the code I tested. Note that I set the filehandle to binary (with
binmode) before writing the contents to disk.
use LWP::UserAgent;
my ($host, $doc)=@ARGV;
my $ua=new LWP::UserAgent;
my $req = new HTTP::Request 'GET', "http://$host$doc";;
my $response = $ua->request($req); # returns a hashref
my $hdrs = $response->headers->as_string();
my $result = $response->status_line;
my $content = $response->content();
print ($hdrs, $result);
open (FILE, ">temp.gif") or die "Could not open output!: $!";
binmode (FILE); # needed for win32
print FILE $content;
close FILE;
On Thu, 25 Apr 2002, Vicki Brown wrote:
> Date: Thu, 25 Apr 2002 22:37:03 -0700
> From: Vicki Brown <[EMAIL PROTECTED]>
> To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]>
> Subject: binary responses via LWP
>
> My send/receive code is working fine (apparently) for HTML and also for text
> (although oddly, text comes back from the remote CGI with a Content-type of
> text/x-usenet-faq which neither I nor my browser had heard of, so it was
> being downloaded instead of displayed; took me ages to figure out).
>
> Anyway, my problem is when the response is binary, e.g. a compressed archive,
> a JPEG, a GIF.
>
> As near as I can determine, I am losing a bunch of ^L (ASCII \012) in the
> response content. Is there something special I need to do to say "it's
> binary, it's not a string, leave it alone"?
>
>
> my ($host, $port, $doc)=@_;
>
> my $ua=new LWP::UserAgent;
> my $req = new HTTP::Request 'GET', "http://$host$doc";;
>
>
> my $response = $ua->request($req); # returns a hashref
>
> my $hdrs = $response->headers->as_string();
> my $result = $response->status_line;
> my $content = $response->content();
>
> return($content, $result, $hdrs);
>
> On the other side, if I run the returned $content through a filter
> $string =~ s/([^a-zA-Z0-9_])/sprintf("\\%03o", ord($1))/eg;
> and then compare the result with the original file (before I ut it on the
> server), the "new" version I got back through LWP is missing a bunch of \012.
>
> If I bypass the LWP gateway and go directly to the remote CGI, the files are
> not mangled.
>
> Is this another simple thing I can't seem to find in the docs?
> --
> - Vicki
>
> Vicki Brown ZZZ Journeyman Sourceror:
> P.O. Box 1269 zz |\ _,,,---,,_Scripts & Philtres
> San Bruno, CA zz /,`.-'`'-. ;-;;,_ Perl, Unix, MacOS
> 94066 USA |,4- ) )-,_. ,\ ( `'-'
> mailto:[EMAIL PROTECTED] '---''(_/--' `-'\_) http://www.cfcl.com/~vlb
>
Re: Help with 403 Access Forbidden
On Fri, 26 Apr 2002, Sean M. Burke wrote: > BTW, since someone asked: I don't think ordering it from Amazon will get it > any slower or faster than from BN.com. Especially since BN.com is (or at least was as of a few months ago) just a different front end for Amazon.com. BN.com farmed out their online operations to Amazon, and they became very tightly integrated, to the point that they almost always had the same prices and availability on books and DVDs. One of my credit cards let me cash in reward points for BN.com gift certificates, and one time while spending one, I miskeyed something and wound up at an Amazon.com error reporting page :-)
Re: Help with 403 Access Forbidden
>This is the coming O'Reilly book I mentioned was available for pre-ordering >on B&N (I don't buy from Amazon :-) Don't let that stop you from reviewing it at Amazon! As far as I'm concerned, Amazon is a communal book rating system that just happens to have a store attached. BTW, since someone asked: I don't think ordering it from Amazon will get it any slower or faster than from BN.com. The book is still in production yet, so don't take either site's time figures literally. (Ditto the page-count; I bet it'll be closer to 300 pages than 400.) -- Sean M. Burkehttp://www.spinn.net/~sburke/
Re: Help with 403 Access Forbidden
At 12:28 -0600 4/26/02, Sean M. Burke wrote: >At 10:30 2002-04-26 -0400, Greg E Tilton wrote: >>(Also, I'm struggling in trying to get a grasp on the materials related to >>LWP (e.g., cookbook). I thank God for documentation like cookbook, but >>I'd like to get a more in-depth understanding of the subject. Is there >>something I can study to get up to speed? Any recommendations would be >>appreciated.) > >I just wrote a book about it all! >I was going to wait until Amazon's record for it is updated, but since you >asked... >http://www.amazon.com/exec/obidos/ASIN/0596001789 This is the coming O'Reilly book I mentioned was available for pre-ordering on B&N (I don't buy from Amazon :-) -- - Vicki Vicki Brown ZZZ Journeyman Sourceror: P.O. Box 1269 zz |\ _,,,---,,_Scripts & Philtres San Bruno, CA zz /,`.-'`'-. ;-;;,_ Perl, Unix, MacOS 94066 USA |,4- ) )-,_. ,\ ( `'-' mailto:[EMAIL PROTECTED] '---''(_/--' `-'\_) http://www.cfcl.com/~vlb
RE: Help with 403 Access Forbidden
[snipped]
>
> I just wrote a book about it all!
> I was going to wait until Amazon's record for it is updated,
> but since you
> asked...
> http://www.amazon.com/exec/obidos/ASIN/0596001789
>
> I should get around to updating the LWP Cookbook at some point, too.
> Currently it doesn't mention the shortcuts (like
> $browser->get) that have
> sprung up recently.
>
> I would love it if people on this list 1) ordered the book,
> and 2) wrote
> reviews about how happy they are with it, as compared to not
> having it at
> all. There is an unwelcome trend lately toward only
> malcontents bothering
> to write Amazon reviews of tech books. ("And on p 43 he
> misspells 'Bette
> Midler' as 'Bette Middler' so ZERO STARS THIS BOOK SUCKS!")
Hi Sean,
This is great that you have taken the time to write this. There is quite a
few postings in the perl newsgroups concerning LWP and perl. Is there a way
to take a look at the table of contents or perhaps a sample chapter?
Ron Hill
Re: Help with 403 Access Forbidden
At 10:30 2002-04-26 -0400, Greg E Tilton wrote:
>I'm hoping that someone can help me with a problem I'm having. I'm still
>pretty new at this and have difficulty in following much of what is
>discussed on the listserv. I have a script that has always allowed me to
>get data from a public web site.
For no particular reason that you'd intuit from the 403 message, the
problem seems to go away when I turn on cookies:
use strict;
use LWP;
my $url="";
my $secs=7200;
$url="http://www.sce.com/esp/Dlf/F2002.dlf";;
my $browser = LWP::UserAgent->new;
$browser->cookie_jar({}); # turn on in-memory cookies.
$browser->timeout($secs);
my $response = $browser->get($url);
if ($response->is_success) {
open(OUTPUT, ">testsceweb" ) or die $!;
binmode(OUTPUT);
print OUTPUT $response->content;
close OUTPUT;
} else {
die "Can't get $url -- ", $response->status_line;
}
--
Sean M. Burkehttp://www.spinn.net/~sburke/
RE: Help with 403 Access Forbidden
At 12:47 2002-04-26 -0700, Hill, Ronald wrote:
>Is there a way to take a look at the table of contents or perhaps a sample
>chapter?
Oo, good question. I think there'll be a sample chapter put up some time
or other, but in the mean time, here's a descriptive ToC from the Preface:
Chapter 1, Preparing for LWP, covers in general terms what LWP does, the
alternatives to using LWP, and when you shouldn't use LWP.
Chapter 2, HTTP and LWP::Simple, explains how the web works and the
easy-to-use yet limited functions for accessing it.
Chapter 3, The LWP Classes for HTTP, covers the more powerful interface to
the web.
Chapter 4, The LWP Classes for URLs, shows how to parse URLs with the URI
class, and how to convert between relative and absolute URLs.
Chapter 5, Forms, describes how to submit GET and POST forms.
Chapter 6, HTML Processing with Regular Expressions, shows how to extract
information from HTML using regular expressions.
Chapter 7, HTML Processing with Tokens, provides an alternative approach to
extracting data from HTML using the HTML::TokeParser module.
Chapter 8, Tokenizing Walkthrough, is a case study of data extraction using
tokens.
Chapter 9, HTML Processing with Trees, shows how to extract data from HTML
using the HTML::TreeBuilder module.
Chapter 10, Modifying HTML with Trees, covers the use of HTML::TreeBuilder
to modify HTML files.
Chapter 11, Cookies, Authentication, and Advanced Requests, deals with the
tougher parts of requests.
Chapter 12, Spiders, explores the technological issues involved in
automating the download of more than one page from a site.
Appendix A, LWP Modules, is a complete list of the LWP modules.
Appendix B, HTTP Status Codes, is a list of HTTP codes, what they mean, and
whether LWP considers them error or succcess.
Appendix C, Common MIME Types, contains the most common MIME types and what
they mean.
Appendix D, Common Language Tags, is a list of the most common language
tags and what they mean (e.g., "zh-cn" means Mainland Chinese, while "sv"
is Swedish).
Appendix E, Common Content Encodings, is a list of the most common
character encodings ("character sets") and the tags that identify them.
Appendix F, ASCII Table, is a table to help you make sense of the most
common Unicode characters. It shows the character, its numeric code (in
decimal, octal, and hex), and any HTML escapes there may be for it.
Appendix G, User's View of Object-Oriented Modules, is an introduction to
the use of Perl's object-oriented programming features.
--
Sean M. Burkehttp://www.spinn.net/~sburke/
Re: Help with 403 Access Forbidden
At 10:30 2002-04-26 -0400, Greg E Tilton wrote:
>(Also, I'm struggling in trying to get a grasp on the materials related to
>LWP (e.g., cookbook). I thank God for documentation like cookbook, but
>I'd like to get a more in-depth understanding of the subject. Is there
>something I can study to get up to speed? Any recommendations would be
>appreciated.)
I just wrote a book about it all!
I was going to wait until Amazon's record for it is updated, but since you
asked...
http://www.amazon.com/exec/obidos/ASIN/0596001789
I should get around to updating the LWP Cookbook at some point, too.
Currently it doesn't mention the shortcuts (like $browser->get) that have
sprung up recently.
I would love it if people on this list 1) ordered the book, and 2) wrote
reviews about how happy they are with it, as compared to not having it at
all. There is an unwelcome trend lately toward only malcontents bothering
to write Amazon reviews of tech books. ("And on p 43 he misspells 'Bette
Midler' as 'Bette Middler' so ZERO STARS THIS BOOK SUCKS!")
--
Sean M. Burkehttp://www.spinn.net/~sburke/
Re: Documentation
At 10:30 -0400 4/26/02, Greg E Tilton wrote: >I'd like to get a more in-depth >understanding of the subject. Is there something I can study to get up to >speed? Any recommendations would be appreciated.) O'Reilly has a book coming in July, "Perl and LWP", pre-orderable at Barnes & Noble.com http://shop.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=2BD824K8Q5&mscssid=TDWJQX26QREK8GVNG6E5HFQGNULA8P95&isbn=0596001789 -- - Vicki Vicki Brown ZZZ Journeyman Sourceror: P.O. Box 1269 zz |\ _,,,---,,_Scripts & Philtres San Bruno, CA zz /,`.-'`'-. ;-;;,_ Perl, Unix, MacOS 94066 USA |,4- ) )-,_. ,\ ( `'-' mailto:[EMAIL PROTECTED] '---''(_/--' `-'\_) http://www.cfcl.com/~vlb
Re: Doesn lynx support SSL
* Siva Namburi wrote: > Does anybody know if lynx would support ssl handshakes. I mean >can you browse sites which >have ssl enabled using lynx. Take a look at http://lynx.browser.org, there you can find how to enable SSL support in Lynx. PS: This mailing list is dedicated to the libwww-perl library, Lynx is not part of that lib and hence offtopic.
Re: Help with 403 Access Forbidden
Hi,
add this line and retry.
Am Freitag, 26. April 2002 16:30 hast Du geschrieben:
> $user_agent=new LWP::UserAgent;
$user_agent->agent("Mozilla/5.0");
> $user_agent->timeout([$secs]);
> $request=new HTTP::Request("GET","${url}");
--
cu boris
Schlägt der Blitz den Bauer tot, spart sein Weib ein Abendbrot.
Help with 403 Access Forbidden
I'm hoping that someone can help me with a problem I'm having. I'm still
pretty new at this and have difficulty in following much of what is
discussed on the listserv. I have a script that has always allowed me to
get data from a public web site. Now I can't seem to pull the data, even
though I can go to the web site by browser.
Here is my code:
$url="";
$Data="";
$secs=7200;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Status;
use HTTP::Response;
use LWP::Protocol;
use LWP::Authen::Basic;
use LWP::Simple;
$url="http://www.sce.com/esp/Dlf/F2002.dlf";;
$user_agent=new LWP::UserAgent;
$user_agent->timeout([$secs]);
$request=new HTTP::Request("GET","${url}");
$response=$user_agent->request($request);
if ($response->is_success)
{
open(OUTPUT, ">testsceweb" ) or die("Can't open the output file
testsceweb $!\n");
print OUTPUT $response->{_content};
close OUTPUT;
}
else
{
die "Fetch of SCE failed: ", $response->status_line;
}
The response is : 403 Access Forbidden
Why would I get this at a public site? (Also, I'm struggling in trying to
get a grasp on the materials related to LWP (e.g., cookbook). I thank God
for documentation like cookbook, but I'd like to get a more in-depth
understanding of the subject. Is there something I can study to get up to
speed? Any recommendations would be appreciated.)
Thanks,
Greg
CSC
