On 5/24/05, Franklin <[EMAIL PROTECTED]> wrote:
> Hi;
>
> I have a small program that uses getstore to fetch a xml webpahe and
> store it. It is:
>
> use strict;
> use warnings;
>
> use LWP::Simple;
>
> my
> $xmlurl="http://www.sec.gov/Archives/edgar/data/903404/000118143105029692/rrd79736.xml";;
> my $xmlfile="xml.xml";
> my $status=getstore($xmlurl,$xmlfile);
>
> if(is_success($status))
> {
> print "xmlfile store is ok\n";
> }
> else
> {
> print "xmlfile store isn't ok\n";
> }
>
>
> It works well on windows platform, but doesn't on freebsd. That is,
> every time the getstore function's response is "200 ok", but the
> stored xml file isn't correct, it means, it isn't as same as the xml
> webpage. What is wrong with it? And is there any solution or
> alternatives to this problems?
>
> Thank you very much!
>
> Best wishes
> Franklin
>
> T
This sounds like a server issue; most people see this frequently when
trying to grab pages, particularly from IIS servers, even with normal
web browsers ("Page doesn't contain any data"). Actually, I'd say if
you're only seeing "several" connection errors in 1,000, you're doing
pretty well for a web app. Of course, it depends on how many is
"several". If it's three, for instance, that's a 99.7% success rate.
Have you tried it a thousand times on Windows?
Next question: what does the code look like? Are the requests being
sent in parallel or sequentially? Either way, the FreeBSD networking
stack is much more efficient, and you may just be overwhelming the
server with 1,000 requests in a way you aren't when issueing the same
request from Windows (which Windows?). If it's your server, up the
connection limit and see if that helps.
If you really think it's FreeBSD, make sure you're not running out of
mbufs, or memory for your tate tables 9assuming you're keeping state).
I doubt it's FreeBSD, though.
In any case, it's almost certainly not an LWP issue.
One last thought, though, related to whether you're running these
requests in sequence or in parallel: You're not using any kind of
file locking. Nor are you performing any kind of check to see if the
system was actually able to open the file for writing. And you're
calling all the files xml.xml. With buffered writes (and even
without) that has the makings race conditions and other wierdness.
since you're (hopefully) creating all of these files within a short
amount of time, try using File::Temp or a similar method to give each
file a separate name. you can unlink the files once you've checked
them against -s.
HTH,
-- jay
--------------------
daggerquill [at] gmail [dot] com
http://www.engatiki.org
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
<http://learn.perl.org/> <http://learn.perl.org/first-response>
