Re: libURL HTTP downloading

[Sivakatirswami]

Phil Davis wrote:
Thanks Trevor! I hadn't thought of downloading in a loop. Good call. 
I'm working with a training system too, where all the media is on a 
server. I really appreciate the feedback (from everyone).


Phil


Trevor DeVore wrote:


I had some training software that would download all of it's media 
from a server. The amount of media varied from course to course but 
there was a lot of it. Mostly still images but there were a number of 
videos as well. libURL worked great. The only thing I remember 
accounting for was timeouts. I would always download a url in a 
repeat loop. If there was no error then exit. If the error was 
timeout then try again, up to 3 - 5 times. I think I picked up this 
trick from Chipp.


Regards,

as for larger files: Andre wrote a downloader into our Hinduism Today 
Navigator (previously name Hinduism Today Digital Edition)  it has to 
download 4 PDF's which total over 30 MB or a single PDF which might be 
around 28 MB.. he also has some logging in there and we can see the 
failures and the re attempts for any given user in his  dbase record. 
One man had our rev client opened at around 8 am in  India.. .he 
clicked to download.. must have  some horrible old 28K modem. but 
after about 80 retries, the PDF finally landed on his hard drive at 
about 4 pm in the afternoon. I thought "wow, 8 hours our stack never 
gave up... libURL is pretty robust!" (if you code it right of course.) 
and I also had to hand it to the old codger (an elderly man with a PC 
and no rupees) for not giving up but just letting our application run 
and run and run. He was so happy to open the PDF that afternoon, one 
small victory for our favorite tool box. So, Phil PWC Proceed with 
Confidence.








___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: libURL HTTP downloading

[Trevor DeVore]

On Aug 8, 2009, at 7:08 PM, roger.e.el...@sealedair.com wrote:


How can one obtain this extra fancy spancy feature filled version of
libURL.  :-)   I have never been able to get Rev talking through a  
proxy.

It works fine from a direct internet connection though.  I need it to
prompt for proxy authentication.


It comes with the GLX Application Framework.




I've documented how to do a couple of things with the custom libURL  
including how to install it though there is still more documentation  
to do. Look at the "Internet Integration" chapter in the manual:




If you want to find the handlers I added to the library then search  
for "--> TKD Additions" in the custom libURL script. You can also  
search for "tkd" in the script to find notes I've made.


Whether or not you need the proxy features in the custom version  
depends on what kind of proxy server you are dealing with. If you  
always know the proxy server IP address, the server uses Basic  
Authentication and you just need to prompt for username/password then  
libUrlSetAuthCallback should work fine. You can read up on it and see  
some examples here:




If you are dealing with Digest or NTLM (Windows only) authentication  
but know the proxy server IP address then you can still use the  
standard libURL library but you will need some additional code to  
handle authorization. If you look at the glxapplicationframework.dat  
stack script (included in the distribution) you can search for  
glxappAuthDigestCallback and glxappAuthNTLMCallback. Note that for  
NTLM you will need the altNTLM.dll external that comes with the  
distribution. Altuit was kind enough to provide this.


If, however, your software is running on a computer that uses a PAC  
file to figure out the proxy server then you will need the custom  
version of libURL. You will also need to pick a number of handlers out  
of the glxapplicationframework stack script. Search for "--> Proxy  
handlers" in the script and you will be taken to all of the proxy  
initialization code.


Now, if you use the GLX Application Framework then all of the gory  
details are handled for you. You just have to make a single call to  
glxapp_initializeProxy when your app loads up and most situations will  
be handled automatically.


Regards,

--
Trevor DeVore
Blue Mango Learning Systems
www.bluemangolearning.com-www.screensteps.com
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: libURL HTTP downloading

[Richard Gaskin]

Roger.E.Eller wrote:


The customized version of libURL doesn't change how files are
downloaded so there would be no difference in regards to what Phil is
trying to do. The customized version of libURL and the GLX App
Framework work together to handle interactions with proxy servers but
the rest of the features just work:

* Automatic handling of cookies
* Support for making HEAD requests
* Callbacks for working with invalid SSL Certs (i.e. display dialog to
user to warn them)
* Non-blocking posts.


How can one obtain this extra fancy spancy feature filled version of
libURL.  :-)   I have never been able to get Rev talking through a proxy.
It works fine from a direct internet connection though.  I need it to
prompt for proxy authentication.


It's great that Trevor has made these available, but I wonder if a 
better question might be:


How can these changes be rolled into the official release of libURL?

--
 Richard Gaskin
 Fourth World
 Revolution training and consulting: http://www.fourthworld.com
 Webzine for Rev developers: http://www.revjournal.com
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: libURL HTTP downloading

[Roger . E . Eller]

> On Aug 8, 2009, at 4:33 PM, stephen barncard wrote:
>> Perhaps Trevor could chime in here... he has a upgraded version of
>> libURL
>> that is used in his GLX Application Framework, however I don't know
>> if it
>> will work outside his environment or if it can solve your problem,
>> but it's
>> worth checking out.

> The customized version of libURL doesn't change how files are
> downloaded so there would be no difference in regards to what Phil is
> trying to do. The customized version of libURL and the GLX App
> Framework work together to handle interactions with proxy servers but
> the rest of the features just work:
>
> * Automatic handling of cookies
> * Support for making HEAD requests
> * Callbacks for working with invalid SSL Certs (i.e. display dialog to
> user to warn them)
> * Non-blocking posts.

How can one obtain this extra fancy spancy feature filled version of
libURL.  :-)   I have never been able to get Rev talking through a proxy.
It works fine from a direct internet connection though.  I need it to
prompt for proxy authentication.

---
Roger Eller


___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: libURL HTTP downloading

[Phil Davis]

Thanks Trevor! I hadn't thought of downloading in a loop. Good call. I'm 
working with a training system too, where all the media is on a server. 
I really appreciate the feedback (from everyone).


Phil


Trevor DeVore wrote:

On Aug 8, 2009, at 4:33 PM, stephen barncard wrote:

Perhaps Trevor could chime in here... he has a upgraded version of 
libURL
that is used in his GLX Application Framework, however I don't know 
if it
will work outside his environment or if it can solve your problem, 
but it's

worth checking out.


The customized version of libURL doesn't change how files are 
downloaded so there would be no difference in regards to what Phil is 
trying to do. The customized version of libURL and the GLX App 
Framework work together to handle interactions with proxy servers but 
the rest of the features just work:


* Automatic handling of cookies
* Support for making HEAD requests
* Callbacks for working with invalid SSL Certs (i.e. display dialog to 
user to warn them)

* Non-blocking posts.

On Aug 8, 2009, at 4:25 PM, Phil Davis wrote:

I assume that, when using Rev's libURL-based HTTP downloading, errors 
are more likely to occur when downloading many smaller files than 
when downloading a few larger ones. Statistically this makes sense, 
but can anyone speak to this from their experience? The "many small 
files" scenario I'm talking about is a few hundred files in a single 
download sequence, with most files in the under-1mb category except 
for a few .mov files that are typically 5 to 70mb each.


I had some training software that would download all of it's media 
from a server. The amount of media varied from course to course but 
there was a lot of it. Mostly still images but there were a number of 
videos as well. libURL worked great. The only thing I remember 
accounting for was timeouts. I would always download a url in a repeat 
loop. If there was no error then exit. If the error was timeout then 
try again, up to 3 - 5 times. I think I picked up this trick from Chipp.


Regards,



--
Phil Davis

PDS Labs
Professional Software Development
http://pdslabs.net

___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: libURL HTTP downloading

[Trevor DeVore]

On Aug 8, 2009, at 4:33 PM, stephen barncard wrote:

Perhaps Trevor could chime in here... he has a upgraded version of  
libURL
that is used in his GLX Application Framework, however I don't know  
if it
will work outside his environment or if it can solve your problem,  
but it's

worth checking out.


The customized version of libURL doesn't change how files are  
downloaded so there would be no difference in regards to what Phil is  
trying to do. The customized version of libURL and the GLX App  
Framework work together to handle interactions with proxy servers but  
the rest of the features just work:


* Automatic handling of cookies
* Support for making HEAD requests
* Callbacks for working with invalid SSL Certs (i.e. display dialog to  
user to warn them)

* Non-blocking posts.

On Aug 8, 2009, at 4:25 PM, Phil Davis wrote:

I assume that, when using Rev's libURL-based HTTP downloading,  
errors are more likely to occur when downloading many smaller files  
than when downloading a few larger ones. Statistically this makes  
sense, but can anyone speak to this from their experience? The "many  
small files" scenario I'm talking about is a few hundred files in a  
single download sequence, with most files in the under-1mb category  
except for a few .mov files that are typically 5 to 70mb each.


I had some training software that would download all of it's media  
from a server. The amount of media varied from course to course but  
there was a lot of it. Mostly still images but there were a number of  
videos as well. libURL worked great. The only thing I remember  
accounting for was timeouts. I would always download a url in a repeat  
loop. If there was no error then exit. If the error was timeout then  
try again, up to 3 - 5 times. I think I picked up this trick from Chipp.


Regards,

--
Trevor DeVore
Blue Mango Learning Systems
www.bluemangolearning.com-www.screensteps.com
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: libURL HTTP downloading

[stephen barncard]

Perhaps Trevor could chime in here... he has a upgraded version of libURL
that is used in his GLX Application Framework, however I don't know if it
will work outside his environment or if it can solve your problem, but it's
worth checking out.

-
Stephen Barncard
San Francisco
http://barncard.com


2009/8/8 Phil Davis 

> I assume that, when using Rev's libURL-based HTTP downloading, errors are
> more likely to occur when downloading many smaller files than when
> downloading a few larger ones. Statistically this makes sense, but can
> anyone speak to this from their experience? The "many small files" scenario
> I'm talking about is a few hundred files in a single download sequence, with
> most files in the under-1mb category except for a few .mov files that are
> typically 5 to 70mb each.
>
> I'm trying to decide if it's worth the effort to add server-side scripting
> that zips the smaller files into larger clumps before downloading.
>
> Thanks for your feedback! I'm using libUrlVersion 1.1.6.
> --
> Phil Davis
>
> PDS Labs
> Professional Software Development
> http://pdslabs.net
>
> ___
> use-revolution mailing list
> use-revolution@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

libURL HTTP downloading

[Phil Davis]

I assume that, when using Rev's libURL-based HTTP downloading, errors 
are more likely to occur when downloading many smaller files than when 
downloading a few larger ones. Statistically this makes sense, but can 
anyone speak to this from their experience? The "many small files" 
scenario I'm talking about is a few hundred files in a single download 
sequence, with most files in the under-1mb category except for a few 
.mov files that are typically 5 to 70mb each.


I'm trying to decide if it's worth the effort to add server-side 
scripting that zips the smaller files into larger clumps before downloading.


Thanks for your feedback! I'm using libUrlVersion 1.1.6.
--
Phil Davis

PDS Labs
Professional Software Development
http://pdslabs.net

___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution