El Vie 23 Oct 2009 20:16:34 Kunsheng Chen escribió:
> I am using Libcurl to fetch http code for a list of URLs in crawling
> application.
>
> When I tested that in a stand-alone machine mode, the speed just not
> satisfying (and I think the same when it came to BOINC infrastructure)
>
> I am wondering if I/O in BOINC affects the speed ?
>
> Currently the algorithms are as below:
>
> For each link in the buffer
> {
> (1) fetch http code for link using Libcurl
> (2) print out into 'output' immediately (fprintf function)
> }
>
> The time for each loop above is around 2 seconds, and since there are
> usually more than 20 links inside the buffer. It takes around one minute to
> finish 20 links.
>
> Does anyone using Libcurl in BOINC know something about that ?
>
> I just want to know whether this is reasonable or whether another way such
> as:
>
> --------------------------
>
> For each link in the buffer
> {
> fetch http code for link and store in memory
> }
>
> Print all links in the memory into 'output' at one time(no loop)
>
>
> -----------------------
>
> might be better ? But this way takes more memory in client side and I
> really don't want to do that unless the speed has a huge change.
Your bottleneck is clearly the network. I really don't think the three bytes
you're writing to the disk are causing any slowdown...
--
Nicolas
_______________________________________________
boinc_dev mailing list
[email protected]
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.
