I am using Libcurl to fetch http code for a list of URLs in crawling
application.
When I tested that in a stand-alone machine mode, the speed just not satisfying
(and I think the same when it came to BOINC infrastructure)
I am wondering if I/O in BOINC affects the speed ?
Currently the algorithms are as below:
For each link in the buffer
{
(1) fetch http code for link using Libcurl
(2) print out into 'output' immediately (fprintf function)
}
The time for each loop above is around 2 seconds, and since there are usually
more than 20 links inside the buffer. It takes around one minute to finish 20
links.
Does anyone using Libcurl in BOINC know something about that ?
I just want to know whether this is reasonable or whether another way such as:
--------------------------
For each link in the buffer
{
fetch http code for link and store in memory
}
Print all links in the memory into 'output' at one time(no loop)
-----------------------
might be better ? But this way takes more memory in client side and I really
don't want to do that unless the speed has a huge change.
_______________________________________________
boinc_dev mailing list
[email protected]
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.
