And we also need support for ICMP in some cases... telnet, FTP... POST some data (from a big database, from filesystem)... support for WebDAV... multithreaded load-stress stimulator (distributed "agents")... What I wrote initially was related to classic "crawler". +++
-----Original Message----- From: florent andré [mailto:[email protected]] Sent: April-27-11 5:12 PM To: [email protected] Subject: Re: About Link and POST requests yep, Actually some tools help to fill common / well identify forms fields (like email, phone, etc...)... So it's totally reasonable to think about a robot with some "how to fill form" rules. And could be valuable in some cases. ++ On 04/27/2011 03:32 PM, [email protected] wrote: > Hello Fuad, > > I can agree that robots should not perform POST requests, but Droids > is a library for building robots, it is not a robot itself. > My suggestion to extends the request types was to widen the scope > where Droids could be useful. > > I would find that an unfortunate decision, but that's just my point of view. > > Thanks, > > Giulio Cesare > > > On Wed, Apr 27, 2011 at 1:55 PM, Fuad Efendi<[email protected]> wrote: >> Hi Giulio, "other request types" - HEAD only, and only if Internet >> Server supports that (I don't think more than 1% support, and I don't >> think it will be performance improvement in an era of sitemaps)... >> there is no "Link" request types, and "POST" shouldn't be done by >> robots - otherwise which URL will you show in a search results page? >> You can't show form submission via POST as a search results -Fuad >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] >> Sent: April-27-11 7:33 AM >> To: [email protected] >> Subject: About Link and POST requests >> >> Hello, >> >> using Droid library I came to the realization that the current API are not suitable for handling anything other than GET requests. >> >> This is due mainly to two choices made in the code: >> - using the Link interface for collecting and processing pages to >> process (all based on simple URI values, both for 'getURI' and >> 'getTo'); >> - content being loaded by CrawlingWorker, passing just the URI to the Protocol. >> >> Is there anyone interested in thinking how to change the API to support also other request types? >> >> Regards, >> >> Giulio Cesare >> >>
