On Mon, Apr 03, 2006 at 05:58:12PM -0400, J. Milgram wrote:
>
> wget does https, at least the one I have (v. 1.9.1)
Shows what I know :-)
> But I wonder...
> Is there a way to develop a list of all domain names?
> Then you could do:
> for d in domain_names:
> if url.open("http://%s"; % d): do stuff
> elif url.open("http://www.%s"; % d): do stuff
> elif url.open("http://www.%s/index.html"% d: do stuff
> etc etc
> maybe this is faster. Or slower.
The problem is this looks like port scanning (well, it basically is :-)
for web addresses. Part of the goal of this project is not to piss
anyone off... also, how do you come up with the list of domain names?
Webcrawling is a very established way of walking around lots and lots
of machines, so no one should raise too much of an eyebrow...
- Rob
.
