Hi all, I am currently planning to write my own web crawler. I know Python but not Perl, and I am interested in knowing which of these two are a better choice given the following scenario:
1) I/O issues: my biggest constraint in terms of resource will be bandwidth throttle neck. 2) Efficiency issues: The crawlers have to be fast, robust and as "memory efficient" as possible. I am running all of my crawlers on cheap pcs with about 500 mb RAM and P3 to P4 processors 3) Compatibility issues: Most of these crawlers will run on Unix (FreeBSD), so there should exist a pretty good compiler that can optimize my code these under the environments. What are your opinions? -- http://mail.python.org/mailman/listinfo/python-list
