On Dec 9, 2009, at 7:39 PM, my name wrote:
I'm currently planning on writing a web crawler in python but have a
question as far as how I should design it. My goal is speed and
maximum
efficient use of the hardware\bandwidth I have available.
As of now I have a Dual 2.4ghz xeon box, 4g
I'm currently planning on writing a web crawler in python but have a
question as far as how I should design it. My goal is speed and maximum
efficient use of the hardware\bandwidth I have available.
As of now I have a Dual 2.4ghz xeon box, 4gb ram, 500gb sata and a 20mbps
bandwidth cap (fo
I think something that may be even more important to consider than just
the pure speed of your program, would be ease of design as well as the
overall stability of your code.
My opinion would be that writing in Python would have many benefits
over the speed gains of using C. For instance, you craw
abhinav wrote:
> I want to strke a balance between development speed and crawler speed.
"The best performance improvement is the transition from the
nonworking state to the working state."- J. Osterhout
Try to get there are soon as possible. You can figure out what
that means. ;^)
When y
Ravi Teja <[EMAIL PROTECTED]> wrote:
...
> The rule of thumb for all your Python Vs C questions is ...
> 1.) Choose Python by default.
+1 QOTW!-)
> 2.) If your program is slow, it's your algorithm that you need to check
Seriously: yes, and (often even more importantly) data structure.
Howev
This is following the pattern of your previous post on language choice
wrt. writing a mail server. It is very common for beginers to over
emphasize performance requirements, size of the executable etc. More is
always good. Right? Yes! But at what cost?
The rule of thumb for all your Python Vs C qu
abhinav wrote:
> Hi guys.I have to implement a topical crawler as a part of my
> project.What language should i implement
> C or Python?Python though has fast development cycle but my concern is
> speed also.I want to strke a balance between development speed and
> crawler speed.Since Python is an
On Wed, 15 Feb 2006 21:56:52 -0800, abhinav wrote:
> Hi guys.I have to implement a topical crawler as a part of my
> project.What language should i implement
> C or Python?Python though has fast development cycle but my concern is
> speed also.I want to strke a balance between development speed an
On 15 Feb 2006 21:56:52 -0800, abhinav <[EMAIL PROTECTED]> wrote:
> Hi guys.I have to implement a topical crawler as a part of my
> project.What language should i implement
> C or Python?
Why does this keep coming up on here as of late? If you search the
archives, you can find numerous posts about
abhinav wrote:
> Hi guys.I have to implement a topical crawler as a part of my
> project.What language should i implement
Oh, and there's some really good books out there, besides the Orilly
Spidering Hacks. Springer Verlag has a couple books on "Text Mining"
and at least a couple books with "we
Paul Rubin wrote:
> "abhinav" <[EMAIL PROTECTED]> writes:
> > maintaining huge data structures.What should be the language so as
> > not to compromise that much on speed.What is the performance of
> > python based crawlers vs C based crawlers.Should I use both the
> > languages(partly C and pytho
"abhinav" <[EMAIL PROTECTED]> writes:
> It is DSL broadband 128kbps.But thats not the point.
But it is the point.
> What i am saying is that would python be fine for implementing fast
> crawler algorithms or should i use C.Handling huge
> data,multithreading,file handling,heuristics for ranking,a
abhinav wrote:
> It is DSL broadband 128kbps.But thats not the point.What i am saying is
> that would python be fine for implementing fast crawler algorithms or
> should i use C.
But a web crawler is going to be *mainly* I/O bound - so language
efficiency won't be the main issue. There are severa
It is DSL broadband 128kbps.But thats not the point.What i am saying is
that would python be fine for implementing fast crawler algorithms or
should i use C.Handling huge data,multithreading,file
handling,heuristics for ranking,and maintaining huge data
structures.What should be the language so as
"abhinav" <[EMAIL PROTECTED]> writes:
> The crawler which will be working on huge set of pages should be
> as fast as possible.
What kind of network connection do you have, that's fast enough
that even a fairly cpu-inefficient crawler won't saturate it?
--
http://mail.python.org/mailman/listinfo/
Hi guys.I have to implement a topical crawler as a part of my
project.What language should i implement
C or Python?Python though has fast development cycle but my concern is
speed also.I want to strke a balance between development speed and
crawler speed.Since Python is an interpreted language it i
16 matches
Mail list logo
