On Thu, 28 Apr 2011 00:58:22 +1000, Chris Angelico
wrote:
: thousand threads? a couple of million? In Python, it'll probably end
: up pretty similar; chances are you won't be taking much advantage of
: multiple CPUs/cores (because the threads will all be waiting for
: socket read, or the sin
On Thu, Apr 28, 2011 at 12:15 AM, Hans Georg Schaathun
wrote:
> Quite. I was referring to some tutorials and documentation recommending
> to use non-blocking sockets and select() within a single thread. I
> cannot say that I understand why, but I can imagine the benefit with
> heavy traffic.
O
On Wed, 27 Apr 2011 23:35:06 +1000, Chris Angelico
wrote:
: On Wed, Apr 27, 2011 at 10:21 PM, Hans Georg Schaathun
: wrote:
: > That's correct. And the client initiates the connection. At the
: > moment, I use one thread per connection, and don't really want to
: > spend the time t
On Wed, Apr 27, 2011 at 10:21 PM, Hans Georg Schaathun
wrote:
> On Wed, 27 Apr 2011 11:35:16 +0200, Thomas Rachel
> wrote:
> : As far as I understand, you acquire a job, send it to a remote host via
> : a socket and then wait for the answer. Is that correct?
>
> That's correct. And the clien
On Wed, 27 Apr 2011 11:35:16 +0200, Thomas Rachel
wrote:
: As far as I understand, you acquire a job, send it to a remote host via
: a socket and then wait for the answer. Is that correct?
That's correct. And the client initiates the connection. At the
moment, I use one thread per connecti
On Tue, 26 Apr 2011 23:54:24 -0700, geremy condra
wrote:
: This sounds like a hadoop job, with the caveat that you still have to
: get your objects across the network somehow. Have you tried xdrlib or
: the struct module? I suspect either would save you some time.
Packing the objects up does
Am 26.04.2011 21:55, schrieb Hans Georg Schaathun:
Now, I would like to use remote hosts as well, more precisely, student
lab boxen which are rather unreliable. By experience I'd expect to
lose roughly 4-5 jobs in 100 CPU hours on average. Thus I need some
way of detecting lost connections and
On Tue, Apr 26, 2011 at 10:58 PM, Hans Georg Schaathun
wrote:
> On Tue, 26 Apr 2011 14:31:59 -0700, geremy condra
> wrote:
> : Without knowledge of what you're doing it's hard to comment
> : intelligently,
>
> I need to calculate map( foobar, L ) where foobar() is a pure function
> with no dep
On Tue, 26 Apr 2011 14:31:59 -0700, geremy condra
wrote:
: Without knowledge of what you're doing it's hard to comment
: intelligently,
I need to calculate map( foobar, L ) where foobar() is a pure function
with no dependency on the global state, L is a list of tuples, each
containing two
On Tue, Apr 26, 2011 at 12:55 PM, Hans Georg Schaathun
wrote:
> I wonder if anyone has any experience with this ...
>
> I try to set up a simple client-server system to do some number
> crunching, using a simple ad hoc protocol over TCP/IP. I use
> two Queue objects on the server side to manage t
On Wed, Apr 27, 2011 at 6:47 AM, Hans Georg Schaathun
wrote:
> Does that answer your question, Chris?
Yup! It does. :)
Chris Angelico
--
http://mail.python.org/mailman/listinfo/python-list
On Tue, Apr 26, 2011 at 1:20 PM, Chris Angelico wrote:
> But question: Why are you doing major number crunching in Python? On
> your quad-core machine, recode in C and see if you can do the whole
> job without bothering the unreliable boxen at all.
The reason is very simple. I cannot afford the
On Wed, Apr 27, 2011 at 6:33 AM, Dan Stromberg wrote:
> On Tue, Apr 26, 2011 at 1:20 PM, Chris Angelico wrote:
>
>> But question: Why are you doing major number crunching in Python? On
>> your quad-core machine, recode in C and see if you can do the whole
>> job without bothering the unreliable b
On Tue, Apr 26, 2011 at 1:20 PM, Chris Angelico wrote:
> But question: Why are you doing major number crunching in Python? On
> your quad-core machine, recode in C and see if you can do the whole
> job without bothering the unreliable boxen at all.
Hmm, or try Cython or PyPy. ^_^
Here's that g
On Tue, Apr 26, 2011 at 12:55 PM, Hans Georg Schaathun
wrote:
> I wonder if anyone has any experience with this ...
>
> I try to set up a simple client-server system to do some number
> crunching, using a simple ad hoc protocol over TCP/IP. I use
> two Queue objects on the server side to manage t
On Wed, Apr 27, 2011 at 5:55 AM, Hans Georg Schaathun
wrote:
> It is, of course, possible for the master thread upon processing the
> results, to requeue the tasks for any missing results, but it seems
> to me to be a cleaner solution if I could detect disconnects and
> requeue the tasks from the
I wonder if anyone has any experience with this ...
I try to set up a simple client-server system to do some number
crunching, using a simple ad hoc protocol over TCP/IP. I use
two Queue objects on the server side to manage the input and the output
of the client process. A basic system running
17 matches
Mail list logo
