On Feb 12, 2010, at 9:51 PM, K. Richard Pixley wrote:
> I'm working with parallelized build servers. We often have raided
> disks, solid state disks, servers with huge amounts of disk cache
> specifically so that an entire build happens in memory, etc. File io is
> our bottleneck.
Yeah, this level of disk manipulation is past the point where a little bit less
blocking on I/O will help... I imagine you've already got some kind of
process/thread pooling solution already, or at least you'll need one.
> I think you probably are also forgetting about NFS. NFS isn't slower
> than native disk in terms of throughput, only in terms of latency, which
> is a fabulous opportunity for asyncronous file io. (Granted, NFS seems
> to have fallen out of fashion recently.)
No, I'm not forgetting about it: I'm just saying that once you've got an API
that applications can start using that gives *some* performance benefit
(non-blocking disk I/O at the expense of spinning up a few threads / processes
behind the scenes), you can always optimize it for other use-cases later,
without necessarily changing the API.
> I think twisted already has everything that's required. It could
> probably use a slightly more friendly interface paradigm so the user
> doesn't have to do his own os.open, but really, even that wouldn't save
> much.
It would allow us to do it more portably, I think. os.open()'s behavior can
vary a lot depending on what you do with it.
> Reactor core was enough to sell me on twisted. That's probably all I'll
> even be using. And compared to writing my own, that's enough to be useful.
Great, glad to hear it!
_______________________________________________
Twisted-Python mailing list
[email protected]
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
