Alexander Jolk wrote:
Hi Murali,
So the option for you I guess is to also disable kernel aio if you dont
wish to patch your kernel.
I'll try that right away.
Tried it, and it works just fine. Thanks for the hint!
Alex
--
Alexander Jolk * BUF Compagnie * [EMAIL PROTECTED]
Tel +33-1 42 6
I'm not sure exactly which kernels from kernel.org are affected, but we
ran into a serious problem on the 2.6 kernel that we were using in RHEL4
(2.6.9-22.0.1.ELsmp). The symptoms occur during a write-heavy
workloads. From the PVFS2 point of view, write throughput on one or
more servers will
The default pvfs2 configuration files (and matching #define's in
pvfs2-server.h) list the following job timeout values:
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
The flow timeouts trigger when X seconds have passed without mo
This is somewhat related to the timeout discussion from the previous
email, but this time the issue is the "op timeout" that the kernel
module uses. This is an absolute timeout associated with every upcall
that the kernel submits, and is fully independent of the job timeouts
that the pvfs2-cli
Background:
This whole issue started off while trying to debug the PVFS2
stall/timeout problem that ended up being caused by the ext3 reservation
bug... but we found some interesting things along the way.
One of the things we noticed while looking at the problem is that
occasionally a Trove w
This sounds like a good idea to me. I agree that competing timeouts are
not a good thing, and that pvfs2-client has more information to work
with than the kernel does.
Rob
Phil Carns wrote:
This is somewhat related to the timeout discussion from the previous
email, but this time the issue is
Hi Phil,
First of all, great work!
There are 2 other parameters that I thought could also make an impact
a) Choice of file-system (this was investigated by Nathan last year, I
think) as well as choice of journaling modes.
b) In case of storage-spaces created directly on top of an IDE/SCSI disks
or
On Fri, Mar 24, 2006 at 09:05:24AM +0100, Phil Carns wrote:
> I'm not sure exactly which kernels from kernel.org are affected, but we
> ran into a serious problem on the 2.6 kernel that we were using in RHEL4
> (2.6.9-22.0.1.ELsmp). The symptoms occur during a write-heavy
> workloads. From the
Murali Vilayannur wrote:
Hi Phil,
First of all, great work!
There are 2 other parameters that I thought could also make an impact
a) Choice of file-system (this was investigated by Nathan last year, I
think) as well as choice of journaling modes.
We looked at this a little bit again recently,
On Fri, Mar 24, 2006 at 09:23:19AM +0100, Phil Carns wrote:
> We have recently found some test scenarios where 30 seconds isn't really
> long enough. In particular, if you have the following combination:
>
> - fast server with a lot of RAM
> - relatively high latency storage (old SAN hardware)
>
Nice Phil. I saw this exact same sort of stalling eight years ago on
grendel at Clemson! But we didn't have alternative schedulers and the
like to play with at the time.
It might be worth our time to explore the dirty_ratio value a little
more in the context of both I/O and metadata tests. Per
Phil, I've done some tests for noncontiguous I/O comparing the
lio_listio, aio_read/aio_write, and normal read/write. In cases where
there are a lot of noncontiguous regions, lio_listio and aio tend to
really fall behind. At least 1 order of magnitude slower than normal
read/write.
Avery
On Fri
Hey Avery,
In what environment were you testing this?
Rob
Avery Ching wrote:
Phil, I've done some tests for noncontiguous I/O comparing the
lio_listio, aio_read/aio_write, and normal read/write. In cases where
there are a lot of noncontiguous regions, lio_listio and aio tend to
really fall be
Hi Phil,
> We looked at this a little bit again recently, but not in the context of
> buffer cache saturation. I don't have numbers handy, but I can share
> some general impressions. It still looks like data=writeback is stil
> probably the fastest mode in general, although it isn't entirely cle
As far as the AIO stuff goes, we kicked around an idea here that didn't
really help the workloads that we were looking at in this case, but it
may help something else.
If you look at what aio does, it spawns off threads for each fd up to a
limit that is tunable by calling aio_init() call (usin
Rob Ross wrote:
Nice Phil. I saw this exact same sort of stalling eight years ago on
grendel at Clemson! But we didn't have alternative schedulers and the
like to play with at the time.
It might be worth our time to explore the dirty_ratio value a little
more in the context of both I/O and me
Hey,
I have been constantly disappointed with the amount of work we have to
do to try to get decent performance out of the Linux VFS stack and GNU
libc, and this is just the latest example of that problem.
Thanks for sharing your experience with us. I keep thinking that the
Linux VFS async I
This stuff seems kind of promising:
http://www.bullopensource.org/posix/
They have a posix compliant aio interface sitting on top of the kernel
aio interface. They also have some kernel patches to eliminate the
O_DIRECT (and alignment limitations?) that normally come along with
using the new
Rob Ross wrote:
Nice Phil. I saw this exact same sort of stalling eight years ago on
grendel at Clemson! But we didn't have alternative schedulers and the
like to play with at the time.
That made me chuckle to think of Grendel. The more technology changes
the more it stays the same...
_
at least i don't have to assemble nodes by hand any more :). -- rob
Phil Carns wrote:
Rob Ross wrote:
Nice Phil. I saw this exact same sort of stalling eight years ago on
grendel at Clemson! But we didn't have alternative schedulers and the
like to play with at the time.
That made me chuckle
I think it was just on Reiser3, ext3 and ext2. It was a long time ago.
But for tiny pieces of noncontiguous data access (i.e. 8 bytes or so),
easily an order of magnitude of difference. It wasn't a PVFS2 specific
test.
Avery
On Fri, 2006-03-24 at 11:22 -0600, Rob Ross wrote:
> Hey Avery,
>
>
Robert Latham wrote:
On Fri, Mar 24, 2006 at 09:05:24AM +0100, Phil Carns wrote:
I'm not sure exactly which kernels from kernel.org are affected, but we
ran into a serious problem on the 2.6 kernel that we were using in RHEL4
(2.6.9-22.0.1.ELsmp). The symptoms occur during a write-heavy
work
On Fri, Mar 24, 2006 at 10:19:33AM -0600, Robert Latham wrote:
> Ugh. Thanks for tracking this down. I guess the only thing we can do
> is add it to the FAQ?
http://www.pvfs.org/pvfs2/pvfs2-faq.html#sec:write_slowdown
Thanks again for doing all the leg work on this
==rob
--
Rob Latham
Math
23 matches
Mail list logo
