Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-25 Thread Lachlan McIlroy
Index: 2.6.x-xfs-new/fs/xfs/xfs_log.c === --- 2.6.x-xfs-new.orig/fs/xfs/xfs_log.c 2007-11-22 10:47:21.945395328 +1100 +++ 2.6.x-xfs-new/fs/xfs/xfs_log.c 2007-11-22 10:53:11.556186722 +1100 @@ -1443,6 +1443,8 @@ xlog_sync(xlog_t

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-25 Thread Lachlan McIlroy
Index: 2.6.x-xfs-new/fs/xfs/xfs_log.c === --- 2.6.x-xfs-new.orig/fs/xfs/xfs_log.c 2007-11-22 10:47:21.945395328 +1100 +++ 2.6.x-xfs-new/fs/xfs/xfs_log.c 2007-11-22 10:53:11.556186722 +1100 @@ -1443,6 +1443,8 @@ xlog_sync(xlog_t

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-24 Thread Matt Mackall
On Fri, Nov 23, 2007 at 01:01:15PM +0100, Andi Kleen wrote: > On Fri, Nov 23, 2007 at 03:03:29PM +1100, David Chinner wrote: > > On Fri, Nov 23, 2007 at 03:53:17AM +0100, Andi Kleen wrote: > > > On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: > > > > On Thu, Nov 22, 2007 at

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-24 Thread Matt Mackall
On Fri, Nov 23, 2007 at 01:01:15PM +0100, Andi Kleen wrote: On Fri, Nov 23, 2007 at 03:03:29PM +1100, David Chinner wrote: On Fri, Nov 23, 2007 at 03:53:17AM +0100, Andi Kleen wrote: On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 01:06:11PM +0100,

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-23 Thread Andi Kleen
On Fri, Nov 23, 2007 at 03:03:29PM +1100, David Chinner wrote: > On Fri, Nov 23, 2007 at 03:53:17AM +0100, Andi Kleen wrote: > > On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: > > > On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: > > > > > FWIW from a "real time"

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-23 Thread Andi Kleen
On Fri, Nov 23, 2007 at 03:03:29PM +1100, David Chinner wrote: On Fri, Nov 23, 2007 at 03:53:17AM +0100, Andi Kleen wrote: On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: FWIW from a real time database POV this

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Fri, Nov 23, 2007 at 03:53:17AM +0100, Andi Kleen wrote: > On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: > > On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: > > > > FWIW from a "real time" database POV this seems to make sense to me... > > > > in fact, we probably

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Andi Kleen
On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: > On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: > > > FWIW from a "real time" database POV this seems to make sense to me... > > > in fact, we probably rely on filesystem metadata way too much > > > (historically it's

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Matt Mackall
On Fri, Nov 23, 2007 at 10:09:22AM +1100, David Chinner wrote: > On Fri, Nov 23, 2007 at 09:29:09AM +1100, David Chinner wrote: > > On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: > > > If I've got XFS on filesystems A and B on the same spindle (or volume > > > group?) and my real RT

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Matt Mackall
On Fri, Nov 23, 2007 at 09:29:09AM +1100, David Chinner wrote: > On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: > > On Thu, Nov 22, 2007 at 09:31:59PM +1100, David Chinner wrote: > > [...] > > > > In other words, I/O priority is per-spindle and not per-filesystem and > > > > thus

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Fri, Nov 23, 2007 at 09:29:09AM +1100, David Chinner wrote: > On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: > > If I've got XFS on filesystems A and B on the same spindle (or volume > > group?) and my real RT I/O takes place only on B, then I want log > > flushing to happen in

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: > On Thu, Nov 22, 2007 at 09:31:59PM +1100, David Chinner wrote: > [...] > > > In other words, I/O priority is per-spindle and not per-filesystem and > > > thus this change has consequences that leak outside the filesystem in > > >

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Matt Mackall
On Thu, Nov 22, 2007 at 09:31:59PM +1100, David Chinner wrote: [...] > > In other words, I/O priority is per-spindle and not per-filesystem and > > thus this change has consequences that leak outside the filesystem in > > question. That's bad. > > This has nothing to do with this patch - it's a

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: > > FWIW from a "real time" database POV this seems to make sense to me... > > in fact, we probably rely on filesystem metadata way too much > > (historically it's just "worked" although we do seem to get issues > > on ext3). > > For

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Andi Kleen
> FWIW from a "real time" database POV this seems to make sense to me... > in fact, we probably rely on filesystem metadata way too much > (historically it's just "worked" although we do seem to get issues > on ext3). For that case you really would need priority inheritance: any metadata IO

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Thu, Nov 22, 2007 at 01:25:49AM -0600, Matt Mackall wrote: > On Thu, Nov 22, 2007 at 02:41:06PM +1100, David Chinner wrote: > > On Wed, Nov 21, 2007 at 08:57:27PM -0600, Matt Mackall wrote: > > > On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: > > > > In all the cases that I know

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Thu, Nov 22, 2007 at 01:25:49AM -0600, Matt Mackall wrote: On Thu, Nov 22, 2007 at 02:41:06PM +1100, David Chinner wrote: On Wed, Nov 21, 2007 at 08:57:27PM -0600, Matt Mackall wrote: On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: In all the cases that I know of where

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Andi Kleen
FWIW from a real time database POV this seems to make sense to me... in fact, we probably rely on filesystem metadata way too much (historically it's just worked although we do seem to get issues on ext3). For that case you really would need priority inheritance: any metadata IO on behalf

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: FWIW from a real time database POV this seems to make sense to me... in fact, we probably rely on filesystem metadata way too much (historically it's just worked although we do seem to get issues on ext3). For that case you

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Matt Mackall
On Thu, Nov 22, 2007 at 09:31:59PM +1100, David Chinner wrote: [...] In other words, I/O priority is per-spindle and not per-filesystem and thus this change has consequences that leak outside the filesystem in question. That's bad. This has nothing to do with this patch - it's a problem

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: On Thu, Nov 22, 2007 at 09:31:59PM +1100, David Chinner wrote: [...] In other words, I/O priority is per-spindle and not per-filesystem and thus this change has consequences that leak outside the filesystem in question.

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Fri, Nov 23, 2007 at 09:29:09AM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: If I've got XFS on filesystems A and B on the same spindle (or volume group?) and my real RT I/O takes place only on B, then I want log flushing to happen in RT on B.

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Matt Mackall
On Fri, Nov 23, 2007 at 10:09:22AM +1100, David Chinner wrote: On Fri, Nov 23, 2007 at 09:29:09AM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: If I've got XFS on filesystems A and B on the same spindle (or volume group?) and my real RT I/O

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread David Chinner
On Fri, Nov 23, 2007 at 03:53:17AM +0100, Andi Kleen wrote: On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: FWIW from a real time database POV this seems to make sense to me... in fact, we probably rely on

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Andi Kleen
On Fri, Nov 23, 2007 at 12:15:39AM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 01:06:11PM +0100, Andi Kleen wrote: FWIW from a real time database POV this seems to make sense to me... in fact, we probably rely on filesystem metadata way too much (historically it's just worked

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-22 Thread Matt Mackall
On Fri, Nov 23, 2007 at 09:29:09AM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 12:10:29PM -0600, Matt Mackall wrote: On Thu, Nov 22, 2007 at 09:31:59PM +1100, David Chinner wrote: [...] In other words, I/O priority is per-spindle and not per-filesystem and thus this change has

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Matt Mackall
On Thu, Nov 22, 2007 at 02:41:06PM +1100, David Chinner wrote: > On Wed, Nov 21, 2007 at 08:57:27PM -0600, Matt Mackall wrote: > > On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: > > > In all the cases that I know of where ppl are using what could > > > be considered real-time I/O

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Stewart Smith
On Thu, 2007-11-22 at 12:12 +1100, David Chinner wrote: > In all the cases that I know of where ppl are using what could > be considered real-time I/O (e.g. media environments where they > do real-time ingest and playout from the same filesystem) the > real-time ingest processes create the files

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread David Chinner
On Wed, Nov 21, 2007 at 08:57:27PM -0600, Matt Mackall wrote: > On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: > > In all the cases that I know of where ppl are using what could > > be considered real-time I/O (e.g. media environments where they > > do real-time ingest and playout

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Matt Mackall
On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: > On Thu, Nov 22, 2007 at 01:49:25AM +0100, Andi Kleen wrote: > > David Chinner <[EMAIL PROTECTED]> writes: > > > > > To ensure that log I/O is issued as the highest priority I/O, set > > > the I/O priority of the log I/O to the

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread David Chinner
On Thu, Nov 22, 2007 at 01:49:25AM +0100, Andi Kleen wrote: > David Chinner <[EMAIL PROTECTED]> writes: > > > To ensure that log I/O is issued as the highest priority I/O, set > > the I/O priority of the log I/O to the highest possible. This will > > ensure that log I/O is not held up behind bulk

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Andi Kleen
David Chinner <[EMAIL PROTECTED]> writes: > To ensure that log I/O is issued as the highest priority I/O, set > the I/O priority of the log I/O to the highest possible. This will > ensure that log I/O is not held up behind bulk data or other > metadata I/O as delaying log I/O can pause the entire

[PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread David Chinner
Reduce log I/O latency To ensure that log I/O is issued as the highest priority I/O, set the I/O priority of the log I/O to the highest possible. This will ensure that log I/O is not held up behind bulk data or other metadata I/O as delaying log I/O can pause the entire transaction subsystem.

[PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread David Chinner
Reduce log I/O latency To ensure that log I/O is issued as the highest priority I/O, set the I/O priority of the log I/O to the highest possible. This will ensure that log I/O is not held up behind bulk data or other metadata I/O as delaying log I/O can pause the entire transaction subsystem.

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Andi Kleen
David Chinner [EMAIL PROTECTED] writes: To ensure that log I/O is issued as the highest priority I/O, set the I/O priority of the log I/O to the highest possible. This will ensure that log I/O is not held up behind bulk data or other metadata I/O as delaying log I/O can pause the entire

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread David Chinner
On Thu, Nov 22, 2007 at 01:49:25AM +0100, Andi Kleen wrote: David Chinner [EMAIL PROTECTED] writes: To ensure that log I/O is issued as the highest priority I/O, set the I/O priority of the log I/O to the highest possible. This will ensure that log I/O is not held up behind bulk data or

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Matt Mackall
On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: On Thu, Nov 22, 2007 at 01:49:25AM +0100, Andi Kleen wrote: David Chinner [EMAIL PROTECTED] writes: To ensure that log I/O is issued as the highest priority I/O, set the I/O priority of the log I/O to the highest possible.

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread David Chinner
On Wed, Nov 21, 2007 at 08:57:27PM -0600, Matt Mackall wrote: On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: In all the cases that I know of where ppl are using what could be considered real-time I/O (e.g. media environments where they do real-time ingest and playout from

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Stewart Smith
On Thu, 2007-11-22 at 12:12 +1100, David Chinner wrote: In all the cases that I know of where ppl are using what could be considered real-time I/O (e.g. media environments where they do real-time ingest and playout from the same filesystem) the real-time ingest processes create the files and

Re: [PATCH 2/9]: Reduce Log I/O latency

2007-11-21 Thread Matt Mackall
On Thu, Nov 22, 2007 at 02:41:06PM +1100, David Chinner wrote: On Wed, Nov 21, 2007 at 08:57:27PM -0600, Matt Mackall wrote: On Thu, Nov 22, 2007 at 12:12:14PM +1100, David Chinner wrote: In all the cases that I know of where ppl are using what could be considered real-time I/O (e.g.