Hi.
On Wed, 2007-01-10 at 22:07 +0800, Fengguang Wu wrote:
> Thanks, Nigel.
> But I'm very sorry that the calculation in the patch was wrong.
>
> Would you give this new patch a run?
Sorry for my slowness. I just did
time find /usr/src | wc -l
again:
Without patch: 35.137, 35.104, 35.351
Hi.
On Wed, 2007-01-10 at 22:07 +0800, Fengguang Wu wrote:
Thanks, Nigel.
But I'm very sorry that the calculation in the patch was wrong.
Would you give this new patch a run?
Sorry for my slowness. I just did
time find /usr/src | wc -l
again:
Without patch: 35.137, 35.104, 35.351 seconds
On Wed, Jan 10, 2007 at 02:20:49PM +1100, Nigel Cunningham wrote:
> Hi.
>
> On Wed, 2007-01-10 at 09:57 +0800, Fengguang Wu wrote:
> > On Tue, Jan 09, 2007 at 08:23:32AM -0800, Linus Torvalds wrote:
> > >
> > >
> > > On Tue, 9 Jan 2007, Fengguang Wu wrote:
> > > > >
> > > > > The fastest and
On Wed, Jan 10, 2007 at 02:20:49PM +1100, Nigel Cunningham wrote:
Hi.
On Wed, 2007-01-10 at 09:57 +0800, Fengguang Wu wrote:
On Tue, Jan 09, 2007 at 08:23:32AM -0800, Linus Torvalds wrote:
On Tue, 9 Jan 2007, Fengguang Wu wrote:
The fastest and probably most important
Hi.
On Wed, 2007-01-10 at 09:57 +0800, Fengguang Wu wrote:
> On Tue, Jan 09, 2007 at 08:23:32AM -0800, Linus Torvalds wrote:
> >
> >
> > On Tue, 9 Jan 2007, Fengguang Wu wrote:
> > > >
> > > > The fastest and probably most important thing to add is some readahead
> > > > smarts to directories ---
On Tue, Jan 09, 2007 at 08:23:32AM -0800, Linus Torvalds wrote:
>
>
> On Tue, 9 Jan 2007, Fengguang Wu wrote:
> > >
> > > The fastest and probably most important thing to add is some readahead
> > > smarts to directories --- both to the htree and non-htree cases. If
> >
> > Here's is a quick hack
On Tue, 9 Jan 2007, Fengguang Wu wrote:
> >
> > The fastest and probably most important thing to add is some readahead
> > smarts to directories --- both to the htree and non-htree cases. If
>
> Here's is a quick hack to practice the directory readahead idea.
> Comments are welcome, it's a
On Mon, Jan 08, 2007 at 07:58:19AM -0500, Theodore Tso wrote:
> On Mon, Jan 08, 2007 at 08:35:55AM +0530, Suparna Bhattacharya wrote:
> > > Yeah, slowly-growing directories will get splattered all over the disk.
> > >
> > > Possible short-term fixes would be to just allocate up to (say) eight
> >
On Mon, Jan 08, 2007 at 07:58:19AM -0500, Theodore Tso wrote:
On Mon, Jan 08, 2007 at 08:35:55AM +0530, Suparna Bhattacharya wrote:
Yeah, slowly-growing directories will get splattered all over the disk.
Possible short-term fixes would be to just allocate up to (say) eight
blocks
On Tue, 9 Jan 2007, Fengguang Wu wrote:
The fastest and probably most important thing to add is some readahead
smarts to directories --- both to the htree and non-htree cases. If
Here's is a quick hack to practice the directory readahead idea.
Comments are welcome, it's a freshman's
On Tue, Jan 09, 2007 at 08:23:32AM -0800, Linus Torvalds wrote:
On Tue, 9 Jan 2007, Fengguang Wu wrote:
The fastest and probably most important thing to add is some readahead
smarts to directories --- both to the htree and non-htree cases. If
Here's is a quick hack to practice the
Hi.
On Wed, 2007-01-10 at 09:57 +0800, Fengguang Wu wrote:
On Tue, Jan 09, 2007 at 08:23:32AM -0800, Linus Torvalds wrote:
On Tue, 9 Jan 2007, Fengguang Wu wrote:
The fastest and probably most important thing to add is some readahead
smarts to directories --- both to the htree
On Mon, Jan 08, 2007 at 05:09:34PM -0800, Paul Jackson wrote:
> Jeff wrote:
> > Something I just thought of: ATA and SCSI hard disks do their own
> > read-ahead.
>
> Probably this is wishful thinking on my part, but I would have hoped
> that most of the read-ahead they did was for stuff that
Jeff wrote:
> Something I just thought of: ATA and SCSI hard disks do their own
> read-ahead.
Probably this is wishful thinking on my part, but I would have hoped
that most of the read-ahead they did was for stuff that happened to be
on the cylinder they were reading anyway. So long as their
On Sun, 7 Jan 2007, Shawn O. Pearce wrote:
> Krzysztof Halasa <[EMAIL PROTECTED]> wrote:
> > Hmm... Perhaps it should be possible to push git updates as a pack
> > file only? I mean, the pack file would stay packed = never individual
> > files and never 256 directories?
>
> Latest Git does this.
On Mon, Jan 08, 2007 at 02:59:52PM +0100, Pavel Machek wrote:
> Hi!
>
> > > Would e2fsck -D help? What kind of optimization
> > > does it perform?
> >
> > It will help a little; e2fsck -D compresses the logical view of the
> > directory, but it doesn't optimize the physical layout on disk at
On Mon, Jan 08, 2007 at 07:58:19AM -0500, Theodore Tso wrote:
>
> The fastest and probably most important thing to add is some readahead
> smarts to directories --- both to the htree and non-htree cases. If
> you're using some kind of b-tree structure, such as XFS does for
> directories,
Hi!
> > Would e2fsck -D help? What kind of optimization
> > does it perform?
>
> It will help a little; e2fsck -D compresses the logical view of the
> directory, but it doesn't optimize the physical layout on disk at all,
> and of course, it won't help with the lack of readahead logic. It's
>
On Mon, Jan 08, 2007 at 02:41:47PM +0100, Johannes Stezenbach wrote:
>
> Would e2fsck -D help? What kind of optimization
> does it perform?
It will help a little; e2fsck -D compresses the logical view of the
directory, but it doesn't optimize the physical layout on disk at all,
and of course, it
Theodore Tso wrote:
The fastest and probably most important thing to add is some readahead
smarts to directories --- both to the htree and non-htree cases. If
you're using some kind of b-tree structure, such as XFS does for
directories, preallocation doesn't help you much. Delayed allocation
On Mon, Jan 08, 2007 at 08:35:55AM +0530, Suparna Bhattacharya wrote:
> > Yeah, slowly-growing directories will get splattered all over the disk.
> >
> > Possible short-term fixes would be to just allocate up to (say) eight
> > blocks when we grow a directory by one block. Or teach the
> >
On Mon, Jan 08, 2007 at 08:35:55AM +0530, Suparna Bhattacharya wrote:
Yeah, slowly-growing directories will get splattered all over the disk.
Possible short-term fixes would be to just allocate up to (say) eight
blocks when we grow a directory by one block. Or teach the
Theodore Tso wrote:
The fastest and probably most important thing to add is some readahead
smarts to directories --- both to the htree and non-htree cases. If
you're using some kind of b-tree structure, such as XFS does for
directories, preallocation doesn't help you much. Delayed allocation
On Mon, Jan 08, 2007 at 02:41:47PM +0100, Johannes Stezenbach wrote:
Would e2fsck -D help? What kind of optimization
does it perform?
It will help a little; e2fsck -D compresses the logical view of the
directory, but it doesn't optimize the physical layout on disk at all,
and of course, it
Hi!
Would e2fsck -D help? What kind of optimization
does it perform?
It will help a little; e2fsck -D compresses the logical view of the
directory, but it doesn't optimize the physical layout on disk at all,
and of course, it won't help with the lack of readahead logic. It's
possible
On Mon, Jan 08, 2007 at 07:58:19AM -0500, Theodore Tso wrote:
The fastest and probably most important thing to add is some readahead
smarts to directories --- both to the htree and non-htree cases. If
you're using some kind of b-tree structure, such as XFS does for
directories,
On Mon, Jan 08, 2007 at 02:59:52PM +0100, Pavel Machek wrote:
Hi!
Would e2fsck -D help? What kind of optimization
does it perform?
It will help a little; e2fsck -D compresses the logical view of the
directory, but it doesn't optimize the physical layout on disk at all,
and of
On Sun, 7 Jan 2007, Shawn O. Pearce wrote:
Krzysztof Halasa [EMAIL PROTECTED] wrote:
Hmm... Perhaps it should be possible to push git updates as a pack
file only? I mean, the pack file would stay packed = never individual
files and never 256 directories?
Latest Git does this. If the
Jeff wrote:
Something I just thought of: ATA and SCSI hard disks do their own
read-ahead.
Probably this is wishful thinking on my part, but I would have hoped
that most of the read-ahead they did was for stuff that happened to be
on the cylinder they were reading anyway. So long as their
On Mon, Jan 08, 2007 at 05:09:34PM -0800, Paul Jackson wrote:
Jeff wrote:
Something I just thought of: ATA and SCSI hard disks do their own
read-ahead.
Probably this is wishful thinking on my part, but I would have hoped
that most of the read-ahead they did was for stuff that happened to
On Sun, Jan 07, 2007 at 01:15:42AM -0800, Andrew Morton wrote:
> On Sun, 7 Jan 2007 09:55:26 +0100
> Willy Tarreau <[EMAIL PROTECTED]> wrote:
>
> > On Sat, Jan 06, 2007 at 09:39:42PM -0800, Linus Torvalds wrote:
> > >
> > >
> > > On Sat, 6 Jan 2007, H. Peter Anvin wrote:
> > > >
> > > > During
Robert Fitzsimons wrote:
>> Some more data on how git affects kernel.org...
>
> I have a quick question about the gitweb configuration, does the
> $projects_list config entry point to a directory or a file?
It can point to both. Usually it is either unset, and then we
do find over $projectroot,
Krzysztof Halasa <[EMAIL PROTECTED]> wrote:
> Hmm... Perhaps it should be possible to push git updates as a pack
> file only? I mean, the pack file would stay packed = never individual
> files and never 256 directories?
Latest Git does this. If the server is later than 1.4.3.3 then
the
On Sun, 7 Jan 2007, Randy Dunlap wrote:
>
> ISTM that Linus is trying to make 2.6.20-final before LCA. We'll see.
No. Hopefully "final -rc" before LCA, but I'll do the actual 2.6.20
release afterwards. I don't want to have a merge window during LCA, as I
and many others will all be out
On Sun, 7 Jan 2007, Jon Smirl wrote:
> >
> > - proper read-ahead. Right now, even if the directory is totally
> >contiguous on disk (just remove the thing that writes data to the
> >files, so that you'll have empty files instead of 8kB files), I think
> >we do those reads totally
On Sun, 7 Jan 2007 20:07:43 +0100 (MET) Jan Engelhardt wrote:
>
> On Jan 7 2007 10:49, Randy Dunlap wrote:
> >On Sun, 7 Jan 2007 11:50:57 +0100 (MET) Jan Engelhardt wrote:
> >> On Jan 7 2007 10:03, Willy Tarreau wrote:
> >> >On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
> >> >>
On Jan 7 2007 10:49, Randy Dunlap wrote:
>On Sun, 7 Jan 2007 11:50:57 +0100 (MET) Jan Engelhardt wrote:
>> On Jan 7 2007 10:03, Willy Tarreau wrote:
>> >On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
>> >> >[..]
>> >> >entries in directories with millions of files on disk. I'm
On Sun, 7 Jan 2007, Linus Torvalds wrote:
>
> A year or two ago I did a totally half-assed code for the non-hashed
> readdir that improved performance by an order of magnitude for ext3 for a
> test-case of mine, but it was subtly buggy and didn't do the hashed case
> AT ALL.
Btw, this isn't
With my gitweb caching changes this isn't as big of a deal as the front
page is only generated once every 10 minutes or so (and with the changes
I'm working on today that timeout will be variable)
- John
On Sun, 2007-01-07 at 14:57 +, Robert Fitzsimons wrote:
> > Some more data on how git
On Sun, 7 Jan 2007 11:50:57 +0100 (MET) Jan Engelhardt wrote:
>
> On Jan 7 2007 10:03, Willy Tarreau wrote:
> >On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
> >> >[..]
> >> >entries in directories with millions of files on disk. I'm not
> >> >certain it would be that easy to
On Sun, 7 Jan 2007, Christoph Hellwig wrote:
>
> On Sun, Jan 07, 2007 at 10:03:36AM +0100, Willy Tarreau wrote:
> > The problem is that I have no sufficient FS knowledge to argument why
> > it helps here. It was a desperate attempt to fix the problem for us
> > and it definitely worked well.
>
"H. Peter Anvin" <[EMAIL PROTECTED]> writes:
> During extremely high load, it appears that what slows kernel.org down
> more than anything else is the time that each individual getdents()
> call takes. When I've looked this I've observed times from 200 ms to
> almost 2 seconds! Since an
> Some more data on how git affects kernel.org...
I have a quick question about the gitweb configuration, does the
$projects_list config entry point to a directory or a file?
When it is a directory gitweb ends up doing the equivalent of a 'find
$project_list' to find all the available projects,
On Jan 7 2007 10:03, Willy Tarreau wrote:
>On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
>> >[..]
>> >entries in directories with millions of files on disk. I'm not
>> >certain it would be that easy to try other filesystems on
>> >kernel.org though :-/
>>
>> Changing
On Sun, Jan 07, 2007 at 10:28:53AM +, Christoph Hellwig wrote:
> On Sun, Jan 07, 2007 at 10:03:36AM +0100, Willy Tarreau wrote:
> > The problem is that I have no sufficient FS knowledge to argument why
> > it helps here. It was a desperate attempt to fix the problem for us
> > and it
On Sun, Jan 07, 2007 at 10:03:36AM +0100, Willy Tarreau wrote:
> The problem is that I have no sufficient FS knowledge to argument why
> it helps here. It was a desperate attempt to fix the problem for us
> and it definitely worked well.
XFS does rather efficient btree directories, and it does
On 01/07/2007 10:15 AM, Andrew Morton wrote:
Yeah, slowly-growing directories will get splattered all over the
disk.
Possible short-term fixes would be to just allocate up to (say) eight
blocks when we grow a directory by one block. Or teach the
directory-growth code to use ext3
On Sun, 7 Jan 2007 09:55:26 +0100
Willy Tarreau <[EMAIL PROTECTED]> wrote:
> On Sat, Jan 06, 2007 at 09:39:42PM -0800, Linus Torvalds wrote:
> >
> >
> > On Sat, 6 Jan 2007, H. Peter Anvin wrote:
> > >
> > > During extremely high load, it appears that what slows kernel.org down
> > > more
> >
On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
> Willy Tarreau wrote:
> >
> >At work, we had the same problem on a file server with ext3. We use rsync
> >to make backups to a local IDE disk, and we noticed that getdents() took
> >about the same time as Peter reports (0.2 to 2
Willy Tarreau wrote:
At work, we had the same problem on a file server with ext3. We use rsync
to make backups to a local IDE disk, and we noticed that getdents() took
about the same time as Peter reports (0.2 to 2 seconds), especially in
maildir directories. We tried many things to fix it with
On Sat, Jan 06, 2007 at 09:39:42PM -0800, Linus Torvalds wrote:
>
>
> On Sat, 6 Jan 2007, H. Peter Anvin wrote:
> >
> > During extremely high load, it appears that what slows kernel.org down more
> > than anything else is the time that each individual getdents() call takes.
> > When I've looked
On Sat, Jan 06, 2007 at 09:39:42PM -0800, Linus Torvalds wrote:
On Sat, 6 Jan 2007, H. Peter Anvin wrote:
During extremely high load, it appears that what slows kernel.org down more
than anything else is the time that each individual getdents() call takes.
When I've looked this I've
Willy Tarreau wrote:
At work, we had the same problem on a file server with ext3. We use rsync
to make backups to a local IDE disk, and we noticed that getdents() took
about the same time as Peter reports (0.2 to 2 seconds), especially in
maildir directories. We tried many things to fix it with
On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
Willy Tarreau wrote:
At work, we had the same problem on a file server with ext3. We use rsync
to make backups to a local IDE disk, and we noticed that getdents() took
about the same time as Peter reports (0.2 to 2 seconds),
On Sun, 7 Jan 2007 09:55:26 +0100
Willy Tarreau [EMAIL PROTECTED] wrote:
On Sat, Jan 06, 2007 at 09:39:42PM -0800, Linus Torvalds wrote:
On Sat, 6 Jan 2007, H. Peter Anvin wrote:
During extremely high load, it appears that what slows kernel.org down
more
than anything else
On 01/07/2007 10:15 AM, Andrew Morton wrote:
Yeah, slowly-growing directories will get splattered all over the
disk.
Possible short-term fixes would be to just allocate up to (say) eight
blocks when we grow a directory by one block. Or teach the
directory-growth code to use ext3
On Sun, Jan 07, 2007 at 10:03:36AM +0100, Willy Tarreau wrote:
The problem is that I have no sufficient FS knowledge to argument why
it helps here. It was a desperate attempt to fix the problem for us
and it definitely worked well.
XFS does rather efficient btree directories, and it does
On Sun, Jan 07, 2007 at 10:28:53AM +, Christoph Hellwig wrote:
On Sun, Jan 07, 2007 at 10:03:36AM +0100, Willy Tarreau wrote:
The problem is that I have no sufficient FS knowledge to argument why
it helps here. It was a desperate attempt to fix the problem for us
and it definitely
On Jan 7 2007 10:03, Willy Tarreau wrote:
On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
[..]
entries in directories with millions of files on disk. I'm not
certain it would be that easy to try other filesystems on
kernel.org though :-/
Changing filesystems would mean
Some more data on how git affects kernel.org...
I have a quick question about the gitweb configuration, does the
$projects_list config entry point to a directory or a file?
When it is a directory gitweb ends up doing the equivalent of a 'find
$project_list' to find all the available projects,
H. Peter Anvin [EMAIL PROTECTED] writes:
During extremely high load, it appears that what slows kernel.org down
more than anything else is the time that each individual getdents()
call takes. When I've looked this I've observed times from 200 ms to
almost 2 seconds! Since an unpacked *OR*
On Sun, 7 Jan 2007, Christoph Hellwig wrote:
On Sun, Jan 07, 2007 at 10:03:36AM +0100, Willy Tarreau wrote:
The problem is that I have no sufficient FS knowledge to argument why
it helps here. It was a desperate attempt to fix the problem for us
and it definitely worked well.
XFS does
On Sun, 7 Jan 2007 11:50:57 +0100 (MET) Jan Engelhardt wrote:
On Jan 7 2007 10:03, Willy Tarreau wrote:
On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
[..]
entries in directories with millions of files on disk. I'm not
certain it would be that easy to try other
With my gitweb caching changes this isn't as big of a deal as the front
page is only generated once every 10 minutes or so (and with the changes
I'm working on today that timeout will be variable)
- John
On Sun, 2007-01-07 at 14:57 +, Robert Fitzsimons wrote:
Some more data on how git
On Sun, 7 Jan 2007, Linus Torvalds wrote:
A year or two ago I did a totally half-assed code for the non-hashed
readdir that improved performance by an order of magnitude for ext3 for a
test-case of mine, but it was subtly buggy and didn't do the hashed case
AT ALL.
Btw, this isn't the
On Jan 7 2007 10:49, Randy Dunlap wrote:
On Sun, 7 Jan 2007 11:50:57 +0100 (MET) Jan Engelhardt wrote:
On Jan 7 2007 10:03, Willy Tarreau wrote:
On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
[..]
entries in directories with millions of files on disk. I'm not
certain it
On Sun, 7 Jan 2007 20:07:43 +0100 (MET) Jan Engelhardt wrote:
On Jan 7 2007 10:49, Randy Dunlap wrote:
On Sun, 7 Jan 2007 11:50:57 +0100 (MET) Jan Engelhardt wrote:
On Jan 7 2007 10:03, Willy Tarreau wrote:
On Sun, Jan 07, 2007 at 12:58:38AM -0800, H. Peter Anvin wrote:
[..]
entries
On Sun, 7 Jan 2007, Jon Smirl wrote:
- proper read-ahead. Right now, even if the directory is totally
contiguous on disk (just remove the thing that writes data to the
files, so that you'll have empty files instead of 8kB files), I think
we do those reads totally
On Sun, 7 Jan 2007, Randy Dunlap wrote:
ISTM that Linus is trying to make 2.6.20-final before LCA. We'll see.
No. Hopefully final -rc before LCA, but I'll do the actual 2.6.20
release afterwards. I don't want to have a merge window during LCA, as I
and many others will all be out anyway.
Krzysztof Halasa [EMAIL PROTECTED] wrote:
Hmm... Perhaps it should be possible to push git updates as a pack
file only? I mean, the pack file would stay packed = never individual
files and never 256 directories?
Latest Git does this. If the server is later than 1.4.3.3 then
the receive-pack
Robert Fitzsimons wrote:
Some more data on how git affects kernel.org...
I have a quick question about the gitweb configuration, does the
$projects_list config entry point to a directory or a file?
It can point to both. Usually it is either unset, and then we
do find over $projectroot, or
On Sun, Jan 07, 2007 at 01:15:42AM -0800, Andrew Morton wrote:
On Sun, 7 Jan 2007 09:55:26 +0100
Willy Tarreau [EMAIL PROTECTED] wrote:
On Sat, Jan 06, 2007 at 09:39:42PM -0800, Linus Torvalds wrote:
On Sat, 6 Jan 2007, H. Peter Anvin wrote:
During extremely high load, it
On Sat, 6 Jan 2007, H. Peter Anvin wrote:
>
> During extremely high load, it appears that what slows kernel.org down more
> than anything else is the time that each individual getdents() call takes.
> When I've looked this I've observed times from 200 ms to almost 2 seconds!
> Since an unpacked
Some more data on how git affects kernel.org...
During extremely high load, it appears that what slows kernel.org down
more than anything else is the time that each individual getdents() call
takes. When I've looked this I've observed times from 200 ms to almost
2 seconds! Since an unpacked
Some more data on how git affects kernel.org...
During extremely high load, it appears that what slows kernel.org down
more than anything else is the time that each individual getdents() call
takes. When I've looked this I've observed times from 200 ms to almost
2 seconds! Since an unpacked
On Sat, 6 Jan 2007, H. Peter Anvin wrote:
During extremely high load, it appears that what slows kernel.org down more
than anything else is the time that each individual getdents() call takes.
When I've looked this I've observed times from 200 ms to almost 2 seconds!
Since an unpacked *OR*
76 matches
Mail list logo
