> From: Daniel Shahaf [mailto:[email protected]]
>
> > I've had the greatest complaints for >15min commits.
> >
>
> So, a commit takes 1min when the server is idle, and 15min when the
> server is busy.
Actually, I'm surprised what I'm learning now. Although it matters if the
server is busy, that's not the root cause of the problem. Also, changing
the compression level makes a difference, but it's not the difference we
were hoping for.
I finally found a rock-solid and reproducible test case, as follows:
. There is a binary file, approx 45M, called layout.oa
. There are only 14 revs where it was changed. It varies slightly
in size, 44M, 45M, 46M...
. If I export all the different rev's of that file (parent dir, no
subdirs) then some rev's are repeatably less than 11sec to export, while
other revs are repeatably 15min.
. During all of the above exports, I am the only user using the
system. On the server, I see precisely one svnserve jump up to 100% cpu
utilization for the duration of the export.
I can't imagine any reason for such an enormous difference. I'm not sure
what I should look at next. I'll have to just start reading and reading and
reading code & documentation to get an idea precisely what other
possibilities may be going on.
Any ideas or suggestions?
This thread is no longer development related, unless this is a bug. We're
running svn 1.5.7 via svnserve, built from source on centos 5.1 x86_64. If
anyone cares, I'll happily move to the users list.
Please see attachments.
> > The way things are right now, svndiff, zlib_encode() take a chunk of
> data,
> (in svndiff.c)
> > performs compression on it, and writes (a) the size of the data, and
> (b)
> > whichever is smaller: the data, or the compressed data.
> >
>
> Note that the correctness of zlib_decode() depends on this check being
> done by the encoder.
I built subversion once with the default compression, once with compression
set to 1, and once with compression disabled. We're currently using the one
that's built with compression set to 1. But for the sake of continuing
discussion which is engaging, here's how I built the one with compression
disabled:
In zlib_encode() there is an if() statement, to see if len <
MIN_COMPRESS_SIZE, in which case, compression will not be done. I simply
commented out the if() {} else {} statement, to make zlib_encode()
unconditionally behave as if len < MIN_COMPRESS_SIZE. That is: No
compression.
Aside from svndiff.c, there are a million places where zlib is called, but
they all use Z_DEFAULT_COMPRESSION. So I also edited zlib/deflate.c:
if (level == Z_DEFAULT_COMPRESSION) level = 0; // was formerly 6
(occurs two times.)
[ehar...@simon trash]$ du -sh */layout.oa
0 layout-19628/layout.oa
32M layout-19637/layout.oa
41M layout-20548/layout.oa
43M layout-21963/layout.oa
43M layout-21993/layout.oa
43M layout-22016/layout.oa
43M layout-22369/layout.oa
45M layout-22529/layout.oa
44M layout-22644/layout.oa
43M layout-22678/layout.oa
44M layout-22679/layout.oa
44M layout-22825/layout.oa
46M layout-22948/layout.oa
46M layout-HEAD/layout.oa
[ehar...@simon trash]$ for f in layout-*/layout.oa ; do echo
"-------------------- $f" ; time gzip -c --fast $f > ${f}.gz1 ; du -sh ${f}.gz1
; done
-------------------- layout-19628/layout.oa
real 0m0.001s
user 0m0.000s
sys 0m0.001s
4.0K layout-19628/layout.oa.gz1
-------------------- layout-19637/layout.oa
real 0m0.817s
user 0m0.789s
sys 0m0.028s
7.8M layout-19637/layout.oa.gz1
-------------------- layout-20548/layout.oa
real 0m1.320s
user 0m0.926s
sys 0m0.036s
8.7M layout-20548/layout.oa.gz1
-------------------- layout-21963/layout.oa
real 0m1.000s
user 0m0.973s
sys 0m0.027s
9.2M layout-21963/layout.oa.gz1
-------------------- layout-21993/layout.oa
real 0m1.101s
user 0m0.972s
sys 0m0.029s
9.2M layout-21993/layout.oa.gz1
-------------------- layout-22016/layout.oa
real 0m0.999s
user 0m0.971s
sys 0m0.028s
9.2M layout-22016/layout.oa.gz1
-------------------- layout-22369/layout.oa
real 0m1.100s
user 0m0.970s
sys 0m0.030s
9.1M layout-22369/layout.oa.gz1
-------------------- layout-22529/layout.oa
real 0m1.058s
user 0m0.983s
sys 0m0.048s
9.2M layout-22529/layout.oa.gz1
-------------------- layout-22644/layout.oa
real 0m1.111s
user 0m0.977s
sys 0m0.031s
9.2M layout-22644/layout.oa.gz1
-------------------- layout-22678/layout.oa
real 0m0.999s
user 0m0.969s
sys 0m0.029s
9.1M layout-22678/layout.oa.gz1
-------------------- layout-22679/layout.oa
real 0m1.106s
user 0m0.978s
sys 0m0.028s
9.2M layout-22679/layout.oa.gz1
-------------------- layout-22825/layout.oa
real 0m1.147s
user 0m0.984s
sys 0m0.038s
9.3M layout-22825/layout.oa.gz1
-------------------- layout-22948/layout.oa
real 0m1.041s
user 0m1.019s
sys 0m0.022s
9.3M layout-22948/layout.oa.gz1
-------------------- layout-HEAD/layout.oa
real 0m1.139s
user 0m1.006s
sys 0m0.033s
9.3M layout-HEAD/layout.oa.gz1
[ehar...@simon trash]$ for f in layout-*/layout.oa ; do echo
"-------------------- $f" ; time gzip -c $f > ${f}.gz6 ; du -sh ${f}.gz6 ; done
-------------------- layout-19628/layout.oa
real 0m0.005s
user 0m0.000s
sys 0m0.002s
4.0K layout-19628/layout.oa.gz6
-------------------- layout-19637/layout.oa
real 0m3.960s
user 0m3.839s
sys 0m0.020s
7.6M layout-19637/layout.oa.gz6
-------------------- layout-20548/layout.oa
real 0m4.500s
user 0m4.368s
sys 0m0.027s
8.0M layout-20548/layout.oa.gz6
-------------------- layout-21963/layout.oa
real 0m3.694s
user 0m3.654s
sys 0m0.039s
8.5M layout-21963/layout.oa.gz6
-------------------- layout-21993/layout.oa
real 0m3.705s
user 0m3.668s
sys 0m0.034s
8.5M layout-21993/layout.oa.gz6
-------------------- layout-22016/layout.oa
real 0m3.719s
user 0m3.687s
sys 0m0.031s
8.5M layout-22016/layout.oa.gz6
-------------------- layout-22369/layout.oa
real 0m3.723s
user 0m3.689s
sys 0m0.034s
8.6M layout-22369/layout.oa.gz6
-------------------- layout-22529/layout.oa
real 0m3.775s
user 0m3.714s
sys 0m0.057s
8.6M layout-22529/layout.oa.gz6
-------------------- layout-22644/layout.oa
real 0m3.735s
user 0m3.701s
sys 0m0.029s
8.6M layout-22644/layout.oa.gz6
-------------------- layout-22678/layout.oa
real 0m3.933s
user 0m3.705s
sys 0m0.026s
8.6M layout-22678/layout.oa.gz6
-------------------- layout-22679/layout.oa
real 0m3.931s
user 0m3.693s
sys 0m0.037s
8.6M layout-22679/layout.oa.gz6
-------------------- layout-22825/layout.oa
real 0m3.963s
user 0m3.722s
sys 0m0.039s
8.7M layout-22825/layout.oa.gz6
-------------------- layout-22948/layout.oa
real 0m3.993s
user 0m3.763s
sys 0m0.028s
8.7M layout-22948/layout.oa.gz6
-------------------- layout-HEAD/layout.oa
real 0m4.328s
user 0m4.290s
sys 0m0.032s
8.7M layout-HEAD/layout.oa.gz6
[ehar...@simon trash]$ for rev in HEAD 22948 22825 22679 22678 22644 22529
22369 22016 21993 21963 20548 19637 19628 ; do echo "-------------------- $rev"
; time svn export -r $rev
svn://svn/lyric/trunk/projects/fft2/versic/trunk/fft2/core_1024pt/layout
./layout-$rev ; done
-------------------- HEAD
A layout-HEAD
A layout-HEAD/master.tag
A layout-HEAD/layout.oa
Exported revision 22952.
real 0m10.157s
user 0m1.328s
sys 0m0.152s
-------------------- 22948
A layout-22948
A layout-22948/master.tag
A layout-22948/layout.oa
Exported revision 22948.
real 0m10.247s
user 0m1.313s
sys 0m0.166s
-------------------- 22825
A layout-22825
A layout-22825/master.tag
A layout-22825/layout.oa
Exported revision 22825.
real 0m10.163s
user 0m1.292s
sys 0m0.163s
-------------------- 22679
A layout-22679
A layout-22679/master.tag
A layout-22679/layout.oa
Exported revision 22679.
real 15m40.676s
user 0m1.278s
sys 0m0.158s
-------------------- 22678
A layout-22678
A layout-22678/master.tag
A layout-22678/layout.oa
Exported revision 22678.
real 15m37.733s
user 0m1.273s
sys 0m0.158s
-------------------- 22644
A layout-22644
A layout-22644/master.tag
A layout-22644/layout.oa
Exported revision 22644.
real 0m7.258s
user 0m1.284s
sys 0m0.162s
-------------------- 22529
A layout-22529
A layout-22529/master.tag
A layout-22529/layout.oa
Exported revision 22529.
real 0m11.355s
user 0m1.293s
sys 0m0.169s
-------------------- 22369
A layout-22369
A layout-22369/master.tag
A layout-22369/layout.oa
Exported revision 22369.
real 15m38.157s
user 0m1.268s
sys 0m0.158s
-------------------- 22016
A layout-22016
A layout-22016/master.tag
A layout-22016/layout.oa
Exported revision 22016.
real 15m40.137s
user 0m1.285s
sys 0m0.133s
-------------------- 21993
A layout-21993
A layout-21993/master.tag
A layout-21993/layout.oa
Exported revision 21993.
real 0m7.278s
user 0m1.259s
sys 0m0.166s
-------------------- 21963
A layout-21963
A layout-21963/master.tag
A layout-21963/layout.oa
Exported revision 21963.
real 0m7.651s
user 0m1.266s
sys 0m0.143s
-------------------- 20548
A layout-20548
A layout-20548/master.tag
A layout-20548/layout.oa
Exported revision 20548.
real 0m7.068s
user 0m1.234s
sys 0m0.150s
-------------------- 19637
A layout-19637
A layout-19637/master.tag
A layout-19637/layout.oa
Exported revision 19637.
real 0m7.777s
user 0m0.979s
sys 0m0.116s
-------------------- 19628
A layout-19628
A layout-19628/master.tag
A layout-19628/layout.oa
Exported revision 19628.
real 0m0.027s
user 0m0.005s
sys 0m0.007s
[r...@svnserver ~]# for rev in 22948 22825 22679 22678 22644 22529 22369 22016
21993 21963 20548 19637 19628 ; do du -sh /repos/lyric/db/revs/$rev ; done
7.4M /repos/lyric/db/revs/22948
39M /repos/lyric/db/revs/22825
1.3M /repos/lyric/db/revs/22679
35M /repos/lyric/db/revs/22678
28M /repos/lyric/db/revs/22644
37M /repos/lyric/db/revs/22529
27M /repos/lyric/db/revs/22369
17M /repos/lyric/db/revs/22016
11M /repos/lyric/db/revs/21993
39M /repos/lyric/db/revs/21963
38M /repos/lyric/db/revs/20548
8.5M /repos/lyric/db/revs/19637
2.1M /repos/lyric/db/revs/19628
