"svn pget svn:externals -r . -R" dramatically slow

[Andry]

Hello Users,

Original issue: https://issues.apache.org/jira/browse/SVN-4681

Just discovered a really strange case where exactly titled command bring really 
slow response.

The repository contains more than 1000 revisions. The WC is in the middle, say 
at rev 1193 (the current), the show log shows 1191 at the last revision, the 
HEAD revision say 1300.
Trying to cd into WC directory and run the command. Needs about 1 minute  to 
wait it's response. The Process Hacker shows traffic with the server up to 
about 2MB.

If try to run command w/o -R flag - command returns immediately.

The content of the externals property without the -R flag is:
https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
^/solutions/project1/sdk proj2-sdk

The content of the externals property with the -R flag is:
https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
^/solutions/project1/sdk proj2-sdk
https://domain.ab/svn/proj2/trunk/proj2-gui -
https://domain.ab/svn/proj2/trunk/proj2-gui/lib/Resource/Files -

I think the 2 last records draws the svn mad and it begin to crawl the server 
for something for about 1 minute.

I tries variations of the command. For example all these having the same result 
as above:
svn pget svn:externals -r "1193" "https://domain.ab/svn/proj2/trunk; -R 
--non-interactive
svn pget svn:externals -r "1193" "https://domain.ab/svn/proj2/trunk@1193; -R 
--non-interactive
svn pget svn:externals "https://domain.ab/svn/proj2/trunk@1193; -R 
--non-interactive

Dig a bit further and found this:
svn info https://domain.ab/svn/proj2/trunk/proj2-gui
svn: warning: W17: URL 'https://domain.ab/svn/proj2/trunk/proj2-gui' 
non-existent in revision 1300
svn: E29: Could not display info for all targets because some targets don't 
exist

Seems the 2 last records reference URL's what does not exist anymore in the 
HEAD.

These 2 records are left after an upgrade from a previous version of database, 
but why is the svn runs so slow about it? Anyway, i can't just cleanup the 
externals because they are from 3dparty repository in which i have no access.

-- 
Best regards,
 Andry  mailto:an...@inbox.ru

Re: Error running make for subversion

[Daniel Shahaf]

Joseph, Anselm wrote on Tue, May 16, 2017 at 21:04:57 +:
> chmod: /opt/eai/ci/subversion-1.9.5/svn/libexec/mod_dav_svn.so: A file or 
> directory in the path name does not exist.
> apxs:Error: Command failed with rc=65536

Note that if you use svn:// or svn+ssh://, you don't need mod_dav_svn
and can disable building it with --without-apxs to configure.

I don't have time to look into the actual problem right now, sorry.

RE: Error running make for subversion

[Joseph, Anselm]

I set LDFLAGS and it got rid of the libexpat warnings. 
Still getting the same error with 'make install' though.
Any guidance is appreciated. 
Thank you.

/opt/eai/ci/subversion/build/install-she -c -m 644 
./subversion/svnversion/svnversion.1 
/opt/eai/ci/subversion-1.9.5/svn/share/man/man1/svnversion.1
if true ; then cd subversion/mod_dav_svn ; 
/opt/eai/ci/subversion/build/install-sh -c -d 
"/opt/eai/ci/subversion-1.9.5/svn/libexec" ; 
/opt/eai/ci/httpd-2.2.32/apache/bin/apxs -i -S 
LIBEXECDIR="/opt/eai/ci/subversion-1.9.5/svn/libexec" -a -n dav_svn 
mod_dav_svn.la ; fi
/opt/eai/ci/httpd-2.2.32/apache/build/instdso.sh 
SH_LIBTOOL='/opt/freeware/lib/apr-1/build/libtool' mod_dav_svn.la 
/opt/eai/ci/subversion-1.9.5/svn/libexec
rm -f /opt/eai/ci/subversion-1.9.5/svn/libexec/mod_dav_svn.so
/opt/freeware/lib/apr-1/build/libtool --mode=install cp mod_dav_svn.la 
/opt/eai/ci/subversion-1.9.5/svn/libexec/
libtool: install: warning: relinking `mod_dav_svn.la'
libtool: install: (cd /opt/eai/ci/subversion/subversion/mod_dav_svn; /bin/sh 
"/opt/eai/ci/subversion/libtool"  --tag CC --silent --mode=relink gcc -shared 
-g -O2 -g -O2 -pthread -L/opt/freeware/lib -rpath 
/opt/eai/ci/subversion-1.9.5/svn/libexec -avoid-version -module -o 
mod_dav_svn.la activity.lo authz.lo deadprops.lo liveprops.lo lock.lo merge.lo 
mirror.lo mod_dav_svn.lo posts/create_txn.lo reports/dated-rev.lo 
reports/deleted-rev.lo reports/file-revs.lo reports/get-location-segments.lo 
reports/get-locations.lo reports/get-locks.lo reports/inherited-props.lo 
reports/log.lo reports/mergeinfo.lo reports/replay.lo reports/update.lo 
repos.lo status.lo util.lo version.lo 
../../subversion/libsvn_repos/libsvn_repos-1.la 
../../subversion/libsvn_fs/libsvn_fs-1.la 
../../subversion/libsvn_delta/libsvn_delta-1.la 
../../subversion/libsvn_subr/libsvn_subr-1.la )
libtool: install: cp .libs/mod_dav_svn.aT 
/opt/eai/ci/subversion-1.9.5/svn/libexec/mod_dav_svn.a
libtool: install: cp .libs/mod_dav_svn.lai 
/opt/eai/ci/subversion-1.9.5/svn/libexec/mod_dav_svn.la
chmod 755 /opt/eai/ci/subversion-1.9.5/svn/libexec/mod_dav_svn.so
chmod: /opt/eai/ci/subversion-1.9.5/svn/libexec/mod_dav_svn.so: A file or 
directory in the path name does not exist.
apxs:Error: Command failed with rc=65536
.
make: 1254-004 The error code from the last command is 1.


Stop.

-Original Message-
From: Joseph, Anselm 
Sent: Monday, May 08, 2017 5:13 PM
To: 'Daniel Shahaf'
Cc: 'users@subversion.apache.org'
Subject: RE: Error running make for subversion

I realized it is removing mod_dav_svn.so and then doing a chmod on the file in 
the same directory...
Not sure what's causing this on the make install...
 Thank you for any guidance...
/opt/eai/ci/subversion/build/install-sh -c -m 644 
./subversion/svnsync/svnsync.1 
/opt/eai/ci/subversion-1.9.5/svn/share/man/man1/svnsync.1
/opt/eai/ci/subversion/build/install-sh -c -m 644 
./subversion/svnversion/svnversion.1 
/opt/eai/ci/subversion-1.9.5/svn/share/man/man1/svnversion.1
if true ; then cd subversion/mod_dav_svn ; 
/opt/eai/ci/subversion/build/install-sh -c -d 
"/opt/eai/ci/subversion-1.9.5/svn/libexec" ; 
/opt/eai/ci/httpd-2.2.32/apache/bin/apxs -i -S 
LIBEXECDIR="/opt/eai/ci/subversion-1.9.5/svn/libexec"  -n dav_svn 
mod_dav_svn.la ; fi /opt/eai/ci/httpd-2.2.32/apache/build/instdso.sh 
SH_LIBTOOL='/opt/freeware/lib/apr-1/build/libtool' mod_dav_svn.la 
/opt/eai/ci/subversion-1.9.5/svn/libexec
rm -f /opt/eai/ci/subversion-1.9.5/svn/libexec/mod_dav_svn.so
/opt/freeware/lib/apr-1/build/libtool --mode=install cp mod_dav_svn.la 
/opt/eai/ci/subversion-1.9.5/svn/libexec/
libtool: install: warning: relinking `mod_dav_svn.la'
libtool: install: (cd /opt/eai/ci/subversion/subversion/mod_dav_svn; /bin/sh 
"/opt/eai/ci/subversion/libtool"  --tag CC --silent --mode=relink gcc -shared 
-g -O2 -g -O2 -pthread -rpath /opt/eai/ci/subversion-1.9.5/svn/libexec 
-avoid-version -module -o mod_dav_svn.la activity.lo authz.lo deadprops.lo 
liveprops.lo lock.lo merge.lo mirror.lo mod_dav_svn.lo posts/create_txn.lo 
reports/dated-rev.lo reports/deleted-rev.lo reports/file-revs.lo 
reports/get-location-segments.lo reports/get-locations.lo reports/get-locks.lo 
reports/inherited-props.lo reports/log.lo reports/mergeinfo.lo 
reports/replay.lo reports/update.lo repos.lo status.lo util.lo version.lo 
../../subversion/libsvn_repos/libsvn_repos-1.la 
../../subversion/libsvn_fs/libsvn_fs-1.la 
../../subversion/libsvn_delta/libsvn_delta-1.la 
../../subversion/libsvn_subr/libsvn_subr-1.la )
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: '/lib/libexpat.la' seems to be moved
libtool: warning: 

Re: svn hotcopy incremental overwrites existing revisions in backup

[Branko Čibej]

On 16.05.2017 21:28, Stefan Sperling wrote:
> On Tue, May 16, 2017 at 12:02:38PM -0700, lumi wrote:
>> Isn't it a mistake in method of getting file size on deduplicated volume?
> Subversion asks APR (a portability library) for the filesize.
> APR does something to find that size. Subversion uses the value reported
> by APR, and Subversion does not care about how APR figured it out.
>
> So if there is a problem with how the size is determined on Windows with
> NTFS de-duplication enabled, then this problem is probably located in APR
> and should be fixed there. The APR project is at https://apr.apache.org
>
> That said, if you know of a way to find the correct size with the win32 API
> we could probably patch Subversion to bypass APR for this specific case.
> But APR would have to be fixed anyway.

I suspect the ReparsePoint attribute on the file is what actually makes
APR hiccup. A "reparse point" is distantly related to a unix symlink ...
it tells the file-system path resolver to restart with a different path.
I bet that APR reports is the size of the reparse-point record instead
of the size of the target file, but when we open the file we get the
actual file contents.

-- Brane

Re: svn hotcopy incremental overwrites existing revisions in backup

[Stefan Sperling]

On Tue, May 16, 2017 at 12:02:38PM -0700, lumi wrote:
> Isn't it a mistake in method of getting file size on deduplicated volume?

Subversion asks APR (a portability library) for the filesize.
APR does something to find that size. Subversion uses the value reported
by APR, and Subversion does not care about how APR figured it out.

So if there is a problem with how the size is determined on Windows with
NTFS de-duplication enabled, then this problem is probably located in APR
and should be fixed there. The APR project is at https://apr.apache.org

That said, if you know of a way to find the correct size with the win32 API
we could probably patch Subversion to bypass APR for this specific case.
But APR would have to be fixed anyway.

Re: svn hotcopy incremental overwrites existing revisions in backup

[lumi]

Isn't it a mistake in method of getting file size on deduplicated volume?
What I showed on screenshots is what Windows Explorer says. Other
applications shows only one size value, e.g. Powershell Get-ItemProperty
gets only actual size, no matter deduplicated volume or not, and this size
is always the same.
/PS C:\Users\Администратор.WIN-DBM2OE9OJ54> Get-ItemProperty -Path
D:\Repositories\Sandbox\db\revs\0\14


Каталог: D:\Repositories\Sandbox\db\revs\0


ModeLastWriteTime Length Name   
 
- --    
 
-a---l   16.03.2017 13:00 126748 14 
 



PS C:\Users\Администратор.WIN-DBM2OE9OJ54> Get-ItemProperty -Path
Y:\RepositoriesBackup\Daily\Sandbox\db\revs\0\14


Каталог: Y:\RepositoriesBackup\Daily\Sandbox\db\revs\0


ModeLastWriteTime Length Name   
 
- --    
 
-a   16.05.2017 11:08 126748 14/
  

The first one is deduplicated file on local drive, the second is hotcopied
file on WebDav Network drive.



--
View this message in context: 
http://subversion.1072662.n5.nabble.com/svn-hotcopy-incremental-overwrites-existing-revisions-in-backup-tp198977p198990.html
Sent from the Subversion Users mailing list archive at Nabble.com.

Re: svn hotcopy incremental overwrites existing revisions in backup

[Stefan Sperling]

On Tue, May 16, 2017 at 10:57:44AM -0700, lumi wrote:
> Size mismatch is definitly takes place. Actual size is normal, but size on
> disk is a bit unreal, again because of deduplication.
>  

The whole point of a hotcopy is to have a 1-to-1 bit-identical backup.
If the NTFS filesystem which stores the backup is de-duplicating files
in a way that makes their filesize change, then incremental hotcopy
cannot work. By design, incremental hotcopy compares the size and
timestamp to see if a revision file must be copied again.
So if you really want to use svnadmin hotcopy you should disable the
de-duplication feature on the target filesystem.

But there are other tools you could use for backup purposes instead,
such as svnadmin dump/load and svnsync (e.g. with file:// URLs).
These should work fine with NTFS de-duplication enabled on backup storage.
See 
http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.migrate
and 
http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.replication

Re: svn hotcopy incremental overwrites existing revisions in backup

[lumi]

Size mismatch is definitly takes place. Actual size is normal, but size on
disk is a bit unreal, again because of deduplication.
 



--
View this message in context: 
http://subversion.1072662.n5.nabble.com/svn-hotcopy-incremental-overwrites-existing-revisions-in-backup-tp198977p198986.html
Sent from the Subversion Users mailing list archive at Nabble.com.

Re: svn hotcopy incremental overwrites existing revisions in backup

[lumi]

NTFS with deduplication enabled (Windows Server 2016). Problem files have APL
attributes (Archive, SparseFile, ReparsePoint), which means that file takes
part in deduplication I guess. Hotcopy of repository to WebDav network drive
gives exactly the same result. It means that problem in source files.



--
View this message in context: 
http://subversion.1072662.n5.nabble.com/svn-hotcopy-incremental-overwrites-existing-revisions-in-backup-tp198977p198984.html
Sent from the Subversion Users mailing list archive at Nabble.com.

Re: svn hotcopy incremental overwrites existing revisions in backup

[Stefan Sperling]

On Tue, May 16, 2017 at 06:21:06AM -0700, lumi wrote:
> And so on with each next hotcopy --incremental command. Binary comparison
> revision 14, 21, 22 files of original repositary and backup gives equal
> result. What reason of this strange behaviour?

The only possible reasons are a size mismatch or a timestamp mismatch
on the affected files.

Re: svn hotcopy incremental overwrites existing revisions in backup

[Daniel Shahaf]

lumi wrote on Tue, May 16, 2017 at 06:21:06 -0700:
> C:\Users\Администратор.WIN-DBM2OE9OJ54>svnadmin hotcopy
> D:\Repositories\Sandbox D:\Test --incremental
> * Copied revision 14.
> * Copied revision 21.
> * Copied revision 22.
> 
> C:\Users\Администратор.WIN-DBM2OE9OJ54>svnadmin hotcopy
> D:\Repositories\Sandbox D:\Test --incremental
> * Copied revision 14.
> * Copied revision 21.
> * Copied revision 22./
> 
> And so on with each next hotcopy --incremental command. Binary comparison
> revision 14, 21, 22 files of original repositary and backup gives equal
> result. What reason of this strange behaviour?

I can't reproduce this:

% rm -rf r d
% svnadmin create r
% repeat 100 svnmucc put -mm -U file://$PWD/r =(dd if=/dev/urandom bs=1k 
count=200 2>/dev/null) f$RANDOM.$RANDOM  >/dev/null
% svnadmin hotcopy --incremental r d  >/dev/null
% svnadmin hotcopy --incremental r d 
% svnadmin hotcopy --incremental r d 
% svnadmin hotcopy --incremental r d 
% svnadmin hotcopy --incremental r d 
%   
13:39

If you delete D:\Test and run the 'hotcopy' command three more times,
does it say 14, 21, 22 in those times too?

What filesystem is D:?  Is it NTFS, or a network drive, or…?