Re: Performance branch ready for review
On Sun, Aug 22, 2010 at 2:56 PM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: Johan Corveleyn wrote: On Wed, Aug 18, 2010 at 9:14 PM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: Hi @all, I just finished my porting work; the performance branch is now fully synchronized with my prototype code. From my point of view, review can start now. According to my measurements, the code is now faster than the original prototype. Large caches provided, a single multi-threaded svnserve instance on a modern quad-core machine should be able to saturate a 10Gb server connection. Open issues / things still to do * there is an issue with log triggering an assertion() - I will investigate that next * test mod_web_dav and add FSFS cache configuration parameters to it * tune membuffer cache eviction strategy such that even small caches can have a large impact * add tests for the new APIs * provide APR patches. There are many things I would like to do but they may better be deferred to 1.8. I tried compiling your branch on Windows (XP) with Visual C++ Express 2008 (which I also use successfully to build trunk). I had a couple of issues. FWIW I'm listing them here. I'm not an expert, just pragmatically trying to get the thing to build, so some of these things may be user error. Eventually, I was able to build svn.exe, but svnadmin.exe and svnserve.exe still fail for me. 1) In build.conf, I added private\svn_temp_serializer.h and private\svn_file_handle_cache.h to the list of msvc-export of libsvn_subr. Otherwise, the linker gave problems: libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__has_file referenced in function _svn_fs_fs__path_rev_absolute libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_stream__from_cached_file_handle referenced in function _get_node_revision_body libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__open referenced in function _get_node_revision_body libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__get_apr_handle referenced in function _open_pack_or_rev_file libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__flush referenced in function _sync_file_handle_cache libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__close referenced in function _svn_fs_fs__rev_get_root libsvn_fs_fs-1.lib(caching.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__create_cache referenced in function _get_global_file_handle_cache libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__pop referenced in function _svn_fs_fs__id_serialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__pop libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__push referenced in function _svn_fs_fs__id_serialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__push libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__add_string referenced in function _serialize_id_private libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__add_string libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__add_string libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_deserializer__resolve referenced in function _svn_fs_fs__id_deserialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_deserializer__resolve libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_deserializer__resolve libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__get referenced in function _svn_fs_fs__serialize_txdelta_window libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__get libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__init referenced in function _svn_fs_fs__serialize_txdelta_window libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__init libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_deserializer__ptr referenced in function _svn_fs_fs__extract_dir_entry libsvn_fs_fs-1.lib(dag.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__set_null referenced in function _svn_fs_fs__dag_serialize ..\..\..\Debug\subversion\libsvn_fs\libsvn_fs-1.dll : fatal error LNK1120: 15 unresolved externals Here's a patch that solves
Re: Performance branch ready for review
Daniel Shahaf wrote: Stefan, you did mention Patch by for Johan's patches which you committed, did you intend to mention Found by or Suggested by for the other two (quoted below)? http://subversion.apache.org/docs/community-guide/conventions.html#crediting Thanks, Daniel Oh, I just was not aware that there are tons of ... by schemes. r987868 and r987869 now rightfully mention Johan. -- Stefan^2.
Re: Performance branch ready for review
On Tue, Aug 24, 2010 at 12:22 AM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: Daniel Shahaf wrote: Stefan, you did mention Patch by for Johan's patches which you committed, did you intend to mention Found by or Suggested by for the other two (quoted below)? http://subversion.apache.org/docs/community-guide/conventions.html#crediting Thanks, Daniel Oh, I just was not aware that there are tons of ... by schemes. r987868 and r987869 now rightfully mention Johan. Thanks. Not terribly important to me, but nice anyway. I just hope some of your performance-work makes it into 1.7. So anything I can do to help ... Cheers, -- Johan
Re: Performance branch ready for review
Johan Corveleyn wrote: On Wed, Aug 18, 2010 at 9:14 PM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: Hi @all, I just finished my porting work; the performance branch is now fully synchronized with my prototype code. From my point of view, review can start now. According to my measurements, the code is now faster than the original prototype. Large caches provided, a single multi-threaded svnserve instance on a modern quad-core machine should be able to saturate a 10Gb server connection. Open issues / things still to do * there is an issue with log triggering an assertion() - I will investigate that next * test mod_web_dav and add FSFS cache configuration parameters to it * tune membuffer cache eviction strategy such that even small caches can have a large impact * add tests for the new APIs * provide APR patches. There are many things I would like to do but they may better be deferred to 1.8. I tried compiling your branch on Windows (XP) with Visual C++ Express 2008 (which I also use successfully to build trunk). I had a couple of issues. FWIW I'm listing them here. I'm not an expert, just pragmatically trying to get the thing to build, so some of these things may be user error. Eventually, I was able to build svn.exe, but svnadmin.exe and svnserve.exe still fail for me. 1) In build.conf, I added private\svn_temp_serializer.h and private\svn_file_handle_cache.h to the list of msvc-export of libsvn_subr. Otherwise, the linker gave problems: libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__has_file referenced in function _svn_fs_fs__path_rev_absolute libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_stream__from_cached_file_handle referenced in function _get_node_revision_body libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__open referenced in function _get_node_revision_body libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__get_apr_handle referenced in function _open_pack_or_rev_file libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__flush referenced in function _sync_file_handle_cache libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__close referenced in function _svn_fs_fs__rev_get_root libsvn_fs_fs-1.lib(caching.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__create_cache referenced in function _get_global_file_handle_cache libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__pop referenced in function _svn_fs_fs__id_serialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__pop libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__push referenced in function _svn_fs_fs__id_serialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__push libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__add_string referenced in function _serialize_id_private libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__add_string libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__add_string libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_deserializer__resolve referenced in function _svn_fs_fs__id_deserialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_deserializer__resolve libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_deserializer__resolve libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__get referenced in function _svn_fs_fs__serialize_txdelta_window libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__get libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__init referenced in function _svn_fs_fs__serialize_txdelta_window libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__init libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_deserializer__ptr referenced in function _svn_fs_fs__extract_dir_entry libsvn_fs_fs-1.lib(dag.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__set_null referenced in function _svn_fs_fs__dag_serialize ..\..\..\Debug\subversion\libsvn_fs\libsvn_fs-1.dll : fatal error LNK1120: 15 unresolved externals Here's a patch that solves this for me: [[[ Index: build.conf === --- build.conf (revision 986928) +++ build.conf (working copy) @@
Re: Performance branch ready for review
Stefan, you did mention Patch by for Johan's patches which you committed, did you intend to mention Found by or Suggested by for the other two (quoted below)? http://subversion.apache.org/docs/community-guide/conventions.html#crediting Thanks, Daniel Stefan Fuhrmann wrote on Sun, Aug 22, 2010 at 14:56:46 +0200: Johan Corveleyn wrote: 4) In temp_serializer.c, some of the sizeof statements gave problems for me: I changed them as follows (not sure if this is correct, but this compiles ok): [[[ Index: subversion/libsvn_fs_fs/temp_serializer.c === --- subversion/libsvn_fs_fs/temp_serializer.c (revision 986928) +++ subversion/libsvn_fs_fs/temp_serializer.c (working copy) @@ -271,7 +271,7 @@ ]]] I think your patch would work in almost all real-world applications. As I understand it, however, [...] Fixed in r987869. I haven't been able to resolve these ... Fixed in r987886.
Performance branch ready for review
Hi @all, I just finished my porting work; the performance branch is now fully synchronized with my prototype code. From my point of view, review can start now. According to my measurements, the code is now faster than the original prototype. Large caches provided, a single multi-threaded svnserve instance on a modern quad-core machine should be able to saturate a 10Gb server connection. Open issues / things still to do * there is an issue with log triggering an assertion() - I will investigate that next * test mod_web_dav and add FSFS cache configuration parameters to it * tune membuffer cache eviction strategy such that even small caches can have a large impact * add tests for the new APIs * provide APR patches. There are many things I would like to do but they may better be deferred to 1.8. -- Stefan^2.
Re: Performance branch ready for review
On Wed, Aug 18, 2010 at 9:14 PM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: Hi @all, I just finished my porting work; the performance branch is now fully synchronized with my prototype code. From my point of view, review can start now. According to my measurements, the code is now faster than the original prototype. Large caches provided, a single multi-threaded svnserve instance on a modern quad-core machine should be able to saturate a 10Gb server connection. Open issues / things still to do * there is an issue with log triggering an assertion() - I will investigate that next * test mod_web_dav and add FSFS cache configuration parameters to it * tune membuffer cache eviction strategy such that even small caches can have a large impact * add tests for the new APIs * provide APR patches. There are many things I would like to do but they may better be deferred to 1.8. I tried compiling your branch on Windows (XP) with Visual C++ Express 2008 (which I also use successfully to build trunk). I had a couple of issues. FWIW I'm listing them here. I'm not an expert, just pragmatically trying to get the thing to build, so some of these things may be user error. Eventually, I was able to build svn.exe, but svnadmin.exe and svnserve.exe still fail for me. 1) In build.conf, I added private\svn_temp_serializer.h and private\svn_file_handle_cache.h to the list of msvc-export of libsvn_subr. Otherwise, the linker gave problems: libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__has_file referenced in function _svn_fs_fs__path_rev_absolute libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_stream__from_cached_file_handle referenced in function _get_node_revision_body libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__open referenced in function _get_node_revision_body libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__get_apr_handle referenced in function _open_pack_or_rev_file libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__flush referenced in function _sync_file_handle_cache libsvn_fs_fs-1.lib(fs_fs.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__close referenced in function _svn_fs_fs__rev_get_root libsvn_fs_fs-1.lib(caching.obj) : error LNK2019: unresolved external symbol _svn_file_handle_cache__create_cache referenced in function _get_global_file_handle_cache libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__pop referenced in function _svn_fs_fs__id_serialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__pop libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__push referenced in function _svn_fs_fs__id_serialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__push libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__add_string referenced in function _serialize_id_private libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__add_string libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__add_string libsvn_fs_fs-1.lib(id.obj) : error LNK2019: unresolved external symbol _svn_temp_deserializer__resolve referenced in function _svn_fs_fs__id_deserialize libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2001: unresolved external symbol _svn_temp_deserializer__resolve libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_deserializer__resolve libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__get referenced in function _svn_fs_fs__serialize_txdelta_window libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__get libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__init referenced in function _svn_fs_fs__serialize_txdelta_window libsvn_fs_fs-1.lib(dag.obj) : error LNK2001: unresolved external symbol _svn_temp_serializer__init libsvn_fs_fs-1.lib(temp_serializer.obj) : error LNK2019: unresolved external symbol _svn_temp_deserializer__ptr referenced in function _svn_fs_fs__extract_dir_entry libsvn_fs_fs-1.lib(dag.obj) : error LNK2019: unresolved external symbol _svn_temp_serializer__set_null referenced in function _svn_fs_fs__dag_serialize ..\..\..\Debug\subversion\libsvn_fs\libsvn_fs-1.dll : fatal error LNK1120: 15 unresolved externals Here's a patch that solves this for me: [[[ Index: build.conf === --- build.conf (revision 986928) +++ build.conf (working copy) @@ -322,6 +322,7 @@
