Re: svn commit: r990537 - /subversion/branches/performance/subversion/libsvn_subr/svn_temp_serializer.c
Johan Corveleyn wrote: On Mon, Sep 6, 2010 at 2:10 AM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: The only way to get closer to understanding the nature of the problem, I've added extensive logging code to the cache. Please run svnserve -M1000 -d --forground -r ... svnserve.log, zip the log file after the programm crashed got terminated and send it to me directly. Ok, here it is in attachment, zipped together with the .log and .dmp files from the crash. The exact log command I executed was: $ svn log svn://localhost/trunk/sourcesEJB/uz/s9server/services/toegang/driver/settings.xml Cheers, Thanks Johan! I finally found the problem and it actually was more likely to hit 32 bit systems then 64 bit ones. The relevant part from the log is this: create cache 1 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:RRI create cache 2 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:DAG create cache 3 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:DIR create cache 4 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:PACK-MANIFEST create cache 5 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:TEXT create cache 6 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:NODEREVS get 6: key=P+, Y3 MD5=52408b02cc58866e204010717f31c0ef hash=42680402 (0,257607) not found set 6: key=P+, Y3 MD5=52408b02cc58866e204010717f31c0ef hash=42680402 (0,257607) 4...@2113312 get 6: key=P+, Y3 MD5=52408b02cc58866e204010717f31c0ef hash=42680402 (0,257607) 4...@2113312 get 2: key=2( /trunk/sourcesEJB/uz/s9server MD5=52408b029f15368c2de222f27705f679 hash=42680402 (0,257607) 4...@2113312 crash As you can see, bucket (0,257607) contains a NODEREV the last get will read it as a DAG node. It only tested the first 4 bytes of the MD5 key hash because the to_find variable in find_entry has been a local array once and has become a pointer later on. sizeof(to_find) was now 4 .. 8 instead of 16. r994956 should fix the problem. -- Stefan^2.
Re: Repeated SQL queries when doing 'svn st'
Philip Martin wrote: Branko Čibej br...@xbc.nu writes: On 06.09.2010 12:16, Philip Martin wrote: To use a per-directory query strategy we would probably have to cache data in memory, although not to the same extent as in 1.6. We should probably avoid having Subversion make status callbacks into the application while a query is in progress, so we would accumulate all the row data and complete the query before making any callbacks. Some sort of private svn_wc__db_node_t to hold the results of the select would probably be sufficient. I wonder if per-directory is really necessary; I guess I'm worrying about the case were the WC tree has lots of directories with few files. Do we not have the whole tree in a single Sqlide DB now? Depending on the schema, it might be possible to load the status information from the database in one single query. Yes, per-tree would probably work but I expect most WCs have more files than directories so the gains over per-dir would be small. One big advantage of doing status per-tree is that it gives a proper snapshot, the tree cannot be modified during the status walk. I'm not pushing per-dir as the final solution, my point is that per-node SQLite queries are not going to be fast enough. There are actually two or three reasons why status should run queries on directory granularity: * directories usually resemble files in that opening them is expensive relative to reading their content * operation can be canceled in a timely manner (may or may not be an issue with huge SQL query results) * maybe: queries for a specific folder may be simpler / faster than for sub-trees (depends on schema) Also, I don't think there is a need to cache query results. Instead, the algorithm should be modified to look like this: dir_status: // get all relevant info; each array sorted by name stat_recorded = sql_query(BASE + recorded change info of dir entries) stat_actual = read_dir() prop_changes = sql_query(find prop changes in dir) // align / merge arrays and send results to client foreach name do recorded= has(stat_recorded,name) ? stat_recorded[name] : NULL; actual = has(stat_actual,name) ? stat_actual[name] : NULL; changed_props = has(prop_changes,name) ? prop_changes[name] : NULL; // compare file content if necessary if (recorded actual needs_content_check(recorded, actual)) actual = check_content(name) send_node_status(recorded, actual, changed_props) Only two SQL queries (give or take) per directory. -- Stefan^2.
Apache Retreat in Hursely, UK
Hi, there are still free slots for the Apache Retreat in Hursely next week: http://sites.google.com/site/apacheretreat/ AFAIK the following Subversion developers will be present: Stephen Butler, Daniel Näslund, Daniel Shahaf, Hyrum Wright, myself. There are still free slots for the event, and the policy has been relaxed from Apache committers only to Apache committers and their invited guests. I'd therefore like to encourage any patch contributors (or potential patch contributors) who'd be willing to attend the event to get in touch so we can invite you. Needless to say, I'd be very happy to meet additional Subversion committers, full or partial, at the event. Stefan
Bindings for svn_auth_get_platform_specific_client_providers missing
Hi folks, while trying to get git-svn to work with gnome-keyring, I found that the (perl) bindings for the svn_auth_get_platform_specific_client_providers function are missing. Or actually, a binding is generated, but it ends with: /* FIXME: Missing argout typemap: svn_auth_get_platform_specific_client_providers arg 1 (apr_array_header_t **) */ SWIG_exception(SWIG_ValueError, svn_auth_get_platform_specific_client_providers is not implemented yet); However, from bindings/swig/INSTALL, I gather The Perl bindings are complete. Does this mean that this function is inentionally left out, of did the function just get added after this INSTALL file was written? :-) Anyway, I had a quick look at implementing this, it seems this needs an argout typemap for the apr_array_header_t **providers typemap. It seems the svn_auth_provider_object_t* type (which is contained in the providers outarg of svn_auth_get_platform_specific_client_providers is already automatically converted to a swig pointer object, so no separate typemap for that is needed. While writing this, I've finally found the include/svn_containers.swg file containing all kinds of typemaps for the apr_*_t types. I'll try poking in there a bit, I'll keep you posted. Gr. Matthijs signature.asc Description: Digital signature
[PATCH] Test case for issue #3471: svn up touches file w/ lock svn:keywords property
Hi, The attached patch adds a test case for issue #3471:svn up touches file w/ lock svn:keywords property and marking it as 'XFail' until the issue is fixed. [[[ Log: Add a test for issue #3471:'svn up touches file w/ lock svn:keywords property' * subversion/tests/cmdline/update_tests.py (update_with_file_lock_and_keywords_property_set): New test case. (test_list): Add update_with_file_lock_and_keywords_property_set and mark it as XFail. Patch by: Vijayaguru G vi...@collab.net ]]] Thanks Regards, Vijayaguru Index: subversion/tests/cmdline/update_tests.py === --- subversion/tests/cmdline/update_tests.py (revision 993600) +++ subversion/tests/cmdline/update_tests.py (working copy) @@ -5720,7 +5720,31 @@ svntest.actions.run_and_verify_update(wc_dir, expected_output, expected_disk, expected_status) +#-- +# Test for issue #3471 'svn up touches file w/ lock svn:keywords property' +# +# Marked as XFail until that issue is fixed. +def update_with_file_lock_and_keywords_property_set(sbox): + update with file lock keywords property set + sbox.build() + + wc_dir = sbox.wc_dir + mu_path = os.path.join(wc_dir, 'A', 'mu') + svntest.main.file_append(mu_path, '$Id$') + svntest.main.run_svn(None, 'ps', 'svn:keywords', 'Id', mu_path) + svntest.main.run_svn(None, 'lock', mu_path) + expected_disk = svntest.main.greek_state.copy() + expected_disk.tweak('A/mu', + contents=expected_disk.desc['A/mu'].contents + '$Id$') + # Issue #3471 manifests itself here; The timestamp of 'mu' gets updated to the time of the last svn up. + svntest.actions.run_and_verify_update(wc_dir, +None, +expected_disk, +None) + + + ### # Run the tests @@ -5792,6 +5816,7 @@ add_moved_file_has_props, XFail(add_moved_file_has_props2), update_with_excluded_subdir, + XFail(update_with_file_lock_and_keywords_property_set) ] if __name__ == '__main__':
Locks on deleted files get broken when a commit fails over DAV
If I lock a file, modify it and commit over DAV then the lock is visible in the pre-commit hook, and if the commit fails the lock remains valid. If I delete the file, rather than modify it, then the lock gets removed before the pre-commit hook runs, and if the commit fails the lock is gone. This leaves a broken lock in the working copy. Would people agree that's a bug? Commit has a --no-unlock option that causes locks to remain after the commit, and this keeps the lock for deleted files as well. So it should be possible for locks on deleted files to behave more like locks on modified files: to persist through pre-commit and not break if the commit fails. -- Philip
Re: Locks on deleted files get broken when a commit fails over DAV
On 09/08/2010 12:53 PM, Philip Martin wrote: If I lock a file, modify it and commit over DAV then the lock is visible in the pre-commit hook, and if the commit fails the lock remains valid. If I delete the file, rather than modify it, then the lock gets removed before the pre-commit hook runs, and if the commit fails the lock is gone. This leaves a broken lock in the working copy. Would people agree that's a bug? I would. And have in the past, too: http://subversion.tigris.org/issues/show_bug.cgi?id=3534 -- C. Michael Pilato cmpil...@collab.net CollabNet www.collab.net Distributed Development On Demand signature.asc Description: OpenPGP digital signature
Re: svn commit: r993544 - in /subversion/branches/javahl-ra/subversion/bindings/javahl: native/ src/org/apache/subversion/javahl/ tests/org/apache/subversion/javahl/
On Tue, Sep 7, 2010 at 5:56 PM, Greg Stein gst...@gmail.com wrote: I don't see a README.branch or whatever it should be named... Thanks for the reminder, I've added one in r995168. What is this branch for, and why is it not on trunk? The purpose is to expose some of the RA-layer APIs in JavaHL. The reason for a branch is that a) I'm unsure how complete this work will be, or its timeline for completion, meaning that b) I'm not comfortable doing it on trunk where it might be released Rather than add more stuff to the JavaHL bindings, and conflate what is already there with this new work, I'd rather have it stay on the branch until it feels ready or we branch 1.7. If there are arguments to the contrary, I'm happy to move it to trunk, however. -Hyrum On Tue, Sep 7, 2010 at 18:00, hwri...@apache.org wrote: Author: hwright Date: Tue Sep 7 22:00:57 2010 New Revision: 993544 URL: http://svn.apache.org/viewvc?rev=993544view=rev Log: On the javahl-ra branch: ...
Re: Bindings for svn_auth_get_platform_specific_client_providers missing
Hi folks, While writing this, I've finally found the include/svn_containers.swg file containing all kinds of typemaps for the apr_*_t types. I'll try poking in there a bit, I'll keep you posted. Using the following patch (which just appends a bit to the end of the file), the svn_auth_get_platform_specific_client_providers becomes available in the perl bindings (and with a small change to git-svn, I can use gnome-keyring from git-svn now :-D). I've copied this code from the typemaps for svn_client_proplist_item_t in svn_client.i, but those seem to contain specific conversion code for Ruby and Python, while the actual pointer type is just preserved in Perl. I'm not completly sure how this would work for Python and Ruby, perhaps someone else can help me out here? Would you considering committing this patch (after completion) to the main repository? Gr. Matthijs Index: subversion-1.6.12dfsg/subversion/bindings/swig/include/svn_containers.swg === --- subversion-1.6.12dfsg.orig/subversion/bindings/swig/include/svn_containers.swg 2010-09-08 20:00:18.0 +0200 +++ subversion-1.6.12dfsg/subversion/bindings/swig/include/svn_containers.swg 2010-09-08 20:00:22.0 +0200 @@ -808,3 +808,13 @@ %typemap(argout) apr_array_header_t **RANGELIST_INOUT = apr_array_header_t **RANGELIST; #endif + +/* --- + Output of apr_array_header_t * svn_auth_provider_object_t * +*/ +#ifdef SWIGPERL +%typemap(argout) apr_array_header_t **providers { + %append_output(svn_swig_pl_convert_array(*$1, +$descriptor(svn_auth_provider_object_t *))); +} +#endif signature.asc Description: Digital signature
Re: Bindings for svn_auth_get_platform_specific_client_providers missing
Hi Hyrym, thanks for your thoughts. Would it be possible to add a test to the Perl bindings tests which exercises this new code? Where do these live? I looked around the tests directory a bit, but I didn't think there were tests for bindings? Additionally, you should probably provide a log message in the standard format [1]. ok. Gr. Matthijs signature.asc Description: Digital signature
Re: problems with serf
On Thu, May 27, 2010 at 2:42 AM, Lieven Govaerts svn...@mobsol.be wrote: Hi, On Mon, May 24, 2010 at 3:51 PM, Lieven Govaerts svn...@mobsol.be wrote: On Sun, May 16, 2010 at 10:38 AM, Stefan Küng tortoise...@gmail.com wrote: Hi, During the last few days, I've changed TSVN to link against the svn trunk. The speed is much better now, so thanks for that. It's not as fast as the 1.6.x branch yet but it's usable. About my problem: serf is now the default DAV lib svn uses. But with serf I get tons of app crashes (serf calls abort() - something I won't discuss here anymore since you all should know my opinion about that). For example: a simple checkout of the svn trunk crashed after about 5 seconds. Subsequent updates did too - I had to run cleanup and restart the update 27 (!!) times until I had the svn trunk on my harddrive. I'm using serf 0.6.1, but the same problem existed with 0.3.1. I updated to 0.6.1 yesterday hoping it would solve the problem, but apparently didn't. I have prepared serf 0.6.2 with this fix at: http://serf.googlecode.com/svn/branches/0.6.x Can anyone of you reporting this problems test if this makes them go away? Checkout takes plenty of memory with ra_serf, way more than when using ra_neon. I suspect these changes didn't make that worse, rather the choice of pools for file-related requests in ra_serf (not the serf library). I'll look into this. Hi Lieven, Did you ever investigate checkout's memory usage with serf? It seems to still be a problem with serf 0.7.0, see http://subversion.tigris.org/issues/show_bug.cgi?id=3684#desc4. I'm seeing some benefits with bumping serf/buckets/allocator.c:STANDARD_NODE_SIZE from 128 to 256, but the results vary...going to dump what I have found thus far into issue #3684. Paul
Re: problems with serf
On Wed, Sep 8, 2010 at 3:00 PM, Paul Burba ptbu...@gmail.com wrote: On Thu, May 27, 2010 at 2:42 AM, Lieven Govaerts svn...@mobsol.be wrote: Hi, On Mon, May 24, 2010 at 3:51 PM, Lieven Govaerts svn...@mobsol.be wrote: On Sun, May 16, 2010 at 10:38 AM, Stefan Küng tortoise...@gmail.com wrote: Hi, During the last few days, I've changed TSVN to link against the svn trunk. The speed is much better now, so thanks for that. It's not as fast as the 1.6.x branch yet but it's usable. About my problem: serf is now the default DAV lib svn uses. But with serf I get tons of app crashes (serf calls abort() - something I won't discuss here anymore since you all should know my opinion about that). For example: a simple checkout of the svn trunk crashed after about 5 seconds. Subsequent updates did too - I had to run cleanup and restart the update 27 (!!) times until I had the svn trunk on my harddrive. I'm using serf 0.6.1, but the same problem existed with 0.3.1. I updated to 0.6.1 yesterday hoping it would solve the problem, but apparently didn't. I have prepared serf 0.6.2 with this fix at: http://serf.googlecode.com/svn/branches/0.6.x Can anyone of you reporting this problems test if this makes them go away? Checkout takes plenty of memory with ra_serf, way more than when using ra_neon. I suspect these changes didn't make that worse, rather the choice of pools for file-related requests in ra_serf (not the serf library). I'll look into this. Hi Lieven, Did you ever investigate checkout's memory usage with serf? It seems to still be a problem with serf 0.7.0, see http://subversion.tigris.org/issues/show_bug.cgi?id=3684#desc4. I'm seeing some benefits with bumping serf/buckets/allocator.c:STANDARD_NODE_SIZE from 128 to 256, but the results vary...going to dump what I have found thus far into issue #3684. http://subversion.tigris.org/issues/show_bug.cgi?id=3684 User lgo changed the following: What|Old value |New value Status|REOPENED |STARTED -- http://subversion.tigris.org/ds/viewMessage.do?dsForumId=463dsMessageId=2656935 To unsubscribe from this discussion, e-mail: [issues-unsubscr...@subversion.tigris.org]. Heh, nice timing.
Re: Bindings for svn_auth_get_platform_specific_client_providers missing
On Wed, Sep 8, 2010 at 1:53 PM, Matthijs Kooijman matth...@stdin.nl wrote: Hi Hyrym, thanks for your thoughts. Would it be possible to add a test to the Perl bindings tests which exercises this new code? Where do these live? I looked around the tests directory a bit, but I didn't think there were tests for bindings? I believe the swig perl tests (invoked by running 'make check-swig-pl') are in subversion/bindings/swig/perl/native/t/ -Hyrum
Re: problems with serf
On Wed, Sep 8, 2010 at 9:00 PM, Paul Burba ptbu...@gmail.com wrote: On Thu, May 27, 2010 at 2:42 AM, Lieven Govaerts svn...@mobsol.be wrote: Hi, On Mon, May 24, 2010 at 3:51 PM, Lieven Govaerts svn...@mobsol.be wrote: On Sun, May 16, 2010 at 10:38 AM, Stefan Küng tortoise...@gmail.com wrote: Hi, During the last few days, I've changed TSVN to link against the svn trunk. The speed is much better now, so thanks for that. It's not as fast as the 1.6.x branch yet but it's usable. About my problem: serf is now the default DAV lib svn uses. But with serf I get tons of app crashes (serf calls abort() - something I won't discuss here anymore since you all should know my opinion about that). For example: a simple checkout of the svn trunk crashed after about 5 seconds. Subsequent updates did too - I had to run cleanup and restart the update 27 (!!) times until I had the svn trunk on my harddrive. I'm using serf 0.6.1, but the same problem existed with 0.3.1. I updated to 0.6.1 yesterday hoping it would solve the problem, but apparently didn't. I have prepared serf 0.6.2 with this fix at: http://serf.googlecode.com/svn/branches/0.6.x Can anyone of you reporting this problems test if this makes them go away? Checkout takes plenty of memory with ra_serf, way more than when using ra_neon. I suspect these changes didn't make that worse, rather the choice of pools for file-related requests in ra_serf (not the serf library). I'll look into this. Hi Lieven, Did you ever investigate checkout's memory usage with serf? It seems to still be a problem with serf 0.7.0, see http://subversion.tigris.org/issues/show_bug.cgi?id=3684#desc4. Yes, that's why started that issue an hour ago :)(although I know see the issue wasn't really assigned to myself.) I'm seeing some benefits with bumping serf/buckets/allocator.c:STANDARD_NODE_SIZE from 128 to 256, but the results vary...going to dump what I have found thus far into issue #3684. The real issue is serf r1388. This rev solved a bucket lifetime issue causing regular crashes, at the cost of changing the mechanism used by ssl to keep decrypted but not yet handled data. The solution I'm looking at now is to undo r1388, and solve the crashes in another way. Lieven
Re: svn commit: r993544 - in /subversion/branches/javahl-ra/subversion/bindings/javahl: native/ src/org/apache/subversion/javahl/ tests/org/apache/subversion/javahl/
I suspected as much, and simply wanted to confirm/prod. Keeping new features from here-on-out on a branch is totally fine with me. Alternatively, placing them under an #ifdef like we did with the obliterate support works well, too. (and yes, I realize Java can't do that very easily...) Cheers, -g On Wed, Sep 8, 2010 at 13:58, Hyrum K. Wright hyrum_wri...@mail.utexas.edu wrote: On Tue, Sep 7, 2010 at 5:56 PM, Greg Stein gst...@gmail.com wrote: I don't see a README.branch or whatever it should be named... Thanks for the reminder, I've added one in r995168. What is this branch for, and why is it not on trunk? The purpose is to expose some of the RA-layer APIs in JavaHL. The reason for a branch is that a) I'm unsure how complete this work will be, or its timeline for completion, meaning that b) I'm not comfortable doing it on trunk where it might be released Rather than add more stuff to the JavaHL bindings, and conflate what is already there with this new work, I'd rather have it stay on the branch until it feels ready or we branch 1.7. If there are arguments to the contrary, I'm happy to move it to trunk, however. -Hyrum On Tue, Sep 7, 2010 at 18:00, hwri...@apache.org wrote: Author: hwright Date: Tue Sep 7 22:00:57 2010 New Revision: 993544 URL: http://svn.apache.org/viewvc?rev=993544view=rev Log: On the javahl-ra branch: ...
Re: svn commit: r993544 - in /subversion/branches/javahl-ra/subversion/bindings/javahl: native/ src/org/apache/subversion/javahl/ tests/org/apache/subversion/javahl/
On Wed, Sep 8, 2010 at 3:31 PM, Greg Stein gst...@gmail.com wrote: I suspected as much, and simply wanted to confirm/prod. No problem, thanks for the prod. Keeping new features from here-on-out on a branch is totally fine with me. Alternatively, placing them under an #ifdef like we did with the obliterate support works well, too. (and yes, I realize Java can't do that very easily...) At some point, I've got some crazy ideas about a unified object model and using a set of C++ bindings as the basis for all the swig stuff, as well as JavaHL. That will * definitely* go on a branch, if only for it's insanity and changes to the build system. -Hyrum
Re: svn commit: r990537 - /subversion/branches/performance/subversion/libsvn_subr/svn_temp_serializer.c
On Wed, Sep 8, 2010 at 12:41 PM, Johan Corveleyn jcor...@gmail.com wrote: On Wed, Sep 8, 2010 at 12:00 PM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: Johan Corveleyn wrote: On Mon, Sep 6, 2010 at 2:10 AM, Stefan Fuhrmann stefanfuhrm...@alice-dsl.de wrote: The only way to get closer to understanding the nature of the problem, I've added extensive logging code to the cache. Please run svnserve -M1000 -d --forground -r ... svnserve.log, zip the log file after the programm crashed got terminated and send it to me directly. Ok, here it is in attachment, zipped together with the .log and .dmp files from the crash. The exact log command I executed was: $ svn log svn://localhost/trunk/sourcesEJB/uz/s9server/services/toegang/driver/settings.xml Cheers, Thanks Johan! I finally found the problem and it actually was more likely to hit 32 bit systems then 64 bit ones. The relevant part from the log is this: create cache 1 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:RRI create cache 2 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:DAG create cache 3 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:DIR create cache 4 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:PACK-MANIFEST create cache 5 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:TEXT create cache 6 prefix = fsfs:8d52877e-3a3d-4b59-a976-c8364f526eee/C:/research/svn/experiment/repos/db:NODEREVS get 6: key=P+, Y3 MD5=52408b02cc58866e204010717f31c0ef hash=42680402 (0,257607) not found set 6: key=P+, Y3 MD5=52408b02cc58866e204010717f31c0ef hash=42680402 (0,257607) 4...@2113312 get 6: key=P+, Y3 MD5=52408b02cc58866e204010717f31c0ef hash=42680402 (0,257607) 4...@2113312 get 2: key=2( /trunk/sourcesEJB/uz/s9server MD5=52408b029f15368c2de222f27705f679 hash=42680402 (0,257607) 4...@2113312 crash As you can see, bucket (0,257607) contains a NODEREV the last get will read it as a DAG node. It only tested the first 4 bytes of the MD5 key hash because the to_find variable in find_entry has been a local array once and has become a pointer later on. sizeof(to_find) was now 4 .. 8 instead of 16. r994956 should fix the problem. That's great news! I'll give it a spin tonight. Yes! My tests pass now, there are no more crashes. It all works fine. Cheers, -- Johan