Re: svn copy --pin-externals pins relative paths to non existing revision/path
On Fri, Feb 11, 2022 at 09:22:25AM +0100, Martin Obermeir wrote:
> Hi,
>
> When using `svn copy --pin-externals` to create a tag, svn:externals with a
> relative path get pinned a non existing path: They get pinned to the
> revision before the commit, but the path doesn't exist yet in that revision.
> I expected them to be pinned to the revision which gets created during the
> svn copy operation.
>
> In the example below:
> actual result: A/D - ../B/lambda@2 lambda
> expected result: A/D - ../B/lambda@3 lambda
>
> Please find the script to reproduce this attached. Commented excerpt:
>
> cd A/D/
> svn propset svn:externals "../B/lambda lambda" . # relative path
> svn commit -m "msg1" # creates revision 2
> cd ../..
> svn copy --pin-externals ${URL}/trunk ${URL}/tags/pinnedExternals -m
> "msg2" # creates revision 3
> svn switch ^/tags/pinnedExternals
> # gives warnings:
> # svn: warning: W205011: Error handling externals definition for
> 'A/D/lambda':
> # svn: warning: W17: URL
> 'file:///tmp/pinExternalsBug/repos/tags/pinnedExternals/A/B/lambda' at
> revision 2 doesn't exist
> svn propget -R svn:externals
> # gives: A/D - ../B/lambda@2 lambda
> # i.e. pinned to revision 2, but the path doesn't exist (created in
> revision 3)
> # I would expect: A/D - ../B/lambda@3 lambda
>
>
> Tested on Kubuntu 20.04.3 LTS and SUSE Linux Enterprise Server 15 SP3
> (x86_64) with:
> - todays trunk (Revision: 1897960): svn, version 1.15.0-dev (under
> development) compiled Feb 11 2022, 08:01:11 on x86_64-pc-linux-gnu
> - svn, version 1.14.1 (r1886195)
> - svn, version 1.13.0 (r1867053) compiled Mar 24 2020, 12:33:36 on
> x86_64-pc-linux-gnu
> - svn, version 1.10.6 (r1863367)
>
> Best regards
> Martin
Hi Martin,
Thank you for this report. This is a problem which was already present in
the commit which added this feature (https://svn.apache.org/r1659395).
I am afraid it is impossible to implement a solution for the issue as
you have presented it because your desired solution has an inherent
chicken-and-egg problem:
The external definition is created by the client as part of creating the
new commit. At this time, the new revision number is not yet known.
The client only learns about the new revision number after the commit has
been created on the server. And the client must prepare properties which
contain external definitions as part of creating the commit. So there is
an inherent contradiction between the behaviour you would like to see and
the way the system works internally.
The revision number cannot even be inferred as something like HEAD+1 because
the server can process commits from multiple clients in parallel while clients
upload data, assigning a new number to whichever commit finishes first.
The best suggestion I have is to only refer to already existing source paths
in externals definitions added with your commit which creates your new tag.
This means instead of using a relative external that points inside the path
space of your new tag, use a path that points into trunk or some other branch
where the same files and directories already exist. Because any tag created
by a URL-URL copy is a copy of something that already exists somewhere else
in the repository, this should always be possible, should it not?
Once the tag has been created and the new revision number is known, you
could then make a subsequent commit which manually pins the external to a
path which now exists inside the tag's path space, if you prefer that.
Does this make sense?
The behaviour exposed by your script is certainly not ideal.
In the URL-URL copy case the client does not check whether the pinned
paths even exist!
The client could at least check whether the path is is pinning does in
fact exist. It could either error out if the path cannot be found, or
simply leave the external alone and avoid pinning it.
Verifying externals in general is not an easy problem. If an external
definition is wrong or no longer working due to circumstances such as
URLs having become unusable, SVN will error out when trying to use the
external. There is little we can do about that, as it is somewhat
inherent in the design of the externals feature. Any external that is
valid today could become invalid tomorrow.
Regards,
Stefan
Re: Help with FAIL: wc-queries-test 3: test query expectations
Daniel Shahaf wrote on Fri, Feb 11, 2022 at 17:39:42 +: > Julian Foad wrote on Fri, Feb 11, 2022 at 14:48:28 +: > > On 'pristines-on-demand-on-mwf': > ⋮ > > I could do with some help on a SQL test FAIL, when testing against the > > older WC format, in wc-queries-test 3: test_query_expectations(). It > > looks like this: > > > > $ (SRC_DIR=$PWD; cd obj-dir/subversion/tests/libsvn_wc && > > ./wc-queries-test --cleanup > > --config-file=$SRC_DIR/subversion/tests/tests.conf > > --srcdir=$SRC_DIR/subversion/tests/libsvn_wc --wc-format-version=1.14 3 > > | sed "s#$SRC_DIR/##") > > > > subversion/tests/libsvn_wc/wc-queries-test.c:815: > > (apr_err=SVN_ERR_TEST_FAILED) > > svn_tests: E26: STMT_SELECT_UNREFERENCED_PRISTINES: Uses pristine with > > only 0 index component: ((null)) > > SELECT checksum FROM pristine WHERE refcount = 0 > > subversion/tests/libsvn_wc/wc-queries-test.c:868: > > (apr_err=SVN_ERR_COMPOSED_ERROR) > > svn_tests: E200042: Additional errors: > > subversion/tests/libsvn_wc/wc-queries-test.c:868: > > (apr_err=SVN_ERR_SQLITE_CONSTRAINT) > > svn_tests: E200035: |SCAN TABLE pristine > > FAIL: wc-queries-test 3: test query expectations > > > > The failure is within: > > if (item->search > > && ((item->expression_vars < 2 && is_node_table(item->table)) > > || (item->expression_vars < 1)) > > && !is_result_table(item->table)) > > > > I have taken a look and cannot work out what it means. > > In short, it's checking whether SQLite is using sufficiently many index > columns. STMT_SELECT_UNREFERENCED_PRISTINES does a table scan without > using an index, meaning that query does O(N) work where N is the number > of rows in the table. > > To confirm that, does the test pass if you do «CREATE INDEX foo ON > pristine(refcount)» when creating the SQLite database the test runs on? > > I haven't looked to see whether that's the right fix. That would depend > on whether STMT_SELECT_UNREFERENCED_PRISTINES needs to be performant, > and if it does, on what kind of index would best serve it without > affecting other queries (e.g., INSERTs and DELETEs to that table). > > > Maybe the statement needs to be made to match the exception clause > > "in_list(primary_key_statements, i)", but that's just a semi-educated guess. > > The statements in that list both have «LIMIT 1» on them. That'd be why > they're exempted from the "all queries must use an index" check. > > > > > Excuse brevity. Looked a bit more. The branch removes STMT_SELECT_UNREFERENCED_PRISTINES from slow_statements[] unconditionally and then creates I_PRISTINE_UNREFERENCED only for f32 wc's. So, just add that statement to that array for f31 wc's. Daniel
Re: Help with FAIL: wc-queries-test 3: test query expectations
Julian Foad wrote on Fri, Feb 11, 2022 at 14:48:28 +: > On 'pristines-on-demand-on-mwf': ⋮ > I could do with some help on a SQL test FAIL, when testing against the > older WC format, in wc-queries-test 3: test_query_expectations(). It > looks like this: > > $ (SRC_DIR=$PWD; cd obj-dir/subversion/tests/libsvn_wc && > ./wc-queries-test --cleanup > --config-file=$SRC_DIR/subversion/tests/tests.conf > --srcdir=$SRC_DIR/subversion/tests/libsvn_wc --wc-format-version=1.14 3 > | sed "s#$SRC_DIR/##") > > subversion/tests/libsvn_wc/wc-queries-test.c:815: > (apr_err=SVN_ERR_TEST_FAILED) > svn_tests: E26: STMT_SELECT_UNREFERENCED_PRISTINES: Uses pristine with > only 0 index component: ((null)) > SELECT checksum FROM pristine WHERE refcount = 0 > subversion/tests/libsvn_wc/wc-queries-test.c:868: > (apr_err=SVN_ERR_COMPOSED_ERROR) > svn_tests: E200042: Additional errors: > subversion/tests/libsvn_wc/wc-queries-test.c:868: > (apr_err=SVN_ERR_SQLITE_CONSTRAINT) > svn_tests: E200035: |SCAN TABLE pristine > FAIL: wc-queries-test 3: test query expectations > > The failure is within: > if (item->search > && ((item->expression_vars < 2 && is_node_table(item->table)) > || (item->expression_vars < 1)) > && !is_result_table(item->table)) > > I have taken a look and cannot work out what it means. In short, it's checking whether SQLite is using sufficiently many index columns. STMT_SELECT_UNREFERENCED_PRISTINES does a table scan without using an index, meaning that query does O(N) work where N is the number of rows in the table. To confirm that, does the test pass if you do «CREATE INDEX foo ON pristine(refcount)» when creating the SQLite database the test runs on? I haven't looked to see whether that's the right fix. That would depend on whether STMT_SELECT_UNREFERENCED_PRISTINES needs to be performant, and if it does, on what kind of index would best serve it without affecting other queries (e.g., INSERTs and DELETEs to that table). > Maybe the statement needs to be made to match the exception clause > "in_list(primary_key_statements, i)", but that's just a semi-educated guess. The statements in that list both have «LIMIT 1» on them. That'd be why they're exempted from the "all queries must use an index" check. > Excuse brevity. Daniel
Re: A two-part vision for Subversion and large binary objects.
Julian Foad wrote: > * Re-base pristines-on-demand on top of multi-wc-format. Done: branch 'pristines-on-demand-on-mwf'. > * Make pristines-on-demand behaviour conditional on WC format. Mostly done in r1897977. I could do with some help on a SQL test failure (below), please. Quoting from that log message: - With 'make check WC_FORMAT_VERSION=1.15': test suite still passes. - With 'make check [WC_FORMAT_VERSION=1.8]': some tests FAIL or XPASS: XPASS: authz_tests.py 31: remove a subdir with authz file XPASS: basic_tests.py 8: basic corruption detection on commit [[Relies on wc.text_base_path()]] XPASS: revert_tests.py 2: revert reexpands manually contracted keyword XPASS: trans_tests.py 1: commit new files with keywords active from birth [[Relies on wc.text_base_path()]] XPASS: trans_tests.py 3: committing eol-style change forces text send [[Relies on wc.text_base_path()]] XPASS: update_tests.py 83: missing tmp update caused segfault [[The error message has changed]] XPASS: upgrade_tests.py 16: upgrade with base and working replaced files [[Can't fetch pristines: the working copy points to file:///tmp/repo]] XPASS: upgrade_tests.py 34: automatic SQLite ANALYZE FAIL: wc-queries-test 3: test query expectations >From a quick look, the XPASSes may be just a matter of '@Wimp' annotations that had been added on one of these branches, now being out of date, or something like that. I will check them. I have posted separately asking for help with the FAIL in test_query_expectations(). - Julian
Help with FAIL: wc-queries-test 3: test query expectations
Re. pristines-on-demand. I have re-based pristines-on-demand on top of multi-wc-format, resulting in a new branch 'pristines-on-demand-on-mwf'. On 'pristines-on-demand-on-mwf': > * Make pristines-on-demand behaviour conditional on WC format. Mostly done in r1897977. I could do with some help on a SQL test FAIL, when testing against the older WC format, in wc-queries-test 3: test_query_expectations(). It looks like this: $ (SRC_DIR=$PWD; cd obj-dir/subversion/tests/libsvn_wc && ./wc-queries-test --cleanup --config-file=$SRC_DIR/subversion/tests/tests.conf --srcdir=$SRC_DIR/subversion/tests/libsvn_wc --wc-format-version=1.14 3 | sed "s#$SRC_DIR/##") subversion/tests/libsvn_wc/wc-queries-test.c:815: (apr_err=SVN_ERR_TEST_FAILED) svn_tests: E26: STMT_SELECT_UNREFERENCED_PRISTINES: Uses pristine with only 0 index component: ((null)) SELECT checksum FROM pristine WHERE refcount = 0 subversion/tests/libsvn_wc/wc-queries-test.c:868: (apr_err=SVN_ERR_COMPOSED_ERROR) svn_tests: E200042: Additional errors: subversion/tests/libsvn_wc/wc-queries-test.c:868: (apr_err=SVN_ERR_SQLITE_CONSTRAINT) svn_tests: E200035: |SCAN TABLE pristine FAIL: wc-queries-test 3: test query expectations The failure is within: if (item->search && ((item->expression_vars < 2 && is_node_table(item->table)) || (item->expression_vars < 1)) && !is_result_table(item->table)) I have taken a look and cannot work out what it means. Maybe the statement needs to be made to match the exception clause "in_list(primary_key_statements, i)", but that's just a semi-educated guess. Can anyone more familiar with SQL or that test help out here please? - Julian
svn copy --pin-externals pins relative paths to non existing revision/path
Hi,
When using `svn copy --pin-externals` to create a tag, svn:externals
with a relative path get pinned a non existing path: They get pinned to
the revision before the commit, but the path doesn't exist yet in that
revision. I expected them to be pinned to the revision which gets
created during the svn copy operation.
In the example below:
actual result: A/D - ../B/lambda@2 lambda
expected result: A/D - ../B/lambda@3 lambda
Please find the script to reproduce this attached. Commented excerpt:
cd A/D/
svn propset svn:externals "../B/lambda lambda" . # relative path
svn commit -m "msg1" # creates revision 2
cd ../..
svn copy --pin-externals ${URL}/trunk ${URL}/tags/pinnedExternals -m
"msg2" # creates revision 3
svn switch ^/tags/pinnedExternals
# gives warnings:
# svn: warning: W205011: Error handling externals definition for
'A/D/lambda':
# svn: warning: W17: URL
'file:///tmp/pinExternalsBug/repos/tags/pinnedExternals/A/B/lambda' at
revision 2 doesn't exist
svn propget -R svn:externals
# gives: A/D - ../B/lambda@2 lambda
# i.e. pinned to revision 2, but the path doesn't exist (created in
revision 3)
# I would expect: A/D - ../B/lambda@3 lambda
Tested on Kubuntu 20.04.3 LTS and SUSE Linux Enterprise Server 15 SP3
(x86_64) with:
- todays trunk (Revision: 1897960): svn, version 1.15.0-dev (under
development) compiled Feb 11 2022, 08:01:11 on x86_64-pc-linux-gnu
- svn, version 1.14.1 (r1886195)
- svn, version 1.13.0 (r1867053) compiled Mar 24 2020, 12:33:36 on
x86_64-pc-linux-gnu
- svn, version 1.10.6 (r1863367)
Best regards
Martin
repro-template.sh
Description: application/shellscript
