Re: Multi-WC-format branch: preparing for merge to trunk
Merged to trunk now. Testers, enhancers welcome. More below.
Please try it out if you haven't.
The basics:
wc format 31:
- compatible with svn 1.8 to 1.14 and now 1.15-dev
wc format 32:
- compatible with svn 1.15-dev only,
- as yet introduces no schema changes (we expect those to come when we
merge 'pristines-on-demand-on-mwf' branch to trunk)
There is no supported downgrade, so be careful with testing this. In
this development state you might not want to upgrade an important wc to
compatible-version=1.15. Check out a disposable WC instead.
(In emergency you might get away with:
sqlite3 .svn/wc.db 'pragma user_version=31;'
)
Commands affected:
svn checkout # WC is compatible with 1.8 to 1.14
svn checkout --compatible-version=1.15 # format bump
svn upgrade # upgrade an old (svn<=1.7) WC to f31 (1.8~1.14)
svn upgrade --compatible-version=1.15 # upgrade to 1.15
Anyone would be very welcome to take a crack at any suggested
enhancements, including but not limited to those I listed earlier:
> - The "svn upgrade" command may grow an option to only show the current
>and target working copy versions. (Perhaps `svn upgrade --dry-run`?)
>
> - Let the default "compatible version" be specified in the user
> config. (That is, the default version for 'checkout' and 'upgrade',
> which are the places where a version can be specified.)
>
> - Clarify/unify the option name. svn commands also have other
>compatibility concerns besides the WC format.
>Current option names:
> --compatible-version # for 'svnadmin': repository version
> --compatible-version # for 'svn': WC version
> --wc-format-version # for the tests: WC version
>Suggestions for WC version everywhere ('svn' and tests):
> --wc-compatible-version
> --wc-compatible-format
>
> - Clarify the supported versions display:
> -Supported working copy (WC) versions: from 1.8 to 1.15
> +Supported working copy (WC) versions:
> + --compatible-version=1.8 (supported by svn 1.8 to 1.15; default)
> + --compatible-version=1.15 (supported by svn 1.15)
>
> We might want some of these, or other changes, before release.
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
TL;DR: OK to merge multi-wc-format to trunk soon?
Having worked with it these last few weeks, I think the essential pieces
are in place for multi-wc-format to be merged to trunk.
I am still working on the last pieces of pristines-on-demand rebased on
top of this branch: fixing tests to work properly with both WC versions.
What I am seeing there is all specific to the pristines-on-demand
changes, and doesn't seem to be requiring anything extra of the basic
multi-wc-format capabilities. That suggests this multi-wc-format branch
provides what it needs to provide.
This is my list of possible enhancements to multi-wc-format:
- The "svn upgrade" command may grow an option to only show the current
and target working copy versions. (Perhaps `svn upgrade --dry-run`?)
- Let the default "compatible version" be specified in the user
config. (That is, the default version for 'checkout' and 'upgrade',
which are the places where a version can be specified.)
- Clarify/unify the option name. svn commands also have other
compatibility concerns besides the WC format.
Current option names:
--compatible-version # for 'svnadmin': repository version
--compatible-version # for 'svn': WC version
--wc-format-version # for the tests: WC version
Suggestions for WC version everywhere ('svn' and tests):
--wc-compatible-version
--wc-compatible-format
- Clarify the supported versions display:
-Supported working copy (WC) versions: from 1.8 to 1.15
+Supported working copy (WC) versions:
+ --compatible-version=1.8 (supported by svn 1.8 to 1.15; default)
+ --compatible-version=1.15 (supported by svn 1.15)
We might want some of these, or other changes, before release. I don't
see any of them being needed before merge to trunk. The criterion is
basically, as much as possible should be done on trunk, as I understand
it. Please speak up about any changes needed, and if anything is needed
before merge.
Does anyone want extra time to review the branch, or have any other
concerns prior to merging it? If not I will do some further review of my
own (double-checking API changes and test changes) and then I should
think I could go ahead and merge it in a few days. I'm not meaning to be
pushy, so do let me know if anything doesn't seem right.
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
r1897943 makes the default for "--compatible-version" (and tests) be "1.8". - Julian
Re: Multi-WC-format branch: preparing for merge to trunk
On Thu, Feb 10, 2022 at 7:25 AM Julian Foad wrote: > > It strikes me now, given Subversion's "mature, stable" phase of life, > that we should default to backward compatibility. > > Change the default to --compatible-version=1.8 (same as =1.14), instead > of 1.15 (format 32). Both for "checkout" and "upgrade". The user will > need to specify "--compatible-version=1.15" when they want to use > new-in-1.15-format features (i.e. pristines-on-demand). > > We should let this option be specified in the user config > (file/registry) too. > > Makes sense? Yes, +1 Thanks for considering it. Mark
Re: Multi-WC-format branch: preparing for merge to trunk
It strikes me now, given Subversion's "mature, stable" phase of life, that we should default to backward compatibility. Change the default to --compatible-version=1.8 (same as =1.14), instead of 1.15 (format 32). Both for "checkout" and "upgrade". The user will need to specify "--compatible-version=1.15" when they want to use new-in-1.15-format features (i.e. pristines-on-demand). We should let this option be specified in the user config (file/registry) too. Makes sense? - Julian
Re: Multi-WC-format branch: preparing for merge to trunk
> I think this branch is now ready to merge to trunk. I am now asking for review. To help, here is an overview of the branch changes, compiled from its log messages: Instead of supporting just one format, introduce a current format (the default for new working copies) and a lowest supported format. Change the way new working copies are created: instead of the base schema defining the current format, it defines the lowest supporting format and a series of format updates are performed to bring it to the current shape. Add the --compatible-version option to the checkout and upgrade commands. Guess the format of dependent working copies (e.g., for externals) from the current working copy context. Add a line to the 'svn --version' output that shows supported WC versions. Include a place-holder format bump. This makes it easier to merge with a branch that includes a format bump (such as the 'pristines-on-demand' branch). Add test suite option '--wc-format-version'. This accepts a version string like '1.14.0' and is used for all WC creation, e.g. 'svn checkout/upgrade --compatible-version=X'. Highlights of API-like additions and changes: * subversion/include/svn_client.h (svn_client_checkout4, svn_client_upgrade2): New versions. (svn_client_supported_wc_version): New. * subversion/include/svn_wc.h (svn_wc_info_t): Add new fields: wc_format, wc_format_min, wc_format_max. * subversion/include/private/svn_wc_private.h (svn_wc__format_from_version svn_wc__is_supported_format svn_wc__max_supported_format svn_wc__min_supported_format svn_wc__is_supported_format_version svn_wc__max_supported_format_version svn_wc__min_supported_format_version svn_wc__format_from_context): New. (svn_wc__ensure_adm): New, replaces svn_wc_ensure_adm4. (svn_wc__upgrade): New. * subversion/libsvn_wc/wc-metadata.sql (STMT_CREATE_SCHEMA): Set 'user_version' to the minimum supported version (SVN_WC__SUPPORTED_VERSION), not the latest current version. - Julian
Re: Multi-WC-format branch: preparing for merge to trunk
I think this branch is now ready to merge to trunk.
In a series of commits I have added '--wc-format-version' argument to
the C and Python test suite framework on this branch, and tested that it
works with both '1.15' (the default, WC format 32) and '1.14' (wc format 31).
I tested that the desired format is actually being used in WCs when this
option is passed in to the test suite, using this temporary code:
```
Index: subversion/libsvn_wc/wc_db.c
===
--- subversion/libsvn_wc/wc_db.c (revision 1897914)
+++ subversion/libsvn_wc/wc_db.c (working copy)
@@ -1508,2 +1508,9 @@ svn_wc__db_init(svn_wc__db_t *db,
+#define FORMAT_EXPECTED 31
+ if (wcroot->format != FORMAT_EXPECTED)
+{
+ SVN_DBG(("### format is %d; expected %d", wcroot->format,
FORMAT_EXPECTED));
+ SVN_ERR_MALFUNCTION_NO_RETURN();
+}
+
/* Any previously cached children may now have a new WCROOT, most
likely that
@@ -13464,2 +13471,8 @@ svn_wc__db_upgrade_begin(svn_sqlite__db_
+ if (wcroot->format != FORMAT_EXPECTED)
+{
+ SVN_DBG(("### format is %d; expected %d", wcroot->format,
FORMAT_EXPECTED));
+ SVN_ERR_MALFUNCTION_NO_RETURN();
+}
+
/* The WCROOT is complete. Stash it into DB. */
```
These checks occur after create_db() calls. I think they cover most if
not all places where a WC can be created.
That would seem to be everything we need on this branch.
Any further thoughts about readiness to merge to trunk, Brane or anyone?
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
On 03.02.2022 17:30, Julian Foad wrote:
Julian Foad wrote:
Branko Čibej wrote:
way to report the actual format (from the wc-db) through libsvn-wc to
the client?
Ah yes, there is no API for that, [...]
These exist on the branch currently ("*_supported_*" are new on the branch):
- At libsvn_client (public):
svn_client_version() // e.g. svn_version_t("1.15.0-dev")
svn_client_supported_wc_version() // e.g. svn_version_t("1.8.0")
- In libsvn_wc (internal declarations):
SVN_WC__VERSION // e.g. "32" for svn 1.15-dev
SVN_WC__SUPPORTED_VERSION // e.g. "31" for svn 1.8 through 1.14
- Conversion between WC format number and svn_version_t:
svn_wc__format_from_version() // exposed in svn_wc_private.h
svn_wc__version_string_from_format() // internal only
I suggest:
- Keep the SVN_WC__* constants internal, as they are.
- Expose *both* conversion functions equally as private APIs. No reason
not to, and I think the new --wc-format option in the test suite is
going to need them. (I could wait and see if it does; I'm writing it
now.) One already is in svn_wc_private.h. The other says:
* ### It's not ideal to encode this sort of knowledge in this low-level
* library. On the other hand, it doesn't need to be updated often and
* should be easily found when it does need to be updated. */
which applies equally to both and is fine for private APIs.
- Expose the min and max WC format numbers privately in
svn_wc_private.h. It seems perverse not to, when higher level APIs
exist. Code outside libsvn_wc, such as wc tests, will need or want
these. (I can wait and see if it does. I'm writing it soon.)
svn_wc__format() // e.g. "32" for svn 1.15-dev
svn_wc__supported_format() // e.g. "31" for svn 1.8 through 1.14
Good points. I wanted to avoid widening the libsvn_wc API, but, indeed,
that train is long gone.
There is no libsvn_client API for getting the WC format number directly.
I suppose it makes sense that client-level code is written with
reference to client-level version numbers.
Exactly. libsvn_client needs to know, but its users do not.
-- Brane
Re: Multi-WC-format branch: preparing for merge to trunk
Julian Foad wrote:
> Branko Čibej wrote:
>> way to report the actual format (from the wc-db) through libsvn-wc to
>> the client?
>
> Ah yes, there is no API for that, [...]
These exist on the branch currently ("*_supported_*" are new on the branch):
- At libsvn_client (public):
svn_client_version() // e.g. svn_version_t("1.15.0-dev")
svn_client_supported_wc_version() // e.g. svn_version_t("1.8.0")
- In libsvn_wc (internal declarations):
SVN_WC__VERSION // e.g. "32" for svn 1.15-dev
SVN_WC__SUPPORTED_VERSION // e.g. "31" for svn 1.8 through 1.14
- Conversion between WC format number and svn_version_t:
svn_wc__format_from_version() // exposed in svn_wc_private.h
svn_wc__version_string_from_format() // internal only
I suggest:
- Keep the SVN_WC__* constants internal, as they are.
- Expose *both* conversion functions equally as private APIs. No reason
not to, and I think the new --wc-format option in the test suite is
going to need them. (I could wait and see if it does; I'm writing it
now.) One already is in svn_wc_private.h. The other says:
* ### It's not ideal to encode this sort of knowledge in this low-level
* library. On the other hand, it doesn't need to be updated often and
* should be easily found when it does need to be updated. */
which applies equally to both and is fine for private APIs.
- Expose the min and max WC format numbers privately in
svn_wc_private.h. It seems perverse not to, when higher level APIs
exist. Code outside libsvn_wc, such as wc tests, will need or want
these. (I can wait and see if it does. I'm writing it soon.)
svn_wc__format() // e.g. "32" for svn 1.15-dev
svn_wc__supported_format() // e.g. "31" for svn 1.8 through 1.14
There is no libsvn_client API for getting the WC format number directly.
I suppose it makes sense that client-level code is written with
reference to client-level version numbers.
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Branko Čibej wrote: > I still seem to remember we have a bit missing on the multi-wc-format > branch, but I can't remember what ... maybe we don't currently have a > way to report the actual format (from the wc-db) through libsvn-wc to > the client? Or has that been fixed? Ah yes, there is no API for that, and any new feature built on this will need to know, in general. (The current state of pristines-on-demand doesn't, but maybe it will by the time it's finished.) I will add it. - Julian
Re: Multi-WC-format branch: preparing for merge to trunk
On 02.02.2022 20:49, Julian Foad wrote: [...] a single test [...] It doesn't seem necessary to automate running over the whole test suite with (format < SVN_WC__VERSION) [...] Meh. That's only true for the place-holder code as present on the 'multi-wc-format' branch, where the code reports multiple formats are supported but there are no actual differences between the formats. Any non-trivial feature (such as pristines-on-demand) that works with multiple WC formats will have code paths that access database tables differently depending on the format. I ran into this as soon as I tested 'pristines-on-demand' merged with 'multi-wc-format'. Those code paths need to be tested against both (or multiple) formats. Therefore it IS necessary at least to add a parameter to the test suite framework to specify the WC format that all the tests will use. That should be done as part of this branch. The test suite already has... --server-minor-version=SERVER_MINOR_VERSION --fsfs-version=FSFS_VERSION I will copy that pattern. Sounds good. I still seem to remember we have a bit missing on the multi-wc-format branch, but I can't remember what ... maybe we don't currently have a way to report the actual format (from the wc-db) through libsvn-wc to the client? Or has that been fixed? -- Brane
Re: Multi-WC-format branch: preparing for merge to trunk
> [...] a single test [...] It doesn't seem necessary to automate > running over the whole test suite with (format < SVN_WC__VERSION) [...] Meh. That's only true for the place-holder code as present on the 'multi-wc-format' branch, where the code reports multiple formats are supported but there are no actual differences between the formats. Any non-trivial feature (such as pristines-on-demand) that works with multiple WC formats will have code paths that access database tables differently depending on the format. I ran into this as soon as I tested 'pristines-on-demand' merged with 'multi-wc-format'. Those code paths need to be tested against both (or multiple) formats. Therefore it IS necessary at least to add a parameter to the test suite framework to specify the WC format that all the tests will use. That should be done as part of this branch. The test suite already has... --server-minor-version=SERVER_MINOR_VERSION --fsfs-version=FSFS_VERSION I will copy that pattern. - Julian
