Re: Format 20 upgrade to NODES
Philip Martin philip.mar...@wandisco.com writes: I'd like to enable NODES as a replacement for BASE_NODE and WORKING_NODE. This would involve bumping the format number, and old working copies would get automatically upgraded. This has been committed to trunk. -- Philip
Re: Format 20 upgrade to NODES
On Wed, 2010-10-06 at 09:32 +0100, Philip Martin wrote: I'd like to enable NODES as a replacement for BASE_NODE and WORKING_NODE. This would involve bumping the format number, and old working copies would get automatically upgraded. +1 from me, ASAP. We're still working on the op_depth support and it's more complex than I originally thought. It looks like doing this transition in two separate format bumps will be more expedient. Please give me 24h to change the order of NODES columns first - see separate email. - Julian This is not the final NODES data model. It currently just uses NODES.op_depth as 0 or 2 to indicate the equivalent of BASE_NODE and WORKING_NODE. Another wc upgrade will be required in the future to enable full op_depth behaviour. The advantages of the proposed upgrade include: dropping the conditional code, having everyone exercise the new code. The disadvantages include: a wc upgrade, the testsuite is slightly (maybe 2% on my machine) slower. If you build the current format 19 with SVN_WC__NODES then both NODES and BASE_NODE/WORKING_NODE tables are created. DB writes modify both NODES and BASE/WORKING and DB reads check that the same data is obtained from NODES and BASE/WORKING. The regression tests pass like this so we have confidence that NODES is an adequate replacement for BASE/WORKING. If you build format 19 with both SVN_WC__NODES and SVN_WC__NODES_ONLY then only the NODES table is written and read, BASE/WORKING remain empty. The upgrade would be using this code, but would also enable the upgrade and dropping of the BASE/WORKING tables. The change would be the patch below: * subversion/libsvn_wc/wc.h (SVN_WC__VERSION): Bump to 20. * subversion/libsvn_wc/wc.h (STMT_UPGRADE_TO_20): Renamed from DISABLED_STMT_UPGRADE_TO_20 and enabled, drop the inherited BASE_NODE format 99 stuff. [...]
Re: Format 20 upgrade to NODES
On Wed, Oct 6, 2010 at 3:32 AM, Philip Martin philip.mar...@wandisco.com wrote: I'd like to enable NODES as a replacement for BASE_NODE and WORKING_NODE. This would involve bumping the format number, and old working copies would get automatically upgraded. This is not the final NODES data model. It currently just uses NODES.op_depth as 0 or 2 to indicate the equivalent of BASE_NODE and WORKING_NODE. Another wc upgrade will be required in the future to enable full op_depth behaviour. The advantages of the proposed upgrade include: dropping the conditional code, having everyone exercise the new code. The disadvantages include: a wc upgrade, the testsuite is slightly (maybe 2% on my machine) slower. Are there any explanations for this behavior? If you build the current format 19 with SVN_WC__NODES then both NODES and BASE_NODE/WORKING_NODE tables are created. DB writes modify both NODES and BASE/WORKING and DB reads check that the same data is obtained from NODES and BASE/WORKING. The regression tests pass like this so we have confidence that NODES is an adequate replacement for BASE/WORKING. If you build format 19 with both SVN_WC__NODES and SVN_WC__NODES_ONLY then only the NODES table is written and read, BASE/WORKING remain empty. The upgrade would be using this code, but would also enable the upgrade and dropping of the BASE/WORKING tables. The change would be the patch below: * subversion/libsvn_wc/wc.h (SVN_WC__VERSION): Bump to 20. * subversion/libsvn_wc/wc.h (STMT_UPGRADE_TO_20): Renamed from DISABLED_STMT_UPGRADE_TO_20 and enabled, drop the inherited BASE_NODE format 99 stuff. Index: subversion/libsvn_wc/wc.h === --- subversion/libsvn_wc/wc.h (revision 1004712) +++ subversion/libsvn_wc/wc.h (working copy) @@ -129,7 +129,7 @@ * Please document any further format changes here. */ -#define SVN_WC__VERSION 19 +#define SVN_WC__VERSION 20 /* Formats = this have no concept of revert text-base/props. */ #define SVN_WC__NO_REVERT_FILES 4 Index: subversion/libsvn_wc/wc-metadata.sql === --- subversion/libsvn_wc/wc-metadata.sql (revision 1004712) +++ subversion/libsvn_wc/wc-metadata.sql (working copy) @@ -987,10 +987,8 @@ /* Format 20 introduces NODES and removes BASE_NODE and WORKING_NODE */ -/* ### Enable this bit and take out the BASE_NODE stuff in format 99 below. +-- STMT_UPGRADE_TO_20 --- DISABLED_STMT_UPGRADE_TO_20 - INSERT INTO NODES SELECT wc_id, local_relpath, 0 AS op_depth, parent_relpath, repos_id, repos_relpath, revnum, @@ -1012,7 +1010,6 @@ DROP TABLE WORKING_NODE; PRAGMA user_version = 20; -*/ /* - */ @@ -1028,81 +1025,6 @@ number will be, however, so we're just marking it as 99 for now. */ -- format: 99 -/* We cannot directly remove columns, so we use a temporary table instead. */ -/* First create the temporary table without the undesired column(s). */ -CREATE TEMPORARY TABLE BASE_NODE_BACKUP( - wc_id INTEGER NOT NULL, - local_relpath TEXT NOT NULL, - repos_id INTEGER, - repos_relpath TEXT, - parent_relpath TEXT, - presence TEXT NOT NULL, - kind TEXT NOT NULL, - revnum INTEGER, - checksum TEXT, - translated_size INTEGER, - changed_rev INTEGER, - changed_date INTEGER, - changed_author TEXT, - depth TEXT, - symlink_target TEXT, - last_mod_time INTEGER, - properties BLOB, - dav_cache BLOB, - file_external TEXT -); - -/* Copy everything into the temporary table. */ -INSERT INTO BASE_NODE_BACKUP SELECT - wc_id, local_relpath, repos_id, repos_relpath, parent_relpath, presence, - kind, revnum, checksum, translated_size, changed_rev, changed_date, - changed_author, depth, symlink_target, last_mod_time, properties, dav_cache, - file_external -FROM BASE_NODE; - -/* Drop the original table. */ -DROP TABLE BASE_NODE; - -/* Recreate the original table, this time less the temporary columns. - Column descriptions are same as BASE_NODE in format 12 */ -CREATE TABLE BASE_NODE( - wc_id INTEGER NOT NULL REFERENCES WCROOT (id), - local_relpath TEXT NOT NULL, - repos_id INTEGER REFERENCES REPOSITORY (id), - repos_relpath TEXT, - parent_relpath TEXT, - presence TEXT NOT NULL, - kind TEXT NOT NULL, - revnum INTEGER, - checksum TEXT, - translated_size INTEGER, - changed_rev INTEGER, - changed_date INTEGER, - changed_author TEXT, - depth TEXT, - symlink_target TEXT, - last_mod_time INTEGER, - properties BLOB, - dav_cache BLOB, - file_external TEXT, - - PRIMARY KEY (wc_id, local_relpath) - ); - -/* Recreate the index. */ -CREATE INDEX I_PARENT ON BASE_NODE (wc_id, parent_relpath); - -/* Copy everything back into the original table. */ -INSERT INTO
Re: Format 20 upgrade to NODES
Hyrum K. Wright hyrum_wri...@mail.utexas.edu writes: philip.mar...@wandisco.com wrote: The disadvantages include: a wc upgrade, the testsuite is slightly (maybe 2% on my machine) slower. Are there any explanations for this behavior? I haven't profiled it. It could be the NODES queries that do op_depth = (SELECT MAX(op_depth) as that's probably more expensive than accessing WORKING_NODE. -- Philip
Re: Format 20 upgrade to NODES
On Wed, Oct 6, 2010 at 1:12 PM, Julian Foad julian.f...@wandisco.com wrote: On Wed, 2010-10-06 at 09:32 +0100, Philip Martin wrote: I'd like to enable NODES as a replacement for BASE_NODE and WORKING_NODE. This would involve bumping the format number, and old working copies would get automatically upgraded. +1 from me, ASAP. We're still working on the op_depth support and it's more complex than I originally thought. It looks like doing this transition in two separate format bumps will be more expedient. Please give me 24h to change the order of NODES columns first - see separate email. +1 from me too. Bye, Erik.
RE: Format 20 upgrade to NODES
-Original Message- From: Erik Huelsmann [mailto:ehu...@gmail.com] Sent: woensdag 6 oktober 2010 17:46 To: Julian Foad Cc: Philip Martin; dev@subversion.apache.org Subject: Re: Format 20 upgrade to NODES On Wed, Oct 6, 2010 at 1:12 PM, Julian Foad julian.f...@wandisco.com wrote: On Wed, 2010-10-06 at 09:32 +0100, Philip Martin wrote: I'd like to enable NODES as a replacement for BASE_NODE and WORKING_NODE. This would involve bumping the format number, and old working copies would get automatically upgraded. +1 from me, ASAP. We're still working on the op_depth support and it's more complex than I originally thought. It looks like doing this transition in two separate format bumps will be more expedient. Please give me 24h to change the order of NODES columns first - see separate email. +1 from me too. +1. Keeping the wc_db code operational on two separate database formats is slowing down development and makes it very hard to really start taking advantage of the new format. Bert
Re: Format 20 upgrade to NODES
What about upgrades from f10 or f19? On Wed, Oct 6, 2010 at 04:32, Philip Martin philip.mar...@wandisco.com wrote: I'd like to enable NODES as a replacement for BASE_NODE and WORKING_NODE. This would involve bumping the format number, and old working copies would get automatically upgraded. This is not the final NODES data model. It currently just uses NODES.op_depth as 0 or 2 to indicate the equivalent of BASE_NODE and WORKING_NODE. Another wc upgrade will be required in the future to enable full op_depth behaviour. The advantages of the proposed upgrade include: dropping the conditional code, having everyone exercise the new code. The disadvantages include: a wc upgrade, the testsuite is slightly (maybe 2% on my machine) slower. If you build the current format 19 with SVN_WC__NODES then both NODES and BASE_NODE/WORKING_NODE tables are created. DB writes modify both NODES and BASE/WORKING and DB reads check that the same data is obtained from NODES and BASE/WORKING. The regression tests pass like this so we have confidence that NODES is an adequate replacement for BASE/WORKING. If you build format 19 with both SVN_WC__NODES and SVN_WC__NODES_ONLY then only the NODES table is written and read, BASE/WORKING remain empty. The upgrade would be using this code, but would also enable the upgrade and dropping of the BASE/WORKING tables. The change would be the patch below: * subversion/libsvn_wc/wc.h (SVN_WC__VERSION): Bump to 20. * subversion/libsvn_wc/wc.h (STMT_UPGRADE_TO_20): Renamed from DISABLED_STMT_UPGRADE_TO_20 and enabled, drop the inherited BASE_NODE format 99 stuff. Index: subversion/libsvn_wc/wc.h === --- subversion/libsvn_wc/wc.h (revision 1004712) +++ subversion/libsvn_wc/wc.h (working copy) @@ -129,7 +129,7 @@ * Please document any further format changes here. */ -#define SVN_WC__VERSION 19 +#define SVN_WC__VERSION 20 /* Formats = this have no concept of revert text-base/props. */ #define SVN_WC__NO_REVERT_FILES 4 Index: subversion/libsvn_wc/wc-metadata.sql === --- subversion/libsvn_wc/wc-metadata.sql (revision 1004712) +++ subversion/libsvn_wc/wc-metadata.sql (working copy) @@ -987,10 +987,8 @@ /* Format 20 introduces NODES and removes BASE_NODE and WORKING_NODE */ -/* ### Enable this bit and take out the BASE_NODE stuff in format 99 below. +-- STMT_UPGRADE_TO_20 --- DISABLED_STMT_UPGRADE_TO_20 - INSERT INTO NODES SELECT wc_id, local_relpath, 0 AS op_depth, parent_relpath, repos_id, repos_relpath, revnum, @@ -1012,7 +1010,6 @@ DROP TABLE WORKING_NODE; PRAGMA user_version = 20; -*/ /* - */ @@ -1028,81 +1025,6 @@ number will be, however, so we're just marking it as 99 for now. */ -- format: 99 -/* We cannot directly remove columns, so we use a temporary table instead. */ -/* First create the temporary table without the undesired column(s). */ -CREATE TEMPORARY TABLE BASE_NODE_BACKUP( - wc_id INTEGER NOT NULL, - local_relpath TEXT NOT NULL, - repos_id INTEGER, - repos_relpath TEXT, - parent_relpath TEXT, - presence TEXT NOT NULL, - kind TEXT NOT NULL, - revnum INTEGER, - checksum TEXT, - translated_size INTEGER, - changed_rev INTEGER, - changed_date INTEGER, - changed_author TEXT, - depth TEXT, - symlink_target TEXT, - last_mod_time INTEGER, - properties BLOB, - dav_cache BLOB, - file_external TEXT -); - -/* Copy everything into the temporary table. */ -INSERT INTO BASE_NODE_BACKUP SELECT - wc_id, local_relpath, repos_id, repos_relpath, parent_relpath, presence, - kind, revnum, checksum, translated_size, changed_rev, changed_date, - changed_author, depth, symlink_target, last_mod_time, properties, dav_cache, - file_external -FROM BASE_NODE; - -/* Drop the original table. */ -DROP TABLE BASE_NODE; - -/* Recreate the original table, this time less the temporary columns. - Column descriptions are same as BASE_NODE in format 12 */ -CREATE TABLE BASE_NODE( - wc_id INTEGER NOT NULL REFERENCES WCROOT (id), - local_relpath TEXT NOT NULL, - repos_id INTEGER REFERENCES REPOSITORY (id), - repos_relpath TEXT, - parent_relpath TEXT, - presence TEXT NOT NULL, - kind TEXT NOT NULL, - revnum INTEGER, - checksum TEXT, - translated_size INTEGER, - changed_rev INTEGER, - changed_date INTEGER, - changed_author TEXT, - depth TEXT, - symlink_target TEXT, - last_mod_time INTEGER, - properties BLOB, - dav_cache BLOB, - file_external TEXT, - - PRIMARY KEY (wc_id, local_relpath) - ); - -/* Recreate the index. */ -CREATE INDEX I_PARENT ON BASE_NODE (wc_id, parent_relpath); - -/* Copy everything back into the original table. */ -INSERT INTO BASE_NODE
Re: Format 20 upgrade to NODES
Greg Stein gst...@gmail.com writes: What about upgrades from f10 or f19? Upgrading from 1.6 works just as well with NODES as with BASE/WORKING, we write NODES with op_depth 0 or 2. Ugrading from wcng 19 to 20 is very simple, we just copy into NODES with op_depth 0 or 2. -- Philip
Re: Format 20 upgrade to NODES
Philip Martin wrote on Wed, Oct 06, 2010 at 19:00:58 +0100: Greg Stein gst...@gmail.com writes: What about upgrades from f10 or f19? Upgrading from 1.6 works just as well with NODES as with BASE/WORKING, we write NODES with op_depth 0 or 2. Ugrading from wcng 19 to 20 is very simple, we just copy into NODES with op_depth 0 or 2. What about older f11-f18 wc's? On Monday I ran into an f16 wc.
Re: Format 20 upgrade to NODES
Daniel Shahaf d...@daniel.shahaf.name writes: Greg Stein gst...@gmail.com writes: What about upgrades from f10 or f19? Upgrading from 1.6 works just as well with NODES as with BASE/WORKING, we write NODES with op_depth 0 or 2. Ugrading from wcng 19 to 20 is very simple, we just copy into NODES with op_depth 0 or 2. What about older f11-f18 wc's? On Monday I ran into an f16 wc. Nothing should have changed. They will auto-upgrade to f18 (you need to do each directory as it hits the error), the single-db upgrade to 19 must be done by a script, and then it will auto-upgrade to 20. -- Philip
Re: Format 20 upgrade to NODES
Philip Martin wrote on Wed, Oct 06, 2010 at 20:31:56 +0100: Daniel Shahaf d...@daniel.shahaf.name writes: Greg Stein gst...@gmail.com writes: What about upgrades from f10 or f19? Upgrading from 1.6 works just as well with NODES as with BASE/WORKING, we write NODES with op_depth 0 or 2. Ugrading from wcng 19 to 20 is very simple, we just copy into NODES with op_depth 0 or 2. What about older f11-f18 wc's? On Monday I ran into an f16 wc. Nothing should have changed. They will auto-upgrade to f18 (you need to do each directory as it hits the error), the single-db upgrade to 19 must be done by a script, and then it will auto-upgrade to 20. Okay, but I can do the f16-f18 upgrade using head of trunk, right? Or do I need to keep an SVN_WC__VERSION=18 build around? -- Philip
Re: Format 20 upgrade to NODES
Daniel Shahaf d...@daniel.shahaf.name writes: Okay, but I can do the f16-f18 upgrade using head of trunk, right? Yes, as far as I know. None of that code has changed. -- Philip
