Re: Do we want 'svn patch' to be able to add empty files?
On 04.09.2010 02:44, Augie Fackler wrote: On Sep 3, 2010, at 7:10 AM, Daniel Näslund wrote: On Fri, Sep 03, 2010 at 12:18:37PM +0200, Branko Čibej wrote: On 02.09.2010 10:50, Branko Čibej wrote: Hmm, this is interesting. :) Git faithfully (blindly?) interprets Unix permission bits, whiles SVN faithfully (blindly?) interprets the contents of special files ... I wonder if svn patch does the right thing here? Anyway, for the sake of interoperability, we'd have to emit and parse the git format for symlinks. Not that I'm too amused by the idea that git probably just does a chmod on the new file without thinking about it, but hey, All the World is Linux, right? :) Did some testing ... apparently git apply completely ignores the permission bits new file mode ... line, at least I haven't been able to force it to do anything with them. From builtin/apply::try_create_file() in the git source code: fd = open(path, O_CREAT | O_EXCL | O_WRONLY, (mode 0100) ? 0777 : 0666); if (fd 0) return -1; Git only checks for the executable bit, AFAIK. Correct, git and hg only store files as 0644 or 0755. Everything else gets handled by the umask on the user's machine. Nice ... that's equivalent to svn:executable. And see, I was right about the magic numbers. :) -- Brane
Re: Do we want 'svn patch' to be able to add empty files?
On 04.09.2010 02:46, Augie Fackler wrote: On Sep 2, 2010, at 4:03 AM, Daniel Näslund wrote: $ svn diff --git Index: empty === diff --git a/trunk/empty b/trunk/empty new directory mode 10644 I'd recommend testing this against hg/git before using it, but it should operate correctly in hg given my recollection of the diff parser. Come to think of it, it should be at least new directory mode 10755 or there might be trouble brewing downstream if anyone ever interprets the execute directory bit in the mode mask ... -- Brane
Re: [PATCH] don't do autoprops on symbolic links
On 03.09.2010 15:08, Stefan Sperling wrote: On Fri, Sep 03, 2010 at 08:58:09PM +0800, Wei-Yin Chen wrote: Sorry, that line should have been matching_filenames = [f for f in matching_filenames if not os.path.islink(dirname+'/'+f)] Hi, thanks for your patch! I think we should use os.sep instead of '/', because os.sep is more portable. os.path.join maybe? -- Brane
Re: [PATCH] don't do autoprops on symbolic links
Hi Wei-Yin, Just thought I would mention that you did not attach your updated patch. Gavin Beau Baumanis E: ga...@thespidernet.com On 04/09/2010, at 2:19 PM, Wei-Yin Chen wrote: Hi Stefan, Thanks for your comment. I've updated the patch accordingly. Regards, Wei-Yin On Fri, Sep 3, 2010 at 9:08 PM, Stefan Sperling s...@elego.de wrote: Hi, thanks for your patch! I think we should use os.sep instead of '/', because os.sep is more portable. Also, please put spaces around operators (a + b, instead of a+b), to keep the style of the script consistent. Thanks, Stefan
RE: Worried about single-db performance
-Original Message- From: justin.erenkra...@gmail.com [mailto:justin.erenkra...@gmail.com] On Behalf Of Justin Erenkrantz Sent: zaterdag 4 september 2010 8:33 To: Greg Stein Cc: Johan Corveleyn; Subversion Development Subject: Re: Worried about single-db performance On Fri, Sep 3, 2010 at 8:39 AM, Greg Stein gst...@gmail.com wrote: It should already be faster. Obviously, that's not the case. I just spent a little bit time with Shark and gdb. A cold run of 'svn st' against Subversion trunk checkouts for 1.6 yields 0.402 seconds and 1.7 is 0.919 seconds. Hot runs for 1.6 are about 0.055 seconds with 1.7 at 0.750 seconds. One striking difference in the perf profile between 1.6 trunk is that we seem to do a larger amount of stat() calls in 1.7. From looking at the traces and code, I *think* svn_wc__db_pdh_parse_local_abspath's call to svn_io_check_special_path may be in play here: SQLite also does a stat call per statement, unless there is already a shared lock open, just to check if there is no other process that opened a transaction. (On Windows this specific stat to check for other processes operating on the same db is the performance killer for svn status: Just this stat takes more than 50% of the total processing). Bert
Re: Worried about single-db performance
On 04.09.2010 11:23, Bert Huijben wrote: -Original Message- From: justin.erenkra...@gmail.com [mailto:justin.erenkra...@gmail.com] On Behalf Of Justin Erenkrantz Sent: zaterdag 4 september 2010 8:33 To: Greg Stein Cc: Johan Corveleyn; Subversion Development Subject: Re: Worried about single-db performance On Fri, Sep 3, 2010 at 8:39 AM, Greg Stein gst...@gmail.com wrote: It should already be faster. Obviously, that's not the case. I just spent a little bit time with Shark and gdb. A cold run of 'svn st' against Subversion trunk checkouts for 1.6 yields 0.402 seconds and 1.7 is 0.919 seconds. Hot runs for 1.6 are about 0.055 seconds with 1.7 at 0.750 seconds. One striking difference in the perf profile between 1.6 trunk is that we seem to do a larger amount of stat() calls in 1.7. From looking at the traces and code, I *think* svn_wc__db_pdh_parse_local_abspath's call to svn_io_check_special_path may be in play here: SQLite also does a stat call per statement, unless there is already a shared lock open, just to check if there is no other process that opened a transaction. (On Windows this specific stat to check for other processes operating on the same db is the performance killer for svn status: Just this stat takes more than 50% of the total processing). Hmmm ... easy solution then, just fork off a process that opens the database and these stats should magically vanish ... :) -- Brane
RE: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES)
-Original Message- From: Greg Stein [mailto:gst...@gmail.com] Sent: zaterdag 4 september 2010 6:28 To: Bert Huijben Cc: Julian Foad; Erik Huelsmann; dev@subversion.apache.org Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES) How confusing are you trying to make this? You're succeeding wonderfully. Well, you are good at helping it more confusing for everybody. We are talking about *adds* and their op_depth value. What the fuck does that have to do with commits? All adds are very simple commit operations. They can all have the same op_depth value. There are no restrictions on adding more or removing them. They are free floating nodes within the various restructuring that copies/moves may have done. You can independently revert any single add. Get back to the original question: adds do not require variant op_depth values. Very simple. Any add under an added parent just uses that parent's op_depth. But not making it use a specific op_depth makes adding the add harder (has to check the parent op), the revert harder (can't just take the op_depth to see that it can be reverted by itself) and the commit (see previous mail). Most of this work will probably be handled by _scan_addition() and specific DB operations like svn_wc__db_(temp_)op_copy(), svn_wc__db_temp_op_make_copy(), etc. I really think all of these would be easier to implement with an explicit (non-inherited) op_depth for each add as that makes it possible to identify the impact of these operations without scanning up all the time. (Just looking at the op_depth would show that scanning up is not needed at all; using the same op_depth implies that you only have to scan one specific parent) But maybe we should look at what Eric finds here, as he is the only one really working on this right now. (Or it must be that I missed your commits) The problem can be solved both ways, but for one of these two use cases this requires more scanning upwards. (Which isn't very cheap for generic use unless we some transaction/lock around the group of statements) Bert
Re: [PATCH] don't do autoprops on symbolic links
Dear Gavin, Thanks. The attachment was in my sent box, but it's absent in the mailing archive. Don't know why. Per Branko's suggestion, I'm using join this time. In case the attachment gets missing again, it is also embedded in the mail body. Regards, Wei-Yin --- svn_apply_autoprops.py.old 2010-09-04 20:16:28.0 +0800 +++ svn_apply_autoprops.py.nolink 2010-09-04 20:17:20.0 +0800 @@ -122,10 +122,12 @@ for autoprops_line in autoprop_lines: fnmatch_str = autoprops_line[0] prop_list = autoprops_line[1] matching_filenames = fnmatch.filter(filenames, fnmatch_str) +matching_filenames = [f for f in matching_filenames \ + if not os.path.islink(os.path.join(dirname, f))] if not matching_filenames: continue for prop in prop_list: command = ['svn', 'propset', prop[0], prop[1]] On Sat, Sep 4, 2010 at 4:08 PM, Gavin Beau Baumanis ga...@thespidernet.comwrote: Hi Wei-Yin, Just thought I would mention that you did not attach your updated patch. Gavin Beau Baumanis E: ga...@thespidernet.com
[PATCH] Support command line options in svn_apply_autoprops.py
Hi,
This patch makes trunk/contrib/client-side/svn_apply_autoprops.py take
command line options.
The usage would be:
svn_apply_autoprops.py [-c config_file] [paths to process...]
The -c option specifies the configuration file name, and it overrides
the setting in environment variable SVN_CONFIG_FILENAME.
The rest of the command line arguments are treated as paths to process.
[[[
Make svn_apply_autoprops.py take command line options.
Option -c for svn configuration filename, and the rest for paths to process.
]]]
Regards,
Wei-Yin
--- svn_apply_autoprops.py.old 2010-09-04 20:16:28.0 +0800
+++ svn_apply_autoprops.py.option 2010-09-04 20:53:47.0 +0800
@@ -8,12 +8,10 @@
# auto-property to a single pathname, this script will apply all
# matching lines to a single pathname.
#
# To do:
# 1) Switch to using the Subversion Python bindings.
-# 2) Allow a command line option to specify the configuration file to
-#load the auto-properties from.
#
# $HeadURL$
# $LastChangedRevision$
# $LastChangedDate$
# $LastChangedBy$
@@ -137,22 +135,39 @@
print 'Command %s failed with exit status %s' \
% (command, status)
sys.exit(1)
def main():
+ from optparse import OptionParser
+ op=OptionParser(usage=%prog [-c config_file] [paths to process...])
+ op.add_option(-c,
+dest=config_filename,action=store,help=SVN
config filename.,metavar=args)
+ options, args = op.parse_args()
+ for p in args:
+if not os.path.exists(p):
+ print Path '%s' doesn't exist. % p
+ sys.exit(1)
+ paths = [p for p in args if os.path.exists(p)]
+ if not paths:
+paths = '.'
+
config_filename = os.path.expandvars(SVN_CONFIG_FILENAME)
+ if options.config_filename:
+config_filename = options.config_filename
try:
fd = file(config_filename)
except IOError:
print Cannot open svn configuration file '%s' for reading: %s \
% (config_filename, sys.exc_value.strerror)
+sys.exit(1)
autoprop_lines = get_autoprop_lines(fd)
fd.close()
autoprop_lines = process_autoprop_lines(autoprop_lines)
- os.path.walk('.', filter_walk, autoprop_lines)
+ for p in paths:
+os.path.walk(p, filter_walk, autoprop_lines)
if __name__ == '__main__':
sys.exit(main())
Re: [PATCH] Support command line options in svn_apply_autoprops.py
Sorry, don't know why the attachment was removed again. Does this
mailing list have a strict Content-Type filter? I'm trying
text/x-patch this time from my linux box. Type
application/octet-stream doesn't seem to pass through.
From web interface of the mailing archive, the format of the inlined
version looks strange, and the indentation seems wrong, so here's
another way to get it:
http://pastebin.com/YWhpnZvF
On Sat, Sep 4, 2010 at 9:05 PM, Wei-Yin Chen wyc...@video.ee.ntu.edu.tw wrote:
Hi,
This patch makes trunk/contrib/client-side/svn_apply_autoprops.py take
command line options.
The usage would be:
svn_apply_autoprops.py [-c config_file] [paths to process...]
The -c option specifies the configuration file name, and it overrides
the setting in environment variable SVN_CONFIG_FILENAME.
The rest of the command line arguments are treated as paths to process.
[[[
Make svn_apply_autoprops.py take command line options.
Option -c for svn configuration filename, and the rest for paths to process.
]]]
Regards,
Wei-Yin
--- svn_apply_autoprops.py.old 2010-09-04 20:16:28.0 +0800
+++ svn_apply_autoprops.py.option 2010-09-04 20:53:47.0 +0800
@@ -8,12 +8,10 @@
# auto-property to a single pathname, this script will apply all
# matching lines to a single pathname.
#
# To do:
# 1) Switch to using the Subversion Python bindings.
-# 2) Allow a command line option to specify the configuration file to
-#load the auto-properties from.
#
# $HeadURL$
# $LastChangedRevision$
# $LastChangedDate$
# $LastChangedBy$
@@ -137,22 +135,39 @@
print 'Command %s failed with exit status %s' \
% (command, status)
sys.exit(1)
def main():
+ from optparse import OptionParser
+ op=OptionParser(usage=%prog [-c config_file] [paths to process...])
+ op.add_option(-c,
+dest=config_filename,action=store,help=SVN config filename.,metavar=args)
+ options, args = op.parse_args()
+ for p in args:
+if not os.path.exists(p):
+ print Path '%s' doesn't exist. % p
+ sys.exit(1)
+ paths = [p for p in args if os.path.exists(p)]
+ if not paths:
+paths = '.'
+
config_filename = os.path.expandvars(SVN_CONFIG_FILENAME)
+ if options.config_filename:
+config_filename = options.config_filename
try:
fd = file(config_filename)
except IOError:
print Cannot open svn configuration file '%s' for reading: %s \
% (config_filename, sys.exc_value.strerror)
+sys.exit(1)
autoprop_lines = get_autoprop_lines(fd)
fd.close()
autoprop_lines = process_autoprop_lines(autoprop_lines)
- os.path.walk('.', filter_walk, autoprop_lines)
+ for p in paths:
+os.path.walk(p, filter_walk, autoprop_lines)
if __name__ == '__main__':
sys.exit(main())
Re: Worried about single-db performance
On Sat, Sep 4, 2010 at 2:23 AM, Bert Huijben b...@qqmail.nl wrote: SQLite also does a stat call per statement, unless there is already a shared lock open, just to check if there is no other process that opened a transaction. (On Windows this specific stat to check for other processes operating on the same db is the performance killer for svn status: Just this stat takes more than 50% of the total processing). Aha. Adding exclusive locking into our pragma [http://www.sqlite.org/pragma.html] calls in svn_sqlite__open: PRAGMA locking_mode=exclusive; brings the time for svn st down from 0.680 to 0.310 seconds. And, yes, the I/O percentages drop dramatically: ~/Applications/svn-trunk/bin/svn st 0.37s user 0.31s system 99% cpu 0.684 total ~/Applications/svn-trunk/bin/svn st 0.26s user 0.05s system 98% cpu 0.308 total I *think* we'd be okay with having Sqlite holding its read/write locks for the duration of our database connection rather than constantly giving it up and refetching it between every read and write operation. As I read the sqlite docs, we should still be able to have shared readers in this model - but, it'd create the case where there were idle shared readers (due to network I/O?) would block an attempted writer. With a normal locking mode, a writer could intercept a reader if it were idle and not active. However, I'd think our other locks would interfere with this anyway...so I think it'd be safe. Thoughts? -- justin
Repeated SQL queries when doing 'svn st'
When compiled with SVN_DEBUG and SQLITE3_DEBUG and 'svn st' against a svn trunk WC, a number of things pop out. We perform 28,062 SQL queries. --- DBG: sqlite.c: 63: sql=select root, uuid from repository where id = 1; --- We execute *this* query (STMT_SELECT_REPOSITORY_BY_ID) 2215 times. Yikes. I think this has to do with svn_wc__db_base_get_info's call to fetch_repos_info. I'd think we'd be able to cache this result. I'll take a stab and see if this reduction saves us any real time. The root and uuid should be constant for an wc_id...right? For each file that we hit the DB for, we execute the following queries: --- DBG: sqlite.c: 63: sql=select repos_id, repos_relpath, presence, kind, revnum, checksum, translated_size, changed_rev, changed_date, changed_author, depth, symlink_target, last_mod_time, properties from base_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select presence, kind, checksum, translated_size, changed_rev, changed_date, changed_author, depth, symlink_target, copyfrom_repos_id, copyfrom_repos_path, copyfrom_revnum, moved_here, moved_to, last_mod_time, properties from working_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select prop_reject, changelist, conflict_old, conflict_new, conflict_working, tree_conflict_data, properties from actual_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select base_node.repos_id, base_node.repos_relpath, presence, kind, revnum, checksum, translated_size, changed_rev, changed_date, changed_author, depth, symlink_target, last_mod_time, properties, lock_token, lock_owner, lock_comment, lock_date from base_node left outer join lock on base_node.repos_id = lock.repos_id and base_node.repos_relpath = lock.repos_relpath where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select presence, kind, checksum, translated_size, changed_rev, changed_date, changed_author, depth, symlink_target, copyfrom_repos_id, copyfrom_repos_path, copyfrom_revnum, moved_here, moved_to, last_mod_time, properties from working_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select prop_reject, changelist, conflict_old, conflict_new, conflict_working, tree_conflict_data, properties from actual_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select root, uuid from repository where id = 1; DBG: sqlite.c: 63: sql=select prop_reject, changelist, conflict_old, conflict_new, conflict_working, tree_conflict_data, properties from actual_node where wc_id = 1 and local_relpath = 'contrib/server-side'; DBG: sqlite.c: 63: sql=SELECT properties, presence FROM WORKING_NODE WHERE wc_id = 1 AND local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select properties from base_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select properties from actual_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=SELECT properties, presence FROM WORKING_NODE WHERE wc_id = 1 AND local_relpath = 'contrib/server-side/fsfsverify.py'; DBG: sqlite.c: 63: sql=select properties from base_node where wc_id = 1 and local_relpath = 'contrib/server-side/fsfsverify.py'; --- Notably, AFAICT, we're repeating a few of these queries: - STMT_SELECT_WORKING_NODE (2 times) - STMT_SELECT_ACTUAL_NODE (3 times) - STMT_SELECT_WORKING_PROPS (2 times) - STMT_SELECT_BASE_PROPS (2 times) I haven't yet dug into why we're repeating the queries. So, I'd bet we can cut our volume of SQL calls dramatically with some minor rejiggering not to lose the values when we do the first calls of the statement. -- justin
Re: Repeated SQL queries when doing 'svn st'
On Sat, Sep 4, 2010 at 10:18 AM, Justin Erenkrantz jus...@erenkrantz.com wrote: When compiled with SVN_DEBUG and SQLITE3_DEBUG and 'svn st' against a svn trunk WC, a number of things pop out. We perform 28,062 SQL queries. --- DBG: sqlite.c: 63: sql=select root, uuid from repository where id = 1; --- We execute *this* query (STMT_SELECT_REPOSITORY_BY_ID) 2215 times. Yikes. I think this has to do with svn_wc__db_base_get_info's call to fetch_repos_info. I'd think we'd be able to cache this result. I'll take a stab and see if this reduction saves us any real time. The root and uuid should be constant for an wc_id...right? It's actually svn_wc__db_read_info's fetch_repos_info call... With 2215 queries: ~/Applications/svn-trunk-no-debug/bin/svn st 0.26s user 0.05s system 98% cpu 0.311 total With a quick-and-hacky cache: ~/Applications/svn-trunk-no-debug/bin/svn st 0.25s user 0.05s system 98% cpu 0.298 total It's worth a good 4% time savings... A quick back-of-the-envelope calculation says that if we can remove all of the extraneous 13,290 SQL queries (out of 28,062 ; leaving behind 14,772 queries) - we will likely gain something like 25% from the 0.311 down to around 0.233 seconds. It's still much higher than 0.050 than 'svn st' on 1.6.x yields, but inching closer... -- justin
Re: Repeated SQL queries when doing 'svn st'
On Sat, Sep 4, 2010 at 10:18 AM, Justin Erenkrantz jus...@erenkrantz.com wrote: Notably, AFAICT, we're repeating a few of these queries: - STMT_SELECT_WORKING_NODE (2 times) - STMT_SELECT_ACTUAL_NODE (3 times) - STMT_SELECT_WORKING_PROPS (2 times) - STMT_SELECT_BASE_PROPS (2 times) I haven't yet dug into why we're repeating the queries. Okay - I now know why we're repeating the core queries twice. In get_dir_status, we want to do a check to identify if the node exists and what kind it is - which is done by a call to svn_wc__db_read_info (around line 1269 in status.c). But, most of the parameters (except for node and kind) are NULL. If it's not excluded and we can go into the depth, then we call handle_dir_entry on the entry a few lines down - which turns right around and calls svn_wc__db_read_info - this time asking for everything. This causes the core per-file queries to be executed twice. I'm going to see what a quick check to retrieve just the kind and status will do for the query volume. I think it's unlikely we have to pull everything out of sqlite to answer that basic question. -- justin
Re: Repeated SQL queries when doing 'svn st'
On Sat, Sep 4, 2010 at 12:45 PM, Justin Erenkrantz
jus...@erenkrantz.com wrote:
I'm going to see what a quick check to retrieve just the kind and
status will do for the query volume. I think it's unlikely we have to
pull everything out of sqlite to answer that basic question. --
justin
We can reduce the query volume by one (per file) as we can skip the
active table query - see quick dirty patch below.
It does replace 2 larger queries with 2 smaller ones, but I think our
bottleneck is likely more around query volume than query complexity...
Anyway, I'll stop replying to myself and enjoy the long weekend. =) -- justin
Index: status.c
===
--- status.c(revision 992534)
+++ status.c(working copy)
@@ -1263,10 +1270,7 @@ get_dir_status(const struct walk_status_baton *wb,
svn_wc__db_status_t node_status;
svn_wc__db_kind_t node_kind;
- SVN_ERR(svn_wc__db_read_info(node_status, node_kind, NULL, NULL,
- NULL, NULL, NULL, NULL, NULL, NULL, NULL,
- NULL, NULL, NULL, NULL, NULL, NULL, NULL,
- NULL, NULL, NULL, NULL, NULL, NULL,
+ SVN_ERR(svn_wc__db_read_info_exist(node_status, node_kind,
wb-db, node_abspath, iterpool, iterpool));
if (node_status != svn_wc__db_status_not_present
Index: wc_db.c
===
--- wc_db.c (revision 992534)
+++ wc_db.c (working copy)
@@ -51,7 +51,6 @@
#include private/svn_wc_private.h
#include private/svn_token.h
-
#define NOT_IMPLEMENTED() SVN__NOT_IMPLEMENTED()
@@ -5051,6 +5050,161 @@ svn_wc__db_temp_op_delete(svn_wc__db_t *db,
svn_error_t *
+svn_wc__db_read_info_exist(svn_wc__db_status_t *status,
+ svn_wc__db_kind_t *kind,
+ svn_wc__db_t *db,
+ const char *local_abspath,
+ apr_pool_t *result_pool,
+ apr_pool_t *scratch_pool)
+{
+ svn_wc__db_pdh_t *pdh;
+ const char *local_relpath;
+ svn_sqlite__stmt_t *stmt_base;
+ svn_sqlite__stmt_t *stmt_work;
+ svn_boolean_t have_base;
+ svn_boolean_t have_work;
+ svn_error_t *err = NULL;
+
+ SVN_ERR_ASSERT(svn_dirent_is_absolute(local_abspath));
+
+ SVN_ERR(svn_wc__db_pdh_parse_local_abspath(pdh, local_relpath, db,
+ local_abspath, svn_sqlite__mode_readonly,
+ scratch_pool, scratch_pool));
+ VERIFY_USABLE_PDH(pdh);
+
+ SVN_ERR(svn_sqlite__get_statement(stmt_base, pdh-wcroot-sdb,
+STMT_SELECT_BASE_NODE_EXIST));
+ SVN_ERR(svn_sqlite__bindf(stmt_base, is,
+pdh-wcroot-wc_id, local_relpath));
+ SVN_ERR(svn_sqlite__step(have_base, stmt_base));
+
+ SVN_ERR(svn_sqlite__get_statement(stmt_work, pdh-wcroot-sdb,
+STMT_SELECT_WORKING_NODE_EXIST));
+ SVN_ERR(svn_sqlite__bindf(stmt_work, is,
+pdh-wcroot-wc_id, local_relpath));
+ SVN_ERR(svn_sqlite__step(have_work, stmt_work));
+
+ if (have_base || have_work)
+{
+ svn_wc__db_kind_t node_kind;
+
+ if (have_work)
+node_kind = svn_sqlite__column_token(stmt_work, 1, kind_map);
+ else
+node_kind = svn_sqlite__column_token(stmt_base, 1, kind_map);
+
+ if (status)
+{
+ if (have_base)
+{
+ *status = svn_sqlite__column_token(stmt_base, 0, presence_map);
+
+ /* We have a presence that allows a WORKING_NODE override
+ (normal or not-present), or we don't have an override. */
+ /* ### for now, allow an override of an incomplete BASE_NODE
+ ### row. it appears possible to get rows in BASE/WORKING
+ ### both set to 'incomplete'. */
+ SVN_ERR_ASSERT((*status != svn_wc__db_status_absent
+ *status != svn_wc__db_status_excluded
+ /* *status != svn_wc__db_status_incomplete */)
+ || !have_work);
+
+#ifndef SVN_WC__SINGLE_DB
+ if (node_kind == svn_wc__db_kind_subdir
+ *status == svn_wc__db_status_normal)
+{
+ /* We should have read a row from the subdir wc.db. It
+ must be obstructed in some way.
+
+ It is also possible that a WORKING node will override
+ this value with a proper status. */
+ *status = svn_wc__db_status_obstructed;
+}
+#endif
+}
+
+ if (have_work)
+{
+ svn_wc__db_status_t work_status;
+
+ work_status = svn_sqlite__column_token(stmt_work, 0,
+ presence_map);
+
Re: svn commit: r990537 - /subversion/branches/performance/subversion/libsvn_subr/svn_temp_serializer.c
On Tue, Aug 31, 2010 at 11:35 PM, Johan Corveleyn jcor...@gmail.com wrote: Stefan, On Tue, Aug 31, 2010 at 10:07 PM, Johan Corveleyn jcor...@gmail.com wrote: Some additional info: - I couldn't reproduce the crash with a narrow range. Not even 9:0 would crash it (right after startup). - BUT: if after 9:0 I run log without -r arguments, I get an error on the client side: [[[ ..\..\..\subversion\svn\log-cmd.c:746: (apr_err=160013) ..\..\..\subversion\libsvn_client\log.c:606: (apr_err=160013) ..\..\..\subversion\libsvn_repos\log.c:1474: (apr_err=160013) svn: File not found: revision 90799, path '?\239' ..\..\..\subversion\libsvn_repos\log.c:372: (apr_err=160013) svn: File not found: revision 90799, path '?\239' ..\..\..\subversion\libsvn_fs_fs\tree.c:3313: (apr_err=160013) svn: File not found: revision 90799, path '?\239' ..\..\..\subversion\libsvn_fs_fs\tree.c:3313: (apr_err=160013) svn: File not found: revision 90799, path '?\239' ..\..\..\subversion\libsvn_fs_fs\tree.c:3159: (apr_err=160013) svn: File not found: revision 90799, path '?\239' ..\..\..\subversion\libsvn_fs_fs\tree.c:668: (apr_err=160013) svn: File not found: revision 90799, path '?\239' ]]] - This also happens when the first run is 6:0 or even 42000:0. If the first run is 41000:0, then the second run doesn't get the client-side error, but the server crashes on the expected spot (after rev 42100). - The above client-side error also happens if the second run is 96000:9 instead of a log without -r argument. - However, if I run log -r96000:9 right after startup, no problem. - Other than that, it crashes reproducibly after 42100 if I run log with no -r arguments right after startup. I experimented some more. There must be something strange with that revision 90799 that also causes the client-side error. Some log runs immediately after startup: - svn log -r90798:0 svn://localhost/path/bigfile.xml: no crash - svn log -r90799:0 svn://localhost/path/bigfile.xml: crash (last log entry: 42104 (one before 42100 of the regular crash)) - svn log -r90921:0 svn://localhost/path/bigfile.xml: crash (last log entry: 42130 (two before 42100 of the regular crash)). r90921 is one before 90799. - svn log -r90998:0 svn://localhost/path/bigfile.xml: crash (last log entry: 42149 (three before 42100 of the regular crash)). r90998 is two before 90799. - svn log svn://localhost/path/bigfile.xml: still crashes consistently with last log entry 42100. Still r90799 itself seems a very normal commit, with only text modifications to bigfile.xml. Stefan, If needed, I could send some individual rev files (like r90779 and 42100) via private email. If that would help in finding the problem ... Cheers, -- Johan
Re: [PATCH] Support command line options in svn_apply_autoprops.py
Try renaming the file as a .txt ? -Hyrum On Sat, Sep 4, 2010 at 8:43 AM, Wei-Yin Chen wyc...@video.ee.ntu.edu.tw wrote: Sorry, don't know why the attachment was removed again. Does this mailing list have a strict Content-Type filter? I'm trying text/x-patch this time from my linux box. Type application/octet-stream doesn't seem to pass through. From web interface of the mailing archive, the format of the inlined version looks strange, and the indentation seems wrong, so here's another way to get it: http://pastebin.com/YWhpnZvF On Sat, Sep 4, 2010 at 9:05 PM, Wei-Yin Chen wyc...@video.ee.ntu.edu.tw wrote: Hi, This patch makes trunk/contrib/client-side/svn_apply_autoprops.py take command line options. The usage would be: svn_apply_autoprops.py [-c config_file] [paths to process...] The -c option specifies the configuration file name, and it overrides the setting in environment variable SVN_CONFIG_FILENAME. The rest of the command line arguments are treated as paths to process. [[[ Make svn_apply_autoprops.py take command line options. Option -c for svn configuration filename, and the rest for paths to process. ]]] Regards, Wei-Yin
Re: Repeated SQL queries when doing 'svn st'
On 04.09.2010 21:45, Justin Erenkrantz wrote: On Sat, Sep 4, 2010 at 10:18 AM, Justin Erenkrantz jus...@erenkrantz.com wrote: Notably, AFAICT, we're repeating a few of these queries: - STMT_SELECT_WORKING_NODE (2 times) - STMT_SELECT_ACTUAL_NODE (3 times) - STMT_SELECT_WORKING_PROPS (2 times) - STMT_SELECT_BASE_PROPS (2 times) I haven't yet dug into why we're repeating the queries. Okay - I now know why we're repeating the core queries twice. In get_dir_status, we want to do a check to identify if the node exists and what kind it is - which is done by a call to svn_wc__db_read_info (around line 1269 in status.c). But, most of the parameters (except for node and kind) are NULL. If it's not excluded and we can go into the depth, then we call handle_dir_entry on the entry a few lines down - which turns right around and calls svn_wc__db_read_info - this time asking for everything. This causes the core per-file queries to be executed twice. I'm going to see what a quick check to retrieve just the kind and status will do for the query volume. I think it's unlikely we have to pull everything out of sqlite to answer that basic question. -- justin Possibly this existence check could be one single query for the whole WC and the results cached in memory? There shouldn't be a significant difference in per-query overhead, and you need all those results in any case for a whole-depth status. Of course it increases memory usage, but really ... I can't see that as terribly significant. $ sudo find -x / -print | wc 775161 1091167 81342644 80 megs of file metadata on my box with some 120 gigs of stuff and OS install on it, I doubt even a fairly large working copy would do worse than that. -- Brane
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES)
On Sat, Sep 4, 2010 at 4:54 AM, Bert Huijben b...@qqmail.nl wrote: -Original Message- From: Greg Stein [mailto:gst...@gmail.com] Sent: zaterdag 4 september 2010 6:28 To: Bert Huijben Cc: Julian Foad; Erik Huelsmann; dev@subversion.apache.org Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES) How confusing are you trying to make this? You're succeeding wonderfully. Well, you are good at helping it more confusing for everybody. ... But maybe we should look at what Eric finds here, as he is the only one really working on this right now. (Or it must be that I missed your commits) All, This is a discussion about a tough problem. Ad hominem arguments muddle the waters even more, and discourage contributions from internal and external participants. Let's focus on the technical problem, and not the failings of the participants. Thanks, -Hyrum
