Re: [darcs-devel] testing: tmp repo names and parallel testing

[Mark Stosberg]

>> Sat Jan 19 22:56:51 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>>   * Major Perl test suite clean-up.
>>   The primary purpose of this patch was make sure all the tests are 
>> executed in
>>   randomly named directories, which allows us to run Perl tests in 
>> parallel,
>>   without the directory names collided.
>>
>>   This isn't enabled by default for "make test", but it is there to play 
>> with.
>>   In the test directory, you can now do:
>>
>>   ./bin/prove -j9 *.pl
>>
>>   to run 9 tests in parallel. There is also "--fork"
>>   option which should be a win on multi-CPU computers.
>>   See "perldoc ./bin/prove" for details.
>>
>>   As part of this, a lot of boiler-plate code at the top and bottom of 
>> the
>>   scripts could be eliminated, and I made few other minor style clean-ups
>>   while I had the files open.
>>
>>   There should be no functional changes to the tests.
>
> I'm not sure I like this idea, as it seems likely to lead to greater
> difficulty in understanding and debugging failing test results.  We can
> already run up to six tests in parallel just by running make -j6 test.

Unless I misunderstand how this would work, it is quite likely both
tests would have tried to create a repo named "temp1" in the same
directory at the same time, and there would be a conflict.

> Also, why make the directory names random?

It's the standard tempfile/tempdir way to name temporary files and
directories.

> Couldn't we just generate a
> unique directory name from the test name itself? So then init_tmp_repo() in
> test foo.pl would generate a repository named temp1-foo.

I have now sent a patch for something like this, that, so that the
directory name is like "add.pl-tmpdir-", where the  become
random characters.

I discovered I need to go back and update the Perl tests again to add a
final "chdir" to them, so that the auto-deletion of the directories can
work properly.

There is a way to disable the deleting of the directories if you want to
debug them, which I'll also try to document.

Also, I'm not necessarily advocating turning parallel tests by default,
but I'm interesting in supporting it as an option.

Here's another useful new testing technique to try:

./bin/prove --directives *.pl

That will show you just tests marked "TODO" or "SKIP". Because it has
been hard to report on these in the past, the remaining TODO tests have
been largely forgetten. The "bugs/" directory serves the same purpose
now.  With an improved harness for the shell tests, the same script and
techniques could work on them, too.

Mark

-- 
http://mark.stosberg.com/


___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue611] wish: --debug, --debug-verbose and --timings should be moved to "Advanced Help"

[Mark Stosberg]

New submission from Mark Stosberg <[EMAIL PROTECTED]>:

I think the --debug, --debug-verbose and --timings flags should be moved out of
the regular help and into the "Advanced Help". Like many of the other "advanced"
options, they are not related to the specific command and are common for many
commands. 

Also, "debug" commands should hopefully be infrequently needed.

--
messages: 2709
nosy: beschmi, droundy, kowey, markstos, tommy
priority: wishlist
status: unread
title: wish: --debug, --debug-verbose and --timings should be moved to 
"Advanced Help"

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue610] wish: hide advanced with --help-advanced

[Mark Stosberg]

New submission from Mark Stosberg <[EMAIL PROTECTED]>:

When I do "darcs record -h", the most basic and fundamental details about how it
works immediately scroll off the screen, and I instead see the "Advanced
options" which is far less likely to be what I need, and is generally the same
for all the commands. 

I would suggest that with --help, the last line is just: 

"Help for advanced options is available through --help-advanced"

Running "--help-advanced" would behave as --help does now, showing the regular
and advanced help. 

Besides the scrolling issue, I think provider shorter, more relevant help by
default will increase the impression that darcs is simple and easy to use.

--
messages: 2708
nosy: beschmi, droundy, kowey, markstos, tommy
priority: wishlist
status: unread
title: wish: hide advanced with --help-advanced

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] darcs patch: update some ChangeLog entries to also cr... (and 1 more)

[Mark Stosberg]

Tue Jan 22 18:54:35 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
  * update some ChangeLog entries to also credit those who contributed through 
bug reporting, test writing or feedback. 

Tue Jan 22 19:04:17 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
  * update restrictive perms test to run a temporary directory and clean up 
after itself.
  Running in a tru temporary directory allows the potential to run tests in 
parallel.

New patches:

[update some ChangeLog entries to also credit those who contributed through bug reporting, test writing or feedback. 
Mark Stosberg <[EMAIL PROTECTED]>**20080122235435] {
hunk ./changelog.in/entries/droundy 16
-> a file that has an unrecorded mv.  (David Roundy)
+> a file that has an unrecorded mv.  (David Roundy, Mark Stosberg, Tuomo Valkonen)
hunk ./changelog.in/entries/droundy 20
-> Fix issue 600, bug in darcs optimize --relink.  (David Roundy)
+> Fix issue 600, bug in darcs optimize --relink.  (David Roundy, Trent Buck, Mark Stosberg, Tommy Pettersson)
}

[update restrictive perms test to run a temporary directory and clean up after itself.
Mark Stosberg <[EMAIL PROTECTED]>**20080123000417
 Running in a tru temporary directory allows the potential to run tests in parallel.
] {
hunk ./tests/restrictive_upstream_permissions.pl 22
+# Start in a randomly named temporary directory that deletes itself when we're done. 
+use File::Temp 'tempdir';
+chdir tempdir( CLEANUP => 1 );
+
hunk ./tests/restrictive_upstream_permissions.pl 30
-
-cleanup 'tmp_remote';
-cleanup 'tmp_restrictive';
-
hunk ./tests/restrictive_upstream_permissions.pl 50
-
-ok(-d 'tmp_remote', "tmp_remote exists here") || `ls -l`;
-rm_rf('tmp_remote');
-ok((!-d 'tmp_remote'), 'tmp_remote directory was deleted');
-
-ok(-d 'tmp_restrictive', "tmp_restrictive exists here");
hunk ./tests/restrictive_upstream_permissions.pl 52
-rm_rf('tmp_restrictive');
-ok((!-d 'tmp_restrictive'), 'tmp_restictive directory was deleted');
-
}

Context:

[reenable mandatory sha1 checks, now that we can link with a faster sha1.
David Roundy <[EMAIL PROTECTED]>**20080123203104] 
[remove (broken) git support and add openssl sha1 support.
David Roundy <[EMAIL PROTECTED]>**20080123202025
 These two changes got merged together as I was introducing the configure.ac
 changes to support openssl as a sha1 alternative to our Haskell code.
 
 (Yes, I'm lazy.)
] 
[remove redundant hash checks in hashed IO code.
David Roundy <[EMAIL PROTECTED]>**20080123173022] 
[output nicer progress in convert.
David Roundy <[EMAIL PROTECTED]>**20080123170428] 
[output timings when --timings is specified.
David Roundy <[EMAIL PROTECTED]>**20080123170314] 
[remove inaccurate message in convert.
David Roundy <[EMAIL PROTECTED]>**20080123170243] 
[use debugMessage in HashedIO.
David Roundy <[EMAIL PROTECTED]>**20080123160835] 
[add --timings flag (that as yet does nothing).
David Roundy <[EMAIL PROTECTED]>**20080123154931] 
[add regression test for amend-record removed file
Tommy Pettersson <[EMAIL PROTECTED]>**2008013231] 
[use UTC in date matching test untill match handles time zones
Tommy Pettersson <[EMAIL PROTECTED]>**20080122134322] 
[fix bug with timestamps and obliterate.
David Roundy <[EMAIL PROTECTED]>**2008014607] 
[Test: unpull may hide changes when using timestamp optimisation.
[EMAIL PROTECTED] 
[avoid printing totals that are less than our current progress.
David Roundy <[EMAIL PROTECTED]>**20080122210546] 
[TAG 2.0.0pre3
David Roundy <[EMAIL PROTECTED]>**20080122200612] 
[fix bug in restrictive_upstream_permissions test, and mark it as passing.
David Roundy <[EMAIL PROTECTED]>**20080122200429] 
[match against any repo (not just last) in determining version/state.
David Roundy <[EMAIL PROTECTED]>**20080122194501] 
[add a few changelog entries.
David Roundy <[EMAIL PROTECTED]>**20080122194307] 
[avoid a withCurrentDirectory if there's nothing to test.
David Roundy <[EMAIL PROTECTED]>**20080122192426] 
[Libwww: fix invalid read reported by valgrind.
Dmitry Kurochkin <[EMAIL PROTECTED]>**20080122162936] 
[fix obliterate to work even on old patches before tags.
David Roundy <[EMAIL PROTECTED]>**20080122150520] 
[make match_a_patch and friends return true for all patches if no match is specified.
David Roundy <[EMAIL PROTECTED]>**20080122150210] 
[fix changes_moved_files bug.
David Roundy <[EMAIL PROTECTED]>**20080122141421] 
[mark optimize-relink as passing now.
David Roundy <[EMAIL PROTECTED]>**20080121191822] 
[don't compate pending and pending.tentative in optimize_relink.sh.
David Roundy <[EMAIL PROTECTED]>**20080121191656] 
[add patch to repo in optimize_relink.sh.
David Roundy <[EMAIL PROTECTED]>**20080121191642] 
[handle relative sibling directory properly.
David Roundy <[EMAIL PROTECTED]>**20080121191225] 
[simplify optimize --relink test, always calling directory temp.
David Roundy <[EMAIL PROTECTED]>**20080121182231] 
[if fs doesn't support hard links, we should pass the relink test.
David Roundy <[EMAIL PROTECTED]>**200801211

[darcs-devel] darcs patch: issue608: a new test for 'mv', following Zooko's bug r...

[Mark Stosberg]

Wed Jan 23 20:38:56 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
  * issue608: a new test for 'mv', following Zooko's bug report

New patches:

[issue608: a new test for 'mv', following Zooko's bug report
Mark Stosberg <[EMAIL PROTECTED]>**20080124013856] {
hunk ./tests/mv.pl 6
-use Test::More tests => 12;
+use Test::More tests => 13;
hunk ./tests/mv.pl 132
+# issue608
+{
+   touch 'gonna_be_deleted';
+   darcs "add gonna_be_deleted";
+   darcs "record -am 'added doomed file'";
+   rm_rf "gonna_be_deleted"; 
+   darcs "record -am 'deleted file'"; 
+   touch 'new_file';
+   darcs "add new_file";
+   my $out = darcs "mv new_file gonna_be_deleted";
+   is($out, "", "darcs mv should succeed");
+}
+
+
+
}

Context:

[reenable mandatory sha1 checks, now that we can link with a faster sha1.
David Roundy <[EMAIL PROTECTED]>**20080123203104] 
[remove (broken) git support and add openssl sha1 support.
David Roundy <[EMAIL PROTECTED]>**20080123202025
 These two changes got merged together as I was introducing the configure.ac
 changes to support openssl as a sha1 alternative to our Haskell code.
 
 (Yes, I'm lazy.)
] 
[remove redundant hash checks in hashed IO code.
David Roundy <[EMAIL PROTECTED]>**20080123173022] 
[output nicer progress in convert.
David Roundy <[EMAIL PROTECTED]>**20080123170428] 
[output timings when --timings is specified.
David Roundy <[EMAIL PROTECTED]>**20080123170314] 
[remove inaccurate message in convert.
David Roundy <[EMAIL PROTECTED]>**20080123170243] 
[use debugMessage in HashedIO.
David Roundy <[EMAIL PROTECTED]>**20080123160835] 
[add --timings flag (that as yet does nothing).
David Roundy <[EMAIL PROTECTED]>**20080123154931] 
[add regression test for amend-record removed file
Tommy Pettersson <[EMAIL PROTECTED]>**2008013231] 
[use UTC in date matching test untill match handles time zones
Tommy Pettersson <[EMAIL PROTECTED]>**20080122134322] 
[fix bug with timestamps and obliterate.
David Roundy <[EMAIL PROTECTED]>**2008014607] 
[Test: unpull may hide changes when using timestamp optimisation.
[EMAIL PROTECTED] 
[avoid printing totals that are less than our current progress.
David Roundy <[EMAIL PROTECTED]>**20080122210546] 
[TAG 2.0.0pre3
David Roundy <[EMAIL PROTECTED]>**20080122200612] 
[fix bug in restrictive_upstream_permissions test, and mark it as passing.
David Roundy <[EMAIL PROTECTED]>**20080122200429] 
[match against any repo (not just last) in determining version/state.
David Roundy <[EMAIL PROTECTED]>**20080122194501] 
[add a few changelog entries.
David Roundy <[EMAIL PROTECTED]>**20080122194307] 
[avoid a withCurrentDirectory if there's nothing to test.
David Roundy <[EMAIL PROTECTED]>**20080122192426] 
[Libwww: fix invalid read reported by valgrind.
Dmitry Kurochkin <[EMAIL PROTECTED]>**20080122162936] 
[fix obliterate to work even on old patches before tags.
David Roundy <[EMAIL PROTECTED]>**20080122150520] 
[make match_a_patch and friends return true for all patches if no match is specified.
David Roundy <[EMAIL PROTECTED]>**20080122150210] 
[fix changes_moved_files bug.
David Roundy <[EMAIL PROTECTED]>**20080122141421] 
[mark optimize-relink as passing now.
David Roundy <[EMAIL PROTECTED]>**20080121191822] 
[don't compate pending and pending.tentative in optimize_relink.sh.
David Roundy <[EMAIL PROTECTED]>**20080121191656] 
[add patch to repo in optimize_relink.sh.
David Roundy <[EMAIL PROTECTED]>**20080121191642] 
[handle relative sibling directory properly.
David Roundy <[EMAIL PROTECTED]>**20080121191225] 
[simplify optimize --relink test, always calling directory temp.
David Roundy <[EMAIL PROTECTED]>**20080121182231] 
[if fs doesn't support hard links, we should pass the relink test.
David Roundy <[EMAIL PROTECTED]>**20080121181647] 
[Libwww new API: waitUrl, copyUrlFirst added; copyUrls removed.
Dmitry Kurochkin <[EMAIL PROTECTED]>**20080120215022] 
[avoid overwriting prompt when waiting for input.
David Roundy <[EMAIL PROTECTED]>**20080121162814] 
[add withoutProgress to halt progress messages for a bit.
David Roundy <[EMAIL PROTECTED]>**20080121162513] 
[add some changelog entries
Tommy Pettersson <[EMAIL PROTECTED]>**20080120003041] 
[print progress messages to stderr.
David Roundy <[EMAIL PROTECTED]>**20080121154515
 Note that to get clean results, we still need to print overwrite spaces to
 stdout, so I'm not sure this is the best of plans.
] 
[make changes show no progess unless debug mode is on.
David Roundy <[EMAIL PROTECTED]>**20080121151145] 
[make progress report quieter.
David Roundy <[EMAIL PROTECTED]>**20080119171250] 
[no progress reports when running quietly.
David Roundy <[EMAIL PROTECTED]>**20080119165947] 
[change directory name in which we run tests.
David Roundy <[EMAIL PROTECTED]>**20080119165251] 
[issue600: bug fix refinement sent on behalf of twb
Mark Stosberg <[EMAIL PROTECTED]>**20080121040050] 
[issue600: a test for optimize --relink
Mark Stosberg <[EMAIL PROTECTED]>**20080121034028] 
[issue562: Update the .sh scripts and th

Re: [darcs-devel] [Haskell-cafe] Re: announcing darcs 2.0.0pre3

[zooko]

> In
> principle it is good to provide a cryptographically secure hash, as  
> this
> allows users to sign their repositories by signing a single file,  
> which
> seems like it's potentially quite a useful feature.

Can you be more specific about this -- who can "sign" a repository?   
How is such a signature checked?  What guarantee can you rely on if  
the check passes?

As far as I know, it is impossible to use a hash value in darcs to  
securely denote a specific patch, because darcs patches do not have a  
canonical form.  In fact, last time I checked the patch *contents*  
didn't even go into the input to SHA-1, just the patch metadata  
(timestamp, author, patch description).

I don't think SHA-1 is necessarily a bad choice if you need a secure  
hash function (although I would like a better argument than "Linus  
chose it."), but I don't understand why we need a secure hash function.

(If you need a secure hash function, Tiger is probably stronger than,  
and is 150% as fast as, SHA-1.)

Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [Haskell-cafe] Re: announcing darcs 2.0.0pre3

[David Roundy]

On Wed, Jan 23, 2008 at 02:55:06PM -0700, zooko wrote:
> I have to ask: why does darcs use SHA-1?
> 
> On the one hand, SHA-1 is cryptographically fragile and is deprecated  
> for use in applications that require collision-resistance and pre- 
> image resistance.  SHA-2 is the current standard for those  
> applications (SHA-2 is about twice as expensive in CPU [1]), and  
> SHA-3 is under development.
> 
> On the other hand, why does darcs need a cryptographically secure  
> hash function at all?  Wouldn't MD5 or a sufficiently wide CRC, such  
> as the one used in ZFS [2], do just as well?  They would certainly be  
> a lot faster to compute.
> 
> Is there some behavior on the part of some malicious actor that darcs  
> tries to prevent, such that the collision-resistance (such as it is)  
> of SHA-1 is necessary to prevent it?

It's mostly historical, but also supported by the assumption that Linus
thought about it when *he* decided to use sha1 for the same purpose.  In
principle it is good to provide a cryptographically secure hash, as this
allows users to sign their repositories by signing a single file, which
seems like it's potentially quite a useful feature.  On the other hand,
using sha2, which is twice as expensive (and twice as large, right) would
perhaps be too costly.  I don't know.  SHA-2 would cost more in disk space
and network bandwidth, as well as in CPU time.

Is SHA-1 optimal? I don't know.  Is it reasonable? I suspect so.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [Haskell-cafe] Re: announcing darcs 2.0.0pre3

[zooko]

I have to ask: why does darcs use SHA-1?

On the one hand, SHA-1 is cryptographically fragile and is deprecated  
for use in applications that require collision-resistance and pre- 
image resistance.  SHA-2 is the current standard for those  
applications (SHA-2 is about twice as expensive in CPU [1]), and  
SHA-3 is under development.

On the other hand, why does darcs need a cryptographically secure  
hash function at all?  Wouldn't MD5 or a sufficiently wide CRC, such  
as the one used in ZFS [2], do just as well?  They would certainly be  
a lot faster to compute.

Is there some behavior on the part of some malicious actor that darcs  
tries to prevent, such that the collision-resistance (such as it is)  
of SHA-1 is necessary to prevent it?

Regards,

Zooko

[1] http://cryptopp.com/benchmarks.html
[2] http://blogs.sun.com/bonwick/entry/zfs_end_to_end_data
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [Haskell-cafe] Re: announcing darcs 2.0.0pre3

[David Roundy]

On Wed, Jan 23, 2008 at 10:09:05PM +0100, Peter Verswyvelen wrote:
> Maybe a dedicated SIMD version of SHA1?
> 
> http://arctic.org/~dean/crypto/sha1.html

>From what that page says, it looks like thta sha1 is not recommended for
actual use (although they don't test with gcc 4.x).  I've come across the
gnulib sha1 routine, which is adequately-licensed, and is fast enough to
give us a 10% speedup in my obliterate test (beyond which we probably hit
diminishing returns--we're probably no longer spending much time in sha1 at
all).
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue609] can't darcs mv file over the place where another file once stood

[Zooko]

New submission from Zooko <[EMAIL PROTECTED]>:

Try this with darcs-2:

rm afile
darcs record
darcs mv anotherfile afile

It will say that it can't proceed since afile already exists (at  
least on Windows), even though afile actually doesn't exist.

--
messages: 2698
nosy: beschmi, droundy, kowey, tommy, zooko
status: unread
title: can't darcs mv file over the place where another file once stood

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] announcing darcs 2.0.0pre3

[David Roundy]

On Wed, Jan 23, 2008 at 03:26:51PM +, Simon Marlow wrote:
> There are still times when I see nothing happening, for example in the 
> unpull test on the GHC repo (see previous messages), the last progress 
> message I get is
> 
> Reading patches in /64playpen/simonmar/ghc-darcs2 17040
> 
> and it sits there for 7-8 seconds before completing.  Does this maybe shed 
> any light on why this unpull is 2 times slower than darcs1?

I'm not entirely certain what's triggering this, but I have identified that
removing a couple of sha1 checks cuts 1s out of 15s for me.  This makes me
wonder whether it's worth looking into a faster sha1 implementation.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] announcing darcs 2.0.0pre3

[David Roundy]

Incidentally, I've just been getting myself a few copies of the ghc
repository for timings, and enjoyed seeing how long it took...

$ time darcs get --complete http://darcs.haskell.org/ghc ghc-oldfashioned
This is the GHC darcs repository (HEAD branch)

For more information, visit the GHC developer wiki at
  http://hackage.haskell.org/trac/ghc
**
Finished getting.

real11m42.452s
user2m49.275s
sys 0m59.780s
$ time darcs get ghc-oldfashioned ghc-oldfashioned2
Finished getting.

real2m31.958s
user1m27.689s
sys 0m57.396s
$ time darcs get --hashed ghc-oldfashioned ghc-hashed
Finished getting.

real6m37.527s
user3m26.321s
sys 2m59.191s
$ time darcs get ghc-hashed/ ghc-hashed2
Finished getting.

real0m1.431s
user0m1.072s
sys 0m0.344s

I'm not sure why the third get was so slow, but the last one was quite
satisfying, actually.  :) To be fair, the third command doesn't copy any
patches... but that's the point.  With the hashed repository formats we can
have truly cheap branches, even of large repositories.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] initial performance measurement of darcs-2

[David Roundy]

Hi Zooko,

Thanks for your clarification (and more timings)!

On Wed, Jan 23, 2008 at 08:43:53AM -0700, zooko wrote:
> On Jan 23, 2008, at 5:02 AM, David Roundy wrote:
> > 1. Am I correct that all these timings involve push -a and pull -a when
> > there is nothing to push or pull?
> 
> Yes.
> 
> > 2. Did you enable a global cache directory with ~/.darcs/sources
> 
> No.  There is only one darcs-2-format repository on each machine so  
> it didn't occur to me that a global cache could be useful.

Actually, I think you're right.  So you can save time and not bother with
the global cache.

> > 3. Did you repeat your measurements? In particular, if you use a global
> > cache the second time you push/pull with either --hashed on both sides
> > or --darcs-2 on both sides should be lightning fast.  (And if it's not,
> > I want to figure out why.)
> 
> Okay I just did it again.  For each operation I did four trials and  
> am reporting the best (fastest):
> 
> A.  darcs 1.0.9, old-format local repo, old-format remote repo
> B.  darcs 2 pre, hashed-format local repo, old-format remote repo

Two cases I'd like to see is darcs 2 pre with hashed-format on each side,
and darcs 2 pre with old-format on each side.

The latter case really ought to be about as fast as darcs 1.0.9, so it can
serve as a test for whether we've done something seriously stupid
(i.e. given the same format, we shouldn't require more network accesses).

The former (hashed-hashed) ought to be faster than accessing a remote
old-fashioned repository.  It really should only take O(1) network
accesses... although, to be honest, the old-fashioned repository will also
take O(1), so maybe it won't actually be faster.

The (small) slowdown you're seeing may be something stupid like accessing
the remote _darcs/format multiple times.

> B2. same but with a newly created global cache on the local side
> B3. same but with an already-populated global cache on the local side
> C.  darcs 2 pre, new-format local repo, new-format remote repo
> C2. same but with a newly created global cache on the local side
> C3. same but with an already-populated global cache on the local side
> 
> column 1: push, column 2: pull ; all values in seconds
> 
> A. 7.918.2
> B.16.314.7
> B2.   14.619.1
> B3.   16.014.3
> C. 9.012.1
> C2.   11.415.7
> C3.9.9 9.8
> 
> Note that these measurements varied by many seconds -- up to 100% in  
> some runs -- possibly because of network contention or disk or CPU  
> contention on the remote repo.  So consider them to be only broad  
> general measurements of performance, and also use these measurements  
> as reminders that your handling of network latency and resource  
> contention are often more important than your algorithm efficiency.
> 
> So far my general feeling is that switching over to darcs-2 and using  
> hashed repository format (the B rows) would not be an improvement  
> over using darcs-1, and that requiring everyone who wants to share  
> source with me to switch to darcs-2 and use darcs-2-repository-format  
> (the C rows) might be an improvement, but it isn't clear yet, and  
> obviously having a "flag day" style upgrade like that is a deterrent.

I think that if your remote repository is hashed, then you should get the
same performance (when there are no conflicts) with --hashed as you do with
--darcs-2, which is an improvement in pull according to your timings.  If
you've got the disk space, a cron job to keep --old-fashioned and --hashed
repositories on the server in sync is pretty easy and safe, and allows
darcs 2 users to have faster results.

> Oh, by the way, I suspect that darcs on windows (the "local" machine  
> in these measurements) isn't actually finding the global cache (I  
> told it "~/.darcs/cache", and I guess it doesn't know what I mean by  
> "~"), so you can probably consider the 1/2/3 rows as just showing the  
> noise in the system rather than actually using the cache.  :-)
> 
> I'll try to explain to darcs-on-windows where to do caching, later.

I'll appreciate that! There's a function in the haskell libraries that is
supposed to give something like a cross-platform "home" directory, but I
know there's been debate about how that's best defined (since Windows
doesn't really have the equivalent of a unix home directory, at least not
in the sense of "a place to put .files").
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] announcing darcs 2.0.0pre3

[David Roundy]

On Wed, Jan 23, 2008 at 03:26:51PM +, Simon Marlow wrote:
> David Roundy wrote:
> > We are happy to announce the third prerelease version of darcs 2! Darcs 2
> > features numerous improvements, and it seems that we have fixed most of the
> > regressions, so we're looking for help, from users willing to try this
> > release out. Read below, to see how you can benefit from this new release,
> > and how you can help us to make the final darcs 2 release the best ever!
> > 
> > The third prerelease features (apart from numerous bug and performance
> > regression fixes) a completely rewritten rollback command and new
> > progress-reporting functionality. If darcs takes more than a couple of
> > seconds for a given operation and provides you with no feedback as to what
> > it's up to, let us know, so we can fine-tune the progress-reporting output!
> 
> The progress reporting is fantastic!  It's worth upgrading to darcs2 just 
> for that :-)

Thanks! I've enjoyed it myself, actually...

> ... Although the progress reporting doesn't appear to work quite as well
> on darcs-1 repositories as it does on darcs-2 repositories - is that
> expected?

No, it's not really expected, but the progress reporting is rather hastily
thrown together, so it's not surprising, either.  Basically I first
converted the existing progress-reporting code (which reported getting
patches with darcs get, and applying them in a few other instances), and
then started throwing in progress annotations in other spots until my
couple of test commands and the ones I normally use seemed to be pretty
snappy (I'm not sure what the right word for a lack of delays in
progress-reporting).

> There are still times when I see nothing happening, for example in the 
> unpull test on the GHC repo (see previous messages), the last progress 
> message I get is
> 
> Reading patches in /64playpen/simonmar/ghc-darcs2 17040
> 
> and it sits there for 7-8 seconds before completing.  Does this maybe shed 
> any light on why this unpull is 2 times slower than darcs1?

Hmmm.  I'll take a look.  This is basically equivalent to something like

darcs obliterate --last 400 -a

I believe? The ghc repo definitely stresses different parts of darcs,
because of its large size (which means you often have the privilege of
reporting poor behavior).
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] initial performance measurement of darcs-2

[zooko]

On Jan 23, 2008, at 5:02 AM, David Roundy wrote:

> 1. Am I correct that all these timings involve push -a and pull -a  
> when
> there is nothing to push or pull?

Yes.

> 2. Did you enable a global cache directory with ~/.darcs/sources

No.  There is only one darcs-2-format repository on each machine so  
it didn't occur to me that a global cache could be useful.

> 3. Did you repeat your measurements? In particular, if you use a  
> global
> cache the second time you push/pull with either --hashed on both  
> sides or
> --darcs-2 on both sides should be lightning fast.  (And if it's  
> not, I want
> to figure out why.)

Okay I just did it again.  For each operation I did four trials and  
am reporting the best (fastest):

A.  darcs 1.0.9, old-format local repo, old-format remote repo
B.  darcs 2 pre, hashed-format local repo, old-format remote repo
B2. same but with a newly created global cache on the local side
B3. same but with an already-populated global cache on the local side
C.  darcs 2 pre, new-format local repo, new-format remote repo
C2. same but with a newly created global cache on the local side
C3. same but with an already-populated global cache on the local side

column 1: push, column 2: pull ; all values in seconds

A.   7.918.2
B.  16.314.7
B2. 14.619.1
B3. 16.014.3
C.   9.012.1
C2. 11.415.7
C3.  9.9 9.8

Note that these measurements varied by many seconds -- up to 100% in  
some runs -- possibly because of network contention or disk or CPU  
contention on the remote repo.  So consider them to be only broad  
general measurements of performance, and also use these measurements  
as reminders that your handling of network latency and resource  
contention are often more important than your algorithm efficiency.


So far my general feeling is that switching over to darcs-2 and using  
hashed repository format (the B rows) would not be an improvement  
over using darcs-1, and that requiring everyone who wants to share  
source with me to switch to darcs-2 and use darcs-2-repository-format  
(the C rows) might be an improvement, but it isn't clear yet, and  
obviously having a "flag day" style upgrade like that is a deterrent.


Oh, by the way, I suspect that darcs on windows (the "local" machine  
in these measurements) isn't actually finding the global cache (I  
told it "~/.darcs/cache", and I guess it doesn't know what I mean by  
"~"), so you can probably consider the 1/2/3 rows as just showing the  
noise in the system rather than actually using the cache.  :-)

I'll try to explain to darcs-on-windows where to do caching, later.


Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] announcing darcs 2.0.0pre3

[Simon Marlow]

David Roundy wrote:
> We are happy to announce the third prerelease version of darcs 2! Darcs 2
> features numerous improvements, and it seems that we have fixed most of the
> regressions, so we're looking for help, from users willing to try this
> release out. Read below, to see how you can benefit from this new release,
> and how you can help us to make the final darcs 2 release the best ever!
> 
> The third prerelease features (apart from numerous bug and performance
> regression fixes) a completely rewritten rollback command and new
> progress-reporting functionality. If darcs takes more than a couple of
> seconds for a given operation and provides you with no feedback as to what
> it's up to, let us know, so we can fine-tune the progress-reporting output!

The progress reporting is fantastic!  It's worth upgrading to darcs2 just 
for that :-)  Although the progress reporting doesn't appear to work quite 
as well on darcs-1 repositories as it does on darcs-2 repositories - is 
that expected?

There are still times when I see nothing happening, for example in the 
unpull test on the GHC repo (see previous messages), the last progress 
message I get is

Reading patches in /64playpen/simonmar/ghc-darcs2 17040

and it sits there for 7-8 seconds before completing.  Does this maybe shed 
any light on why this unpull is 2 times slower than darcs1?

Cheers,
Simon
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: Libwww new API: waitUrl, copyUrlFirst added; copyUrls ...

[Simon Marlow]

David Roundy wrote:
> On Wed, Jan 23, 2008 at 01:31:30PM +, Simon Marlow wrote:
>> David Roundy wrote:
>>> On Mon, Jan 21, 2008 at 08:03:22PM +0300, Dmitry Kurochkin wrote:
 I will try to reproduce and fix it.  BTW How can i build darcs with
 debug symbols?
>>> I don't know.  :( I see that ghc has a -debug option, but am unsure what it
>>> does.
>> It links the program with a debug version of the runtime, which adds 
>> assertions and other sanity-checking, some debug output options (+RTS 
>> -D) and also has debug symbols.
> 
> Ah good, that does sound like it could be a real help (particularly the
> debug symbols).  Perhaps we should add a --enable-debug option to our
> configure script, and maybe even make that the default for prerelease
> builds?

-debug slows things down: GC by as much as a factor of 2, and other things 
to a lesser extent.  Apart from that there should be no downsides.

Cheers,
Simon

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: Libwww new API: waitUrl, copyUrlFirst added; copyUrls ...

[David Roundy]

On Wed, Jan 23, 2008 at 01:31:30PM +, Simon Marlow wrote:
> David Roundy wrote:
> > On Mon, Jan 21, 2008 at 08:03:22PM +0300, Dmitry Kurochkin wrote:
> >> I will try to reproduce and fix it.  BTW How can i build darcs with
> >> debug symbols?
> > 
> > I don't know.  :( I see that ghc has a -debug option, but am unsure what it
> > does.
> 
> It links the program with a debug version of the runtime, which adds 
> assertions and other sanity-checking, some debug output options (+RTS 
> -D) and also has debug symbols.

Ah good, that does sound like it could be a real help (particularly the
debug symbols).  Perhaps we should add a --enable-debug option to our
configure script, and maybe even make that the default for prerelease
builds?
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [issue604] Windows builds of darcs-2 don't work for me

[Simon Marlow]

Zooko wrote:
> New submission from Zooko <[EMAIL PROTECTED]>:
> 
> Folks:
> 
> I tried the executables from
> 
> http://www.cs.mu.oz.au/~rgm/darcs/
> 
> but they don't do anything on my Windows XP system, not even pop up a  
> crash dialog.  "./darcs.exe --version", for example, returns silently  
> to the prompt.

This often indicates a missing DLL.  IIRC, the exit code is 128 in this 
case.  If so, try http://www.dependencywalker.com/ to find out what is missing.

Cheers,
Simon
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: Libwww new API: waitUrl, copyUrlFirst added; copyUrls ...

[Simon Marlow]

David Roundy wrote:
> On Mon, Jan 21, 2008 at 08:03:22PM +0300, Dmitry Kurochkin wrote:
>> 2008/1/21, David Roundy <[EMAIL PROTECTED]>:
>>> Alas, I am now getting segfaults when I do a darcs pull over http with
>>> libwww.  :(  I get this pretty reliably if I do
>>>
>>> darcs get http://darcs.net/repos/unstable
>>> cd unstable
>>> darcs obliterate --last 1000 -a
>>> darcs pull -a
>>>
>>> (I've been running these slightly-crazy-length commands to test the
>>> progress code.)
>> I will try to reproduce and fix it.
>> BTW How can i build darcs with debug symbols?
> 
> I don't know.  :( I see that ghc has a -debug option, but am unsure what it
> does.

It links the program with a debug version of the runtime, which adds 
assertions and other sanity-checking, some debug output options (+RTS 
-D) and also has debug symbols.

Cheers,
Simon

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] initial performance measurement of darcs-2

[David Roundy]

Hi Zooko,

I've got a few questions:

1. Am I correct that all these timings involve push -a and pull -a when
there is nothing to push or pull? (That's That's definitely an interesting
case, but I'm worried by it taking 12-17s to report there are no patches to
push/pull.

2. Did you enable a global cache directory with ~/.darcs/sources as
described in

http://wiki.darcs.net/index.html/DarcsTwo

(search for global cache)

3. Did you repeat your measurements? In particular, if you use a global
cache the second time you push/pull with either --hashed on both sides or
--darcs-2 on both sides should be lightning fast.  (And if it's not, I want
to figure out why.)

I think that's all the questions I've got just now.

David

On Tue, Jan 22, 2008 at 10:02:35PM -0700, zooko wrote:
> Folks:
> 
> I just made a quick measurement of darcs-2 performance.
> 
> Using darcs 1.0.9 I can do "push -a" to another repo which is  
> connected by 802.11g (and which has all the same patches as the  
> source repo) in 3.5 s (all measurements are best of four trials), and  
> can do a "pull -a" in 12.7 s.
>
> Using darcs 2 with this repo in hashed format and the remote repo in  
> old format, push takes 0.15 s and pull takes 0.16 s.
> 
> Using darcs 2 with this repo and the remote repo in darcs-2 format,  
> push takes 10.7 s and pull takes 17.0 s!
> 
> 
> So as far as pushing and pulling when both repos have the same set of  
> patches, using hashed format is much faster than using darcs-2 format!
> 
> 
> The repo in question is publically available (and chock full of very  
> cool source code that you are welcome to use):
> 
> http://allmydata.org/source/tahoe/trunk
> 
> Regards,
> 
> Zooko
> 
> ___
> darcs-devel mailing list
> darcs-devel@darcs.net
> http://lists.osuosl.org/mailman/listinfo/darcs-devel

-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: use UTC in date matching test untill mat... (and 1 more)

[David Roundy]

On Wed, Jan 23, 2008 at 01:04:39AM +0100, Tommy Pettersson wrote:
> On Tue, Jan 22, 2008 at 05:52:28PM -0500, David Roundy wrote:
> > On Tue, Jan 22, 2008 at 11:37:33PM +0100, [EMAIL PROTECTED] wrote:
> > > Tue Jan 22 14:43:22 CET 2008  Tommy Pettersson <[EMAIL PROTECTED]>
> > >   * use UTC in date matching test untill match handles time zones
> > 
> > I'm not entirely comfortable with this, as it hides the fact that we've got
> > a real bug remaining.  I guess so long as we don't close it on the bug
> > tracker, we'll not forget it... (and it's hardly fair that the test suite
> > usually passes for me, just because I'm not too far from England and prefer
> > not to work on the computer at night).
> 
> It's also a bit uncomfortable when a test fails some times but
> not other times. For me the test fails between midnight and
> 01:00, which happens at times. :-)

I agree that that is a pain.  On the other hand, once the code is fixed, I
think this will be an advantage, having the test suite run a subtly
different test every time it's run.  The result will be that when averaged
over all the folks who run make check--provided folks notify us when it
goes bad--we'll have verification that the test works at all times of day
in all time zones.  Admittedly, it may be somewhat sparse sampling, but
over time this ought to ensure that darcs' date matching is robust.  And
once darcs' date matching *is* robust, then the test should pass for all
time zones (as long as we avoid actually trying to parse the three-letter
timezone abbreviation in the test).
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: Issue395: avoid single letter patch name... (and 6 more)

[David Roundy]

On Tue, Jan 22, 2008 at 08:39:17PM -0500, Mark Stosberg wrote:
> Thu Jan 17 21:06:34 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * Issue395: avoid single letter patch names in the test suite.  
> 
> Thu Jan 17 22:02:41 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * use --ignore-time in tests instead of "sleep", for faster, more reliable 
> results
> 
> Sat Jan 19 19:45:03 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * Take advantage of updated Perl testing infrastructure by removing 
> needless author mentions in tests
> 
> Sat Jan 19 19:52:42 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * Take advantage of new Perl testing infrastructure by eliminating needless 
> --ignore-time mentions
> 
> Sat Jan 19 22:56:51 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * Major Perl test suite clean-up.
>   The primary purpose of this patch was make sure all the tests are 
> executed in
>   randomly named directories, which allows us to run Perl tests in 
> parallel, 
>   without the directory names collided. 
>   
>   This isn't enabled by default for "make test", but it is there to play 
> with.
>   In the test directory, you can now do:
>   
>   ./bin/prove -j9 *.pl 
>   
>   to run 9 tests in parallel. There is also "--fork"
>   option which should be a win on multi-CPU computers. 
>   See "perldoc ./bin/prove" for details. 
>   
>   As part of this, a lot of boiler-plate code at the top and bottom of the
>   scripts could be eliminated, and I made few other minor style clean-ups
>   while I had the files open. 
>   
>   There should be no functional changes to the tests. 

I'm not sure I like this idea, as it seems likely to lead to greater
difficulty in understanding and debugging failing test results.  We can
already run up to six tests in parallel just by running make -j6 test.

Also, why make the directory names random? Couldn't we just generate a
unique directory name from the test name itself? So then init_tmp_repo() in
test foo.pl would generate a repository named temp1-foo.

Then we'd have a relatively readable scheme for generating repository
directories, and hopefully it'd be pretty easy to find the right
repository, when investigating a bug, and you'd have parallelization, too.

I'm applying this patch, but hope that you'll be willing to rewrite this
init_tmp_repo to create something predictable.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: Actually run perl tests. (and 1 more)

[Nicolas Pouillard]

Excerpts from David Roundy's message of Tue Jan 22 22:06:48 +0100 2008:
> On Jan 18, 2008 5:42 AM, Nicolas Pouillard <[EMAIL PROTECTED]> wrote:
> > Excerpts from David Roundy's message of Thu Jan 17 21:36:02 +0100 2008:
> > > Thanks for the patch! However, I've now moved both of these messages to
> > > debugMessage, since I'd rather keep --verbose free for "useful"
> > > (i.e. non-progress) output.
> >
> > What  about  a --progress flag? I completely agree that one should not 
> > pollute
> > --verbose  with progress stuff, but I rather keep --debug when I felt 
> > that's a
> > bug in darcs.
> >
> > Perhaps  that  would  be  an  occasion to give an argument to progress like 
> > an
> > integer  or  better  a  list  of  tags  that  we  want  to see the 
> > progression
> > (commutations,  network,  disk,  reading patches...). That should be 
> > something
> > quite  nice  looking  so  that  people  can  leave  some  of  these  in  
> > their
> > ~/.darcs/defaults file.
> >
> > What do you think of it?
> 
> I think I'd rather keep the number of flags low.  Hopefully we can get
> a default that is good enough for most folks.

That's wise. And it's even better is their is good progress messages by default.

-- 
Nicolas Pouillard aka Ertai


signature.asc
Description: PGP signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel