Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

On Jan 25, 2008, at 5:13 PM, zooko wrote:

> Hm.  This file is 390 KB.  I'll just give you the head -20 and  
> attach the compressed version.  There is no occurrence of the  
> string "sftp" in it.

That last sentence is wrong.

> ssh -O an_invalid_command ; # Null Null Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/prefs/motd darcs378c77 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/format darcse2816e ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/.git/darcs-format darcs797999 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventory darcs5ecf92 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/hashed_inventory darcs44aef8 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/.git/HEAD darcs79ac31 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/format darcs3bacf2 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/.git/darcs-format darcs84f1c8 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/prefs/sources darcs5f9ca5 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventory darcsaf ; # AsIs AsIs Null
> Getting the inventory...
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/ 
> 20080108184749-92b7f-5df11c2d78599efdd56baece7488fa6a749d65c7.gz  
> darcs9d3c45 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/ 
> 20071016041556-92b7f-95c1d3dcb8a417ed1d884cbcb5b81a38173c15ed.gz  
> darcsd49fc3 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/ 
> 20070924214358-92b7f-0d177d6726975bc04dc0066d830ae4eae5919094.gz  
> darcs1ef2a4 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/ 
> 20070823210315-92b7f-888fbf1b1867ccda8218ad9f955b835cbd474164.gz  
> darcsd68c51 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/ 
> 2007063254-92b7f-4b19b2708a8832f8ec8eea6968c2ef99315355e4.gz  
> darcsc5b11a ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/ 
> 20070611162221-92b7f-2c56a212abacccdc16da52833225fdafe51d8b64.gz  
> darcsc78136 ; # AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/20070504024220-4233b- 
> f5dfde96dca1eb23bfb22cfe2dd412c8456b826b.gz darcsfce705 ; # AsIs  
> AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/20070426182525- 
> eb44e-97f3b8656eb88d899e2eced99ce2ce018451e600.gz darcs570878 ; #  
> AsIs AsIs Null
> scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
> trunk/_darcs/inventories/ 
> 20070425235105-92b7f-52f87be1b8b1bca805373129e02bf9c4c42e89b9.gz  
> darcs33bb7a ; # AsIs AsIs Null
> Copying patches...
> sftp [EMAIL PROTECTED] ; # File "darcs58163c" File  
> "darcsd124bd" Null
> Patches copied
> Repo lazily read
> Repo local: "False"
> Applying patch 1 of 1979: Sat Oct 21 21:10:04 Mountain Daylight  
> Time 2006  [EMAIL PROTECTED]
>   * tool to generate an overhead/alacrity table for various hashing  
> schemes
> Applying patch 2 of 1979: Sat Oct 21 21:39:12 Mountain Daylight  
> Time 2006  [EMAIL PROTECTED]
>   * emit tables up to 1TiB
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] darcs patch: issue612: regression test demonstrates that hashed and...

[Mark Stosberg]

Fri Jan 25 22:14:45 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
  * issue612: regression test demonstrates that hashed and darcs-2 formats 
resist a certain kind of corruption.
 (but the old fashioned repo format does not).  

New patches:

[issue612: regression test demonstrates that hashed and darcs-2 formats resist a certain kind of corruption.
Mark Stosberg <[EMAIL PROTECTED]>**20080126031445
(but the old fashioned repo format does not).  
] {
addfile ./tests/repo_not_writable.pl
hunk ./tests/repo_not_writable.pl 1
+#!/usr/bin/env perl
+
+# Test that darcs fails appropriately when the target repo inventory file is not writable.
+# See issue612
+
+use lib 'lib/perl';
+use Test::More 'no_plan';
+use Test::Darcs;
+use Shell::Command;
+use strict;
+use warnings;
+use File::Slurp 'append_file';
+use File::Temp 'tempdir';
+
+# test_with_format('--old-fashioned-inventory'); known not to work. 
+test_with_format('--darcs-2');
+test_with_format('--hashed');
+
+sub test_with_format {
+my $format = shift;
+
+my $work_dir = tempdir( CLEANUP => 1 );
+chdir $work_dir;
+
+ok(mkpath('r1'), "r1 created (starting tests for format: $format)");
+ok(chdir('r1'), "chdir r1");
+darcs "init $format";
+touch 't.t';
+darcs 'add t.t';
+darcs "record -am 'initial add'";
+CORE::chmod 0555, glob('_darcs/inventories/*') if -r '_darcs/inventories';
+CORE::chmod 0555, glob('_darcs/inventories')   if -r '_darcs/inventories';
+CORE::chmod 0555, glob('_darcs/inventory') if -r '_darcs/inventory';
+ok(chdir('../'), "chdir ../");
+darcs "get r1 r2";
+ok(chdir('r2'), "chdir r2");
+append_file('t.t',"new");
+darcs "record -am 'new patch'";
+my $out = darcs "push -a ../r1";
+like($out, qr/failed/, "expect failure when pushing to a repo with unwritable inventory (format: $format)");
+
+chdir '../../'; # move out the way for auto-tmpdir cleanup. 
+}
+
+
}

Context:

[resolve conflict with Eric on controlMasterPath.
David Roundy <[EMAIL PROTECTED]>**20080125203903] 
[More concise backup warning.
Eric Kow <[EMAIL PROTECTED]>**20071105012930] 
[Remove now obsolete wrapper for Map (we now require GHC >= 6.4).
Eric Kow <[EMAIL PROTECTED]>**20071105192636] 
[Modernise Data.Map import.
Eric Kow <[EMAIL PROTECTED]>**20071105192530] 
[Give ssh CM socket a unique name for each darcs process.
Eric Kow <[EMAIL PROTECTED]>**20071105021956
 Delete the socket in the unlikely event that a previous darcs had a socket
 with the same name left over.
] 
[Create ssh CM socket in $HOME/.darcs if possible.
Eric Kow <[EMAIL PROTECTED]>**20071105015525] 
[Refactor y/n prompts.
Eric Kow <[EMAIL PROTECTED]>**20071019213307] 
[eliminate lazy parsing of patches, which gives bad error messages (issue364)
David Roundy <[EMAIL PROTECTED]>**20080125191836] 
[[issue492] Check that context file actually exists in darcs get.
Eric Kow <[EMAIL PROTECTED]>**20080125183741] 
[[issue227] Platform-independent absolute paths in get --context
Eric Kow <[EMAIL PROTECTED]>**20080125181702] 
[make uniqueoptions.sh test give friendlier output.
David Roundy <[EMAIL PROTECTED]>**20080125183430] 
[fix code to avoid duplicate --verbose in --help (so tests will pass).
David Roundy <[EMAIL PROTECTED]>**20080125183420] 
[Make verbosity flags advanced options universally.
Eric Kow <[EMAIL PROTECTED]>**20080125181005] 
[More error messages for libwww.
Dmitry Kurochkin <[EMAIL PROTECTED]>**20080124092600] 
[report progress in writing the inventory out for hashed repos.
David Roundy <[EMAIL PROTECTED]>**20080125172017] 
[make empty key case of progress reporting fast.
David Roundy <[EMAIL PROTECTED]>**20080125171859] 
[fix issue where we overwrote prompt with progress info.
David Roundy <[EMAIL PROTECTED]>**20080125164609] 
[fix bug where we used show on an exception (and thus printed "User error").
David Roundy <[EMAIL PROTECTED]>**20080125164209
 This partially addresses issue168 by improving the error message.
] 
[add gnulib sha1.c file as a faster sha1 option.
David Roundy <[EMAIL PROTECTED]>**20080123212502] 
[fix embarrassing bug in External.
David Roundy <[EMAIL PROTECTED]>**20080125152329
 (which demonstrates that I didn't compile before pushing)
] 
[for now, print progress reports to stdout.
David Roundy <[EMAIL PROTECTED]>**20080125152105
 My hope is that this will alleviate some of the issues with progress
 reports overwriting prompts.
] 
[revamp progress reporting, making it more efficient and adding more output.
David Roundy <[EMAIL PROTECTED]>**20080125151540
 Note that there is still at least one time sink that remains to be identified.
] 
[avoid creating darcs-ssh if we aren't using ControlMaster. (issue613)
David Roundy <[EMAIL PROTECTED]>**20080125150846] 
[fix bug where darcs-ssh got even worse name (issue613).
David Roundy <[EMAIL PROTECTED]>**20080125150355] 
[provide more detailed progress reports in HashedIO.
David Roundy <[EMAIL PROTECTED]>**20080124145156] 
[print additional debug data in

[darcs-devel] [issue227] get --context fails with absolute path in win32

[Mark Stosberg]

I see that patches were committed for this, and I just sent a regression test,
so I'm marking it resolved for now. The regression test should smoke out the
issue if it is still present.

--
status: chatting -> resolved-in-unstable

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] darcs patch: issue227: regression test for get --context with an ab...

[Mark Stosberg]

Fri Jan 25 21:11:39 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
  * issue227: regression test for get --context with an absolute path

New patches:

[issue227: regression test for get --context with an absolute path
Mark Stosberg <[EMAIL PROTECTED]>**20080126021139] {
addfile ./tests/get.pl
hunk ./tests/get.pl 1
+#!/usr/bin/env perl
+
+# Some tests for darcs get
+# in particular, we text --context with absolute paths, as reported in in issue227.
+
+use lib 'lib/perl';
+use Test::More 'no_plan';
+use Test::Darcs;
+use Shell::Command;
+use File::Slurp;
+use Cwd 'abs_path';
+use strict;
+
+use File::Temp 'tempdir';
+chdir tempdir( CLEANUP => 1 );
+
+mkpath 'first';
+chdir  'first';
+darcs  'init';
+
+touch 't.t';
+darcs 'add t.t';
+darcs 'record -am "initial add"';
+my $context = darcs "changes --context";
+ok($context,"got some context");
+
+write_file('my_context.txt',$context) || die;
+
+my $abs_to_context = abs_path('my_context.txt');  
+
+chdir '../';
+
+my $out = darcs "get first --context=$abs_to_context second";
+like($out,qr/finished getting/i, "got expected success message when using absolute path with --context");
+
+
+
}

Context:

[resolve conflict with Eric on controlMasterPath.
David Roundy <[EMAIL PROTECTED]>**20080125203903] 
[More concise backup warning.
Eric Kow <[EMAIL PROTECTED]>**20071105012930] 
[Remove now obsolete wrapper for Map (we now require GHC >= 6.4).
Eric Kow <[EMAIL PROTECTED]>**20071105192636] 
[Modernise Data.Map import.
Eric Kow <[EMAIL PROTECTED]>**20071105192530] 
[Give ssh CM socket a unique name for each darcs process.
Eric Kow <[EMAIL PROTECTED]>**20071105021956
 Delete the socket in the unlikely event that a previous darcs had a socket
 with the same name left over.
] 
[Create ssh CM socket in $HOME/.darcs if possible.
Eric Kow <[EMAIL PROTECTED]>**20071105015525] 
[Refactor y/n prompts.
Eric Kow <[EMAIL PROTECTED]>**20071019213307] 
[eliminate lazy parsing of patches, which gives bad error messages (issue364)
David Roundy <[EMAIL PROTECTED]>**20080125191836] 
[[issue492] Check that context file actually exists in darcs get.
Eric Kow <[EMAIL PROTECTED]>**20080125183741] 
[[issue227] Platform-independent absolute paths in get --context
Eric Kow <[EMAIL PROTECTED]>**20080125181702] 
[make uniqueoptions.sh test give friendlier output.
David Roundy <[EMAIL PROTECTED]>**20080125183430] 
[fix code to avoid duplicate --verbose in --help (so tests will pass).
David Roundy <[EMAIL PROTECTED]>**20080125183420] 
[Make verbosity flags advanced options universally.
Eric Kow <[EMAIL PROTECTED]>**20080125181005] 
[More error messages for libwww.
Dmitry Kurochkin <[EMAIL PROTECTED]>**20080124092600] 
[report progress in writing the inventory out for hashed repos.
David Roundy <[EMAIL PROTECTED]>**20080125172017] 
[make empty key case of progress reporting fast.
David Roundy <[EMAIL PROTECTED]>**20080125171859] 
[fix issue where we overwrote prompt with progress info.
David Roundy <[EMAIL PROTECTED]>**20080125164609] 
[fix bug where we used show on an exception (and thus printed "User error").
David Roundy <[EMAIL PROTECTED]>**20080125164209
 This partially addresses issue168 by improving the error message.
] 
[add gnulib sha1.c file as a faster sha1 option.
David Roundy <[EMAIL PROTECTED]>**20080123212502] 
[fix embarrassing bug in External.
David Roundy <[EMAIL PROTECTED]>**20080125152329
 (which demonstrates that I didn't compile before pushing)
] 
[for now, print progress reports to stdout.
David Roundy <[EMAIL PROTECTED]>**20080125152105
 My hope is that this will alleviate some of the issues with progress
 reports overwriting prompts.
] 
[revamp progress reporting, making it more efficient and adding more output.
David Roundy <[EMAIL PROTECTED]>**20080125151540
 Note that there is still at least one time sink that remains to be identified.
] 
[avoid creating darcs-ssh if we aren't using ControlMaster. (issue613)
David Roundy <[EMAIL PROTECTED]>**20080125150846] 
[fix bug where darcs-ssh got even worse name (issue613).
David Roundy <[EMAIL PROTECTED]>**20080125150355] 
[provide more detailed progress reports in HashedIO.
David Roundy <[EMAIL PROTECTED]>**20080124145156] 
[print additional debug data in Progress.
David Roundy <[EMAIL PROTECTED]>**20080124145114] 
[add a few more debug messages in Repository.Internal.
David Roundy <[EMAIL PROTECTED]>**20080124144829] 
[fix incorrect report that we were reading patches.
David Roundy <[EMAIL PROTECTED]>**20080124125040] 
[adding File::Temp 0.20 to tree for more consistent test results. It is GPL-licensed.
Mark Stosberg <[EMAIL PROTECTED]>**20080124033049] 
[issue608: a new test for 'mv', following Zooko's bug report
Mark Stosberg <[EMAIL PROTECTED]>**20080124013856] 
[ issue602: part 1: Always prefer our private copy of Test::More over the system-wide one for more consistent results
Mark Stosberg <[EMAIL PROTECTED]>**20080124005407] 
[reenable mandatory sha1 checks, now that we can link with a faster sha1.
David Roundy <[EMAIL PROTE

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[Kevin Quick]

Just throwing my 2 cents into this conversation: poor windows  
performance due to exactly these ssh issues is one of the two main  
reasons why I've been unsuccessful in generating sustainable interest  
in using darcs @ my workplace (the other is line-ending support).   
This would be a very good issue to resolve IMHO, and as Zooko points  
out, I don't think we'd need anything as robust/featureful as a full  
"ftp"-style application.

-KQ

On 25 Jan 2008, at 4:43 PM, zooko wrote:

> On Jan 25, 2008, at 4:08 PM, David Roundy wrote:
>
>> It also would require that we develop an entire ftp protocol  
>> including
>> transmition of errors.
>
> This part I don't understand.  You just define a protocol, like:
>
> I send you a stream of patchids (patchids have a known length, so
> there doesn't need to be any terminating character or length
> encoding), and you send me a stream of patches, with each patch
> prefixed by its length in bytes expressed as a big-endian, unsigned 8
> byte integer.
>
> This would be a rich enough protocol to do the job of an FTP server,
> but it doesn't need to be.
>
>>   I think this would definitely be harder than
>> learning haskell.  Which isn't to say it's not a reasonable  
>> problem to
>> tackle, but rather that your lack of Haskell knowledge isn't an
>> excuse!
>
> Heh heh.  To me, implementing network protocols is easy, and learning
> Haskell is hard.
>
> Regards,
>
> Zooko
>
> ___
> darcs-devel mailing list
> darcs-devel@darcs.net
> http://lists.osuosl.org/mailman/listinfo/darcs-devel

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

On Jan 25, 2008, at 4:10 PM, David Roundy wrote:

Odd.  I'll have to look into this tomorrow.  Darcs should be using  
sftp for
a darcs get.  Is this using darcs-1 repositories? If not, could you  
recheck
with darcs1 repositories on both sides? Thanks!  (I checked  
similarly that

my copy of darcs2 uses sftp on darcs1 repositories.)


Here it is with darcs-1 repositories.  It took 41m46s to run.  (It  
appeared to hang for something on the order of 10 minutes between a  
particular pair of "scp" debug outputs.)


Regards,

Zooko

Hm.  This file is 390 KB.  I'll just give you the head -20 and attach  
the compressed version.  There is no occurrence of the string "sftp"  
in it.


ssh -O an_invalid_command ; # Null Null Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/prefs/motd darcs378c77 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/format darcse2816e ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/.git/darcs-format darcs797999 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventory darcs5ecf92 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/hashed_inventory darcs44aef8 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/.git/HEAD darcs79ac31 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/format darcs3bacf2 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/.git/darcs-format darcs84f1c8 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/prefs/sources darcs5f9ca5 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventory darcsaf ; # AsIs AsIs Null

Getting the inventory...
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/ 
20080108184749-92b7f-5df11c2d78599efdd56baece7488fa6a749d65c7.gz  
darcs9d3c45 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/ 
20071016041556-92b7f-95c1d3dcb8a417ed1d884cbcb5b81a38173c15ed.gz  
darcsd49fc3 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/ 
20070924214358-92b7f-0d177d6726975bc04dc0066d830ae4eae5919094.gz  
darcs1ef2a4 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/ 
20070823210315-92b7f-888fbf1b1867ccda8218ad9f955b835cbd474164.gz  
darcsd68c51 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/ 
2007063254-92b7f-4b19b2708a8832f8ec8eea6968c2ef99315355e4.gz  
darcsc5b11a ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/ 
20070611162221-92b7f-2c56a212abacccdc16da52833225fdafe51d8b64.gz  
darcsc78136 ; # AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/20070504024220-4233b- 
f5dfde96dca1eb23bfb22cfe2dd412c8456b826b.gz darcsfce705 ; # AsIs AsIs  
Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/20070426182525- 
eb44e-97f3b8656eb88d899e2eced99ce2ce018451e600.gz darcs570878 ; #  
AsIs AsIs Null
scp -q [EMAIL PROTECTED]:playground/allmydata/tahoe/ 
trunk/_darcs/inventories/ 
20070425235105-92b7f-52f87be1b8b1bca805373129e02bf9c4c42e89b9.gz  
darcs33bb7a ; # AsIs AsIs Null

Copying patches...
sftp [EMAIL PROTECTED] ; # File "darcs58163c" File  
"darcsd124bd" Null

Patches copied
Repo lazily read
Repo local: "False"
Applying patch 1 of 1979: Sat Oct 21 21:10:04 Mountain Daylight Time  
2006  [EMAIL PROTECTED]
  * tool to generate an overhead/alacrity table for various hashing  
schemes
Applying patch 2 of 1979: Sat Oct 21 21:39:12 Mountain Daylight Time  
2006  [EMAIL PROTECTED]

  * emit tables up to 1TiB




zooko.darcs2_get_d2f_verbose_debug_verbose_sshcm.log.txt.7z
Description: Binary data
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

On Jan 25, 2008, at 4:39 PM, Alexander Staubo wrote:

> On 1/26/08, David Roundy <[EMAIL PROTECTED]> wrote:
>> I think this would definitely be harder than
>> learning haskell.  Which isn't to say it's not a reasonable  
>> problem to
>> tackle, but rather that your lack of Haskell knowledge isn't an  
>> excuse!
>
> Really? I don't think it sounds difficult at all in any of the
> languages I do know. :) If it's designed to be purely sequentially
> pipelined, you don't even need to handle much state.

So clearly what we need here is a pair of programmers -- one already  
versant in Haskell and the other (Alexander) already skilled at  
custom pipelined network protocols -- to work together on this task.

Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[Alexander Staubo]

On 1/26/08, Alexander Staubo <[EMAIL PROTECTED]> wrote:
[...]
> The command set required is fairly small. I'm sure I'm missing a few
> details, but you need at least get, put, rm, mv, mkdir, stat -- and
> they would all be very simple, just the barest minimum needed to serve
> Darcs' needs.

Actually, you don't even need this, do you? When pulling, all you need
remotely is a "get" command. When pushing, the remote Darcs is doing
all the work anyway, so you don't need rm, mv etc.

Alexander.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

On Jan 25, 2008, at 4:08 PM, David Roundy wrote:

> It also would require that we develop an entire ftp protocol including
> transmition of errors.

This part I don't understand.  You just define a protocol, like:

I send you a stream of patchids (patchids have a known length, so  
there doesn't need to be any terminating character or length  
encoding), and you send me a stream of patches, with each patch  
prefixed by its length in bytes expressed as a big-endian, unsigned 8  
byte integer.

This would be a rich enough protocol to do the job of an FTP server,  
but it doesn't need to be.

>   I think this would definitely be harder than
> learning haskell.  Which isn't to say it's not a reasonable problem to
> tackle, but rather that your lack of Haskell knowledge isn't an  
> excuse!

Heh heh.  To me, implementing network protocols is easy, and learning  
Haskell is hard.

Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[Alexander Staubo]

On 1/26/08, David Roundy <[EMAIL PROTECTED]> wrote:
> Yes, that would be nice.  It *would* add a requirement for darcs to be
> present on the server (not a huge requirement, and we could always fall
> back on scp).  And most folks who get and pull over ssh will also be
> pushing over ssh, which already requires darcs be present on the server.

I agree.

> It also would require that we develop an entire ftp protocol including
> transmition of errors.

The command set required is fairly small. I'm sure I'm missing a few
details, but you need at least get, put, rm, mv, mkdir, stat -- and
they would all be very simple, just the barest minimum needed to serve
Darcs' needs.

You would not need working-directory management or globbing. As far as
I understand the Darcs repo format, you would not need ls, ch{mod,grp}
or ln.

> I think this would definitely be harder than
> learning haskell.  Which isn't to say it's not a reasonable problem to
> tackle, but rather that your lack of Haskell knowledge isn't an excuse!

Really? I don't think it sounds difficult at all in any of the
languages I do know. :) If it's designed to be purely sequentially
pipelined, you don't even need to handle much state.

> One might also wonder why we don't use the existing sftp program, which
> itself implements quite a similar protocol to what you're proposing.  In
> fact, we do so, but haven't wanted to deal with the headache of maintaining
> the connection, so we only do so during darcs get (when we know in advance
> an entire list of files we'd like to grab).  And this itself turns out to
> be somewhat fragile, as not all sftp servers are identical (and of course,
> putty never names any of its commands in a standard manner).

I forgot about the SFTP command mode. I don't know if it's pipelined,
but I can see how it could be fragile.

Alexander.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[David Roundy]

On Sat, Jan 26, 2008 at 12:09:05AM +0100, Alexander Staubo wrote:
> On 1/26/08, zooko <[EMAIL PROTECTED]> wrote:
> > > I presume you have a good reason not for allowing access via http?
> >
> > No, I haven't.  Now that I think about it, my not-so-good reason is
> > that simply that it doesn't set _darcs/prefs/defaultrepo for me so
> > that I can subsequently "darcs push".  In other words, it just seems
> > natural to use ssh when you have the privilege of doing ssh to the
> > server, because it has strictly better access options.
> 
> Not sure if David was asking in the general case -- but there are a
> few reasons why using HTTP is awkward. The main one is what Zooko
> points out here; if you're set up for HTTP, you can't push to a
> default repo.
> 
> But HTTP is also awkward for commercial projects where the repository
> cannot be made publicly available. I'm sure Darcs supports SSL/TLS,
> but you have to go all the way and make the server do HTTP
> authentication, and then I bet you have to put the user name and
> password in the URL so that Darcs can authenticate.

I understand that.  But for many folks it's a nicer workaround rather than
using rsync.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[David Roundy]

On Fri, Jan 25, 2008 at 04:08:09PM -0700, zooko wrote:
> >>The client machine running the "darcs get" is Windows, using the
> >>putty ssh executables, so I think they can't do the ssh ControlMaster
> >>feature.  I've added "--ssh-cm" to the command-line, and I can see
> >>that it is still doing an scp for each patch.
> >
> >Does sftp run on this machine?
> 
> There is an executable named "sftp.exe" which is actually putty's  
> psftp.exe, renamed.
> 
> >Do you see darcs ever calling sftp?
> 
> No.  Here is the full output:

Odd.  I'll have to look into this tomorrow.  Darcs should be using sftp for
a darcs get.  Is this using darcs-1 repositories? If not, could you recheck
with darcs1 repositories on both sides? Thanks!  (I checked similarly that
my copy of darcs2 uses sftp on darcs1 repositories.)
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[Alexander Staubo]

On 1/26/08, zooko <[EMAIL PROTECTED]> wrote:
> > I presume you have a good reason not for allowing access via http?
>
> No, I haven't.  Now that I think about it, my not-so-good reason is
> that simply that it doesn't set _darcs/prefs/defaultrepo for me so
> that I can subsequently "darcs push".  In other words, it just seems
> natural to use ssh when you have the privilege of doing ssh to the
> server, because it has strictly better access options.

Not sure if David was asking in the general case -- but there are a
few reasons why using HTTP is awkward. The main one is what Zooko
points out here; if you're set up for HTTP, you can't push to a
default repo.

But HTTP is also awkward for commercial projects where the repository
cannot be made publicly available. I'm sure Darcs supports SSL/TLS,
but you have to go all the way and make the server do HTTP
authentication, and then I bet you have to put the user name and
password in the URL so that Darcs can authenticate.

Alexander.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[David Roundy]

On Fri, Jan 25, 2008 at 11:37:12PM +0100, Alexander Staubo wrote:
> Not to hijack the thread, but a while back I suggested to Eric that it
> might be simpler to bypass the whole control master shebang, and
> implement a Darcs server mode. After all, all Darcs is doing is
> reading and writing remote files -- why invoke an entire process [1]
> for every file, and incur so much process and connection overhead?
> 
> A better solution is to do something like "ssh remoteserver darcs
> server", to which Darcs then can issue read and write commands in a
> simple text-based protocol. This protocol in turn could easily be
> pipelined for added performance.
> 
> Such a server mode would (1) sidestep the need for flaky control
> master connections, (2) would eliminate the need to rely on scp and
> sftp (and the need, on Windows, to set up the Putty semi-equivalents),
> and (3) would perform and function equally well on Windows. And (4) it
> should perform great.
> 
> I wish I could volunteer to implement something like this, if only to
> prove that it would be an improvement. Alas, my Haskell skillz are
> near zero.

Yes, that would be nice.  It *would* add a requirement for darcs to be
present on the server (not a huge requirement, and we could always fall
back on scp).  And most folks who get and pull over ssh will also be
pushing over ssh, which already requires darcs be present on the server.

It also would require that we develop an entire ftp protocol including
transmition of errors.  I think this would definitely be harder than
learning haskell.  Which isn't to say it's not a reasonable problem to
tackle, but rather that your lack of Haskell knowledge isn't an excuse!

One might also wonder why we don't use the existing sftp program, which
itself implements quite a similar protocol to what you're proposing.  In
fact, we do so, but haven't wanted to deal with the headache of maintaining
the connection, so we only do so during darcs get (when we know in advance
an entire list of files we'd like to grab).  And this itself turns out to
be somewhat fragile, as not all sftp servers are identical (and of course,
putty never names any of its commands in a standard manner).
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

> I presume you have a good reason not for allowing access via http?

No, I haven't.  Now that I think about it, my not-so-good reason is  
that simply that it doesn't set _darcs/prefs/defaultrepo for me so  
that I can subsequently "darcs push".  In other words, it just seems  
natural to use ssh when you have the privilege of doing ssh to the  
server, because it has strictly better access options.

So yes, advising my users to use http on get is a good work-around, too.

Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] darcs patch: Fix optimize_relink for POSIX environment (no cp -a or...

[quick]

The optimize_relink tests used some GNU-extension arguments to both cp and du, 
which
didn't work under MacOS 10.x.  Unfortunately, I couldn't find consistent ls or 
stat
output which would suffice, so this incorporates a little perl script to get at 
inodes
for hardlink validation; the result is a little less flexible (only files in a 
subdir),
but that's all the test needed.  This version works for Linux and MacOS.

Fri Jan 25 15:01:19 MST 2008  Kevin Quick <[EMAIL PROTECTED]>
  * Fix optimize_relink for POSIX environment (no cp -a or du -l support; now 
needs perl)

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


New patches:

[Fix optimize_relink for POSIX environment (no cp -a or du -l support; now needs perl)
Kevin Quick <[EMAIL PROTECTED]>**20080125220119] {
hunk ./tests/optimize_relink.sh 14
- -compare () { test `c -cs "$@"` -ne `c -csl "$@"`; }
- -c () { du "$@" | sed -n '$ { s/[[:space:]].*//; p; }'; }
+compare () {
+  echo 'use File::Basename; $res=0; while ($fn=<'$1'/*>) { $fn2="'$2'/" . basename($fn); @fd1=lstat($fn); @fd2=lstat($fn2); $res += ($fd1[1] != $fd2[1]);}; exit($res);' | perl
+}
hunk ./tests/optimize_relink.sh 31
- -cp -a x y
- -cp -al x z
- -if compare x y || ! compare x z
- -then
- -echo Filesystem for $d does not support hard links.
- -exit 0
+mkdir z1
+echo "hi" > z1/foo
+mkdir z2
+if ! ln z1/foo z2/foo ; then
+  echo No ln command for `pwd`; assuming no hard links.
+  exit 0
+fi
+if ! compare z1 z2 ; then
+  echo Filesystem for `pwd` does not support hard links.
+  exit 0
hunk ./tests/optimize_relink.sh 42
+cp -a x y || (rm -rf y && mkdir y && tar cf - -C x . | tar xpf - -C y .)
}

Context:

[resolve conflict with Eric on controlMasterPath.
David Roundy <[EMAIL PROTECTED]>**20080125203903] 
[More concise backup warning.
Eric Kow <[EMAIL PROTECTED]>**20071105012930] 
[Remove now obsolete wrapper for Map (we now require GHC >= 6.4).
Eric Kow <[EMAIL PROTECTED]>**20071105192636] 
[Modernise Data.Map import.
Eric Kow <[EMAIL PROTECTED]>**20071105192530] 
[Give ssh CM socket a unique name for each darcs process.
Eric Kow <[EMAIL PROTECTED]>**20071105021956
 Delete the socket in the unlikely event that a previous darcs had a socket
 with the same name left over.
] 
[Create ssh CM socket in $HOME/.darcs if possible.
Eric Kow <[EMAIL PROTECTED]>**20071105015525] 
[Refactor y/n prompts.
Eric Kow <[EMAIL PROTECTED]>**20071019213307] 
[issue578: steve and monica test for rolling back a rollback
Mark Stosberg <[EMAIL PROTECTED]>**20080118031606] 
[eliminate lazy parsing of patches, which gives bad error messages (issue364)
David Roundy <[EMAIL PROTECTED]>**20080125191836] 
[[issue492] Check that context file actually exists in darcs get.
Eric Kow <[EMAIL PROTECTED]>**20080125183741] 
[[issue227] Platform-independent absolute paths in get --context
Eric Kow <[EMAIL PROTECTED]>**20080125181702] 
[make uniqueoptions.sh test give friendlier output.
David Roundy <[EMAIL PROTECTED]>**20080125183430] 
[fix code to avoid duplicate --verbose in --help (so tests will pass).
David Roundy <[EMAIL PROTECTED]>**20080125183420] 
[Make verbosity flags advanced options universally.
Eric Kow <[EMAIL PROTECTED]>**20080125181005] 
[adding File::Temp 0.20 to tree for more consistent test results. It is GPL-licensed.
Mark Stosberg <[EMAIL PROTECTED]>**20080124033049] 
[update restrictive perms test to run a temporary directory and clean up after itself.
Mark Stosberg <[EMAIL PROTECTED]>**20080123000417
 Running in a tru temporary directory allows the potential to run tests in parallel.
] 
[update some ChangeLog entries to also credit those who contributed through bug reporting, test writing or feedback. 
Mark Stosberg <[EMAIL PROTECTED]>**20080122235435] 
[ issue602: part 1: Always prefer our private copy of Test::More over the system-wide one for more consistent results
Mark Stosberg <[EMAIL PROTECTED]>**20080124005407] 
[ issue602, part 2: freshen our versions of Test::More and Test::Builder
Mark Stosberg <[EMAIL PROTECTED]>**20080123013642] 
[More error messages for libwww.
Dmitry Kurochkin <[EMAIL PROTECTED]>**20080124092600] 
[issue608: a new test for 'mv', following Zooko's bug report
Mark Stosberg <[EMAIL PROTECTED]>**20080124013856] 
[report progress in writing the inventory out for hashed repos.
David Roundy <[EMAIL PROTECTED]>**20080125172017] 
[make empty key case of progress reporting fast.
David Roundy <[EMAIL PROTECTED]>**20080125171859] 
[fix issue where we overwrote prompt with progress info.
David Roundy <[EMAIL PROTECTED]>**20080125164609] 
[fix bug where we used show on an exception (and thus printed "User error").
David Roundy <[EMAIL PROTECTED]>**20080125164209
 This partially addresses issue168 by improving the error message.
] 
[add gnulib sha1.c file as a faster sha1 option.
David Roundy <[EMAIL PROTECTED]>**20080123212502] 
[fix embarrassing bug in External.
David Roundy <[EMAIL PROTECTED]>**20080125152329
 (which demonstrates that I didn't compile before pushing)
] 
[for no

[darcs-devel] darcs patch: Added "darcs show repo" command to display repository ...

[quick]

Wed Nov  7 17:16:04 MST 2007  Kevin Quick <[EMAIL PROTECTED]>
  * Added "darcs show repo" command to display repository information
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


New patches:

[Added "darcs show repo" command to display repository information
Kevin Quick <[EMAIL PROTECTED]>**20071108001604] {
hunk ./GNUmakefile 106
- -	Show.lhs ShowContents.lhs ShowFiles.lhs ShowTags.lhs	\
+	Show.lhs ShowContents.lhs ShowFiles.lhs ShowTags.lhs ShowRepo.lhs	\
hunk ./src/Darcs/Commands/Show.lhs 28
+import Darcs.Commands.ShowRepo ( show_repo )
hunk ./src/Darcs/Commands/Show.lhs 46
+  Command_data show_repo,
hunk ./src/Darcs/Commands/Show.lhs 65
+\input{Darcs/Commands/ShowRepo.lhs}
+
addfile ./src/Darcs/Commands/ShowRepo.lhs
hunk ./src/Darcs/Commands/ShowRepo.lhs 1
+%  Copyright (C) 2007 Kevin Quick
+%
+%  This program is free software; you can redistribute it and/or modify
+%  it under the terms of the GNU General Public License as published by
+%  the Free Software Foundation; either version 2, or (at your option)
+%  any later version.
+%
+%  This program is distributed in the hope that it will be useful,
+%  but WITHOUT ANY WARRANTY; without even the implied warranty of
+%  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+%  GNU General Public License for more details.
+%
+%  You should have received a copy of the GNU General Public License
+%  along with this program; see the file COPYING.  If not, write to
+%  the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+%  Boston, MA 02110-1301, USA.
+
+\subsubsection{darcs show repo}
+\label{show-repo}
+
+\options{show repo}
+
+The \verb!show repo! (or \verb!query repo!) displays information about
+the current repository: the location, the type, etc.  
+
+This is provided as informational output for two purposes: curious
+users and scripts invoking darcs.  For the latter, this information
+can be parsed to facilitate the script; for example, \verb!darcs query
+repo | grep Root: | awk {print $2}! can be used to locate the
+top-level _darcs directory from anyplace within a darcs repository
+working directory.
+
+\begin{code}
+module Darcs.Commands.ShowRepo ( show_repo ) where
+
+import Data.Char ( toLower )
+import Data.List ( intersperse )
+import Control.Monad ( when, unless )
+import Text.Html ( tag, stringToHtml )
+import FastPackedString ( unpackPS, nilPS )
+import Darcs.Arguments ( DarcsFlag(..), working_repo_dir, verbose, files, xmloutput )
+import Darcs.Commands ( DarcsCommand(..), nodefaults )
+import Darcs.Repository ( identifyRepository, amInRepository, read_repo )
+import Darcs.Repository.Internal ( Repository(..), RepoType(..) )
+import Darcs.Repository.Format ( RepoFormat(..) )
+import Darcs.Repository.Pristine ( Pristine )
+import Darcs.Repository.Prefs ( Cache, get_preflist )
+import Darcs.Patch.Ordered ( lengthRL, mapRL_RL, unsafeUnRL )
+import Darcs.Utils ( catchall )
+import Darcs.External ( fetchFilePS, Cachable(..) )
+\end{code}
+
+\begin{code}
+show_repo_help :: String
+show_repo_help =
+ "The repo command displays information about the current repository\n" ++
+ "(location, type, etc.).  Some of this information is already available\n" ++
+ "by inspecting files within the _darcs directory and some is internal\n" ++
+ "information that is informational only (i.e. for developers).  This\n" ++
+ "command collects all of the repository information into a readily\n" ++
+ "available source.\n"
+
+show_repo_description :: String
+show_repo_description = "Show repository information"
+\end{code}
+
+
+\begin{code}
+show_repo :: DarcsCommand
+show_repo = DarcsCommand { command_name = "repo",
+   command_help = show_repo_help,
+   command_description = show_repo_description,
+   command_extra_args = 0,
+   command_extra_arg_help = [],
+   command_command = repo_cmd,
+   command_prereq = amInRepository,
+   command_get_arg_possibilities = return [],
+   command_argdefaults = nodefaults,
+   command_advanced_options = [],
+   command_basic_options = [working_repo_dir, verbose, files, xmloutput] }
+\end{code}
+
+\begin{options}
+--files, --no-files
+\end{options}
+
+If the \verb!--files! option is specified (the default), then the
+\verb!show repo! operation will read patch information from the
+repository and display the number of patches in the repository.  The
+\verb!--no-files! option can be used to suppress this operation (and
+improve performance).
+
+\begin{code}
+repo_cmd :: [DarcsFlag] -> [String] -> IO ()
+repo_cmd opts _ = identifyRepository opts "." >>= showRepo (putInfoLn $ if XMLOutput `elem` opts then showInfoXML else showInfoUsr)
+\end{code}
+
+\begin{options}
+--human-readable, --xml-output
+\end{options}
+
+By default, the \ve

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

I agree that simply using a single ssh connection for all requests,  
and pipelining them, is the right way to do it.

Regards,

Zooko
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[Alexander Staubo]

On 1/25/08, zooko <[EMAIL PROTECTED]> wrote:
> The client machine running the "darcs get" is Windows, using the
> putty ssh executables, so I think they can't do the ssh ControlMaster
> feature.  I've added "--ssh-cm" to the command-line, and I can see
> that it is still doing an scp for each patch.

Putty doesn't cache connections, so performance is really going to
suck on Windows when using Darcs over SSH.

Not to hijack the thread, but a while back I suggested to Eric that it
might be simpler to bypass the whole control master shebang, and
implement a Darcs server mode. After all, all Darcs is doing is
reading and writing remote files -- why invoke an entire process [1]
for every file, and incur so much process and connection overhead?

A better solution is to do something like "ssh remoteserver darcs
server", to which Darcs then can issue read and write commands in a
simple text-based protocol. This protocol in turn could easily be
pipelined for added performance.

Such a server mode would (1) sidestep the need for flaky control
master connections, (2) would eliminate the need to rely on scp and
sftp (and the need, on Windows, to set up the Putty semi-equivalents),
and (3) would perform and function equally well on Windows. And (4) it
should perform great.

I wish I could volunteer to implement something like this, if only to
prove that it would be an improvement. Alas, my Haskell skillz are
near zero.

[1] Actually three: scp is a light wrapper which invokes ssh to run
(when copying a remote file) scp on the remote end to simply "cat" the
file. So each file invokes two local processes and one remote.

Alexander.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [issue610] when to use a $PAGER

[David Roundy]

On Fri, Jan 25, 2008 at 10:33:21PM -, Mark Stosberg wrote:
> This well-commented perl function "is_interactive" may be of interest 
> when writing code that decides when to show color and use a $PAGER by 
> default and when to not.  It is what the 'ack' utility uses:

The problem is that it's never safe to show color and use a $PAGER by
default at the same time, because color causes many popular pagers to barf
by default.  So it's one or the other, not both.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue610] when to use a $PAGER

[Mark Stosberg]

This well-commented perl function "is_interactive" may be of interest 
when writing code that decides when to show color and use a $PAGER by 
default and when to not.  It is what the 'ack' utility uses:

http://search.cpan.org/src/DCONWAY/IO-Interactive-v0.0.3/lib/IO/Interactive.pm

( ack is a programmer-friendly alternative to "grep" ).

   Mark

--
title: wish: hide advanced with --help-advanced -> when to use a $PAGER

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [issue610] wish: hide advanced with --help-advanced

[David Roundy]

On Fri, Jan 25, 2008 at 10:23:21PM -, Mark Stosberg wrote:
> Related: I believe "less" doesn't handle color escape codes by default. 
> You have to invoke it with "-R" for that.

Right, which is why we have an environment variable to set darcs to always
colorize.  You can also set an environment variable to make less always
behave as -R.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue610] wish: hide advanced with --help-advanced

[Mark Stosberg]

Wagner Ferenc wrote:
> 
> David Roundy <[EMAIL PROTECTED]> writes:
> 
>> On Thu, Jan 24, 2008 at 09:56:18AM +0100, Nicolas Pouillard wrote:
>>> What about automatically launching a pager ($PAGER) when the output is a 
>>> tty?
>>>
>>> Note: That's git default
>> I think this is probably the better option.
> 
> It would be especially nice if the pager preserved coloring of the
> output.  Is there a way to get darcs1 output escape sequences when
> writing to a pipe?  Er, I could as well have a look...

I think what you are getting at might be how the defaults are set up for 
the Perl-based 'ack' program:

If the output is a tty, color is on by default. However, if output is a 
pipe, color is off by default, since other programs tend to expect plain 
text.

Related: I believe "less" doesn't handle color escape codes by default. 
You have to invoke it with "-R" for that.

   Mark

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[David Roundy]

On Fri, Jan 25, 2008 at 02:17:45PM -0700, zooko wrote:
> One practical consequence of these measurements is that I will  
> strongly encourage my current users to use "rsync -r" instead of  
> "darcs get".

I presume you have a good reason not for allowing access via http?
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[David Roundy]

On Fri, Jan 25, 2008 at 03:07:49PM -0700, zooko wrote:
> The client machine running the "darcs get" is Windows, using the  
> putty ssh executables, so I think they can't do the ssh ControlMaster  
> feature.  I've added "--ssh-cm" to the command-line, and I can see  
> that it is still doing an scp for each patch.

Does sftp run on this machine? Do you see darcs ever calling sftp? It ought
to, I believe, when running darcs get (and that should give better speed
than using all-scp, since we then grab a bunch of files at once, with a
single ssh connection made).
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

The client machine running the "darcs get" is Windows, using the  
putty ssh executables, so I think they can't do the ssh ControlMaster  
feature.  I've added "--ssh-cm" to the command-line, and I can see  
that it is still doing an scp for each patch.

cf issue28, issue29, issue504
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[David Roundy]

On Fri, Jan 25, 2008 at 02:46:50PM -0700, zooko wrote:
> I added "--debug-verbose" and sure enough it is calling scp for, it  
> looks like, every single patch.

Ah, it would have helped a lot if you had mentioned from the start that you
were testing over ssh! As Alexander suggested, trying again with
control-master enabled would be worthwhile.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[David Roundy]

On Fri, Jan 25, 2008 at 10:44:59PM +0100, Alexander Staubo wrote:
> > > You should be able to see all the SSH commands
> > > issued by Darcs by adding "-v -v -v".
> >
> > Actually, I did have "--verbose --verbose --verbose" on the command-
> > line, but nothing about ssh command was emitted, and in fact only 3
> > lines were emitted:
> 
> I see. I know Eric Kow added -v counting and SSH logging, but his
> patches might not have been included on the branch from which David
> builds darcs-2.

The -v counting was replaced with a --debug flag.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

I added "--debug-verbose" and sure enough it is calling scp for, it  
looks like, every single patch.

Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[Alexander Staubo]

On 1/25/08, zooko <[EMAIL PROTECTED]> wrote:
> > Are you using Darcs' control master feature?
>
> I measured the default behavior of "darcs get --darcs-2", "darcs get
> --hashed" and "darcs get", with the matching formats of source
> (remote) repositories.

The control master feature was turned off after reports that it was
buggy on MacOS X and possibly Linux. As far as I know, the bug does
manifest itself on Linux (and Leopard apparently fixes what looks a
lot like a kernel bug in Tiger).

You really should run with the control master turned on -- it caches
SSH connections and thus eliminates all the connection setup/teardown
overhead, which is fairly expensive. To do this, add --ssh-cm to the
command line.

> > You should be able to see all the SSH commands
> > issued by Darcs by adding "-v -v -v".
>
> Actually, I did have "--verbose --verbose --verbose" on the command-
> line, but nothing about ssh command was emitted, and in fact only 3
> lines were emitted:

I see. I know Eric Kow added -v counting and SSH logging, but his
patches might not have been included on the branch from which David
builds darcs-2.

On Linux you should be able to trace SSH invocations with something like:

  strace -p PID 2>&1 | grep "exec.*(ssh|scp|sftp)"

Alexander.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

> Are you using Darcs' control master feature?

I measured the default behavior of "darcs get --darcs-2", "darcs get  
--hashed" and "darcs get", with the matching formats of source  
(remote) repositories.

> You should be able to see all the SSH commands
> issued by Darcs by adding "-v -v -v".

Actually, I did have "--verbose --verbose --verbose" on the command- 
line, but nothing about ssh command was emitted, and in fact only 3  
lines were emitted:

Writing working directory contents...
Copying prefs
Finished getting.

So there was no progress output during the long wait.

Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[Alexander Staubo]

On 1/25/08, zooko <[EMAIL PROTECTED]> wrote:
> Here are the times for replicating an entire repository of http://
> allmydata.org including all of its history:
[...]

Are you using Darcs' control master feature? If not, I bet it's
spawning a bunch of separate scp operations, and the setup overhead of
these is pretty huge. You should be able to see all the SSH commands
issued by Darcs by adding "-v -v -v".

Alexander.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] performance measurements of "darcs get", "hg clone", and "rsync -r"

[zooko]

On Jan 23, 2008, at 9:46 AM, David Roundy wrote:

> Two cases I'd like to see is darcs 2 pre with hashed-format on each  
> side,
> and darcs 2 pre with old-format on each side.

Here are the times for replicating an entire repository of http:// 
allmydata.org including all of its history:

*  darcs2pre with hashed format on each side:

14m19s, 12m38s, 11m27s

*  darcs2pre with old format on each side:

16m14s, 20m7s, 17m9s

*  darcs2pre with darcs-2-format on each side:

26m45s, 11m23s, 14m28s

Note that there was no progress-indication for any of these.


*  "hg clone --pull" with the complete history of the repository in  
hg format (thanks to tailor)

2m44s, 2m30s

*  "rsync -r" of a darcs-2 format repository

2m25s, 2m25s

*  rsync of a very tightly compressed (7z) tarball containing all of  
the information of the darcs-2-format repository:

1m9s, 1m12s


One practical consequence of these measurements is that I will  
strongly encourage my current users to use "rsync -r" instead of  
"darcs get".

Regards,

Zooko

___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue589] buggy error message: Impossible case in darcs PatchRead.lhs or Patch/Read.lhs

[Mark Stosberg]

> Sorry, I'm going to need some advice: I can't figure out how to get your 
> latest version.  Running `darcs pull` tells me that I'm up to date, 
> which I suspect is because I'm on http://darcs.net/repos/stable ... 
> unfortunately a fairly quick search gives me no clue as to where the 
> current bleeding edge version of darcs is!  Sorry for being obtuse here.

Michael,

darcs get  http://darcs.net/repos/unstable

Should work for you.

   Mark

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [issue589] buggy error message: Impossible case in darcs PatchRead.lhs or Patch/Read.lhs

[Michael Abbott]

On Fri, 25 Jan 2008, David Roundy wrote:
> Micheal, have you still got your repository/proxy combination that 
> corrupts patches?
>
> If so, could you try this test again with the latest darcs 2? I've made 
> a change that will (I think) improve the error message (by changing to a 
> strict parsing), but I'd rather close this after hearing from you that 
> it's good enough.

Alas, after I complained, the administrators fixed the problem (by adding 
the web site to an exceptions list, the idiots, but you can't have 
everything), so alas I can't reproduce the problem anymore!

However, come to think of it, I can reproduce it simply by trashing one of 
the patches in place and running `darcs repair`.  Let's see...

Sorry, I'm going to need some advice: I can't figure out how to get your 
latest version.  Running `darcs pull` tells me that I'm up to date, which 
I suspect is because I'm on http://darcs.net/repos/stable ... 
unfortunately a fairly quick search gives me no clue as to where the 
current bleeding edge version of darcs is!  Sorry for being obtuse here.

I'll have a go some time tomorrow if you can point me in the right 
direction.

> Thanks for your bug report!


P.S. The page http://www.darcs.net/manual/node3.html still refers me to 
http://abridgegame.org/repos/darcs which is dormant;  I think ertai 
(message 2538) said they'd made a patch to fix this?
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] new bugs.darcs.net home page

[David Roundy]

On Thu, Jan 17, 2008 at 11:13:11PM -0500, Mark Stosberg wrote:
> I have now updated the homepage of http://bugs.darcs.net to be more
> useful.
> 
> It now runs a more useful query for the bugs to display by default,
> eliminating bugs that are in any state of being resolved, "wont-fix",
> etc. This should make it easier to find bugs that actually need
> attention.

Great! How did you manage this, and can we achieve the same result with our
own searches? It'd be nice to update the "Show Unassigned" search in the
same way, and we could also introduce canned searches for
"non-feature-request/wishlist/etc" bugs.

Thanks for the work!

David
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue589] buggy error message: Impossible case in darcs PatchRead.lhs or Patch/Read.lhs

[David Roundy]

Micheal, have you still got your repository/proxy combination that corrupts 
patches?

If so, could you try this test again with the latest darcs 2? I've made a change
that will (I think) improve the error message (by changing to a strict parsing),
but I'd rather close this after hearing from you that it's good enough.

Thanks for your bug report!

David

--
status: in-progress -> testing

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [issue610] wish: hide advanced with --help-advanced

[Wagner Ferenc]

David Roundy <[EMAIL PROTECTED]> writes:

> On Thu, Jan 24, 2008 at 09:56:18AM +0100, Nicolas Pouillard wrote:
>> What about automatically launching a pager ($PAGER) when the output is a tty?
>> 
>> Note: That's git default
>
> I think this is probably the better option.

It would be especially nice if the pager preserved coloring of the
output.  Is there a way to get darcs1 output escape sequences when
writing to a pipe?  Er, I could as well have a look...
-- 
Thanks,
Feri.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [issue610] wish: hide advanced with --help-advanced

[David Roundy]

On Thu, Jan 24, 2008 at 09:56:18AM +0100, Nicolas Pouillard wrote:
> What about automatically launching a pager ($PAGER) when the output is a tty?
> 
> Note: That's git default

I think this is probably the better option.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] darcs patch: [issue227] Platform-independent absolute paths in get ...

[E . Y . Kow]

Fri Jan 25 18:17:02 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
  * [issue227] Platform-independent absolute paths in get --context

New patches:

[[issue227] Platform-independent absolute paths in get --context
Eric Kow <[EMAIL PROTECTED]>**20080125181702] {
hunk ./src/Darcs/Commands/Get.lhs 64
+import Darcs.URL ( is_absolute )
hunk ./src/Darcs/Commands/Get.lhs 123
-  fix_context o@(Context ('/':_):_) = o
+  fix_context o@(Context f:_) | is_absolute f = o
}

Context:

[report progress in writing the inventory out for hashed repos.
David Roundy <[EMAIL PROTECTED]>**20080125172017] 
[make empty key case of progress reporting fast.
David Roundy <[EMAIL PROTECTED]>**20080125171859] 
[fix issue where we overwrote prompt with progress info.
David Roundy <[EMAIL PROTECTED]>**20080125164609] 
[fix bug where we used show on an exception (and thus printed "User error").
David Roundy <[EMAIL PROTECTED]>**20080125164209
 This partially addresses issue168 by improving the error message.
] 
[add gnulib sha1.c file as a faster sha1 option.
David Roundy <[EMAIL PROTECTED]>**20080123212502] 
[fix embarrassing bug in External.
David Roundy <[EMAIL PROTECTED]>**20080125152329
 (which demonstrates that I didn't compile before pushing)
] 
[for now, print progress reports to stdout.
David Roundy <[EMAIL PROTECTED]>**20080125152105
 My hope is that this will alleviate some of the issues with progress
 reports overwriting prompts.
] 
[revamp progress reporting, making it more efficient and adding more output.
David Roundy <[EMAIL PROTECTED]>**20080125151540
 Note that there is still at least one time sink that remains to be identified.
] 
[avoid creating darcs-ssh if we aren't using ControlMaster. (issue613)
David Roundy <[EMAIL PROTECTED]>**20080125150846] 
[fix bug where darcs-ssh got even worse name (issue613).
David Roundy <[EMAIL PROTECTED]>**20080125150355] 
[provide more detailed progress reports in HashedIO.
David Roundy <[EMAIL PROTECTED]>**20080124145156] 
[print additional debug data in Progress.
David Roundy <[EMAIL PROTECTED]>**20080124145114] 
[add a few more debug messages in Repository.Internal.
David Roundy <[EMAIL PROTECTED]>**20080124144829] 
[fix incorrect report that we were reading patches.
David Roundy <[EMAIL PROTECTED]>**20080124125040] 
[reenable mandatory sha1 checks, now that we can link with a faster sha1.
David Roundy <[EMAIL PROTECTED]>**20080123203104] 
[remove (broken) git support and add openssl sha1 support.
David Roundy <[EMAIL PROTECTED]>**20080123202025
 These two changes got merged together as I was introducing the configure.ac
 changes to support openssl as a sha1 alternative to our Haskell code.
 
 (Yes, I'm lazy.)
] 
[remove redundant hash checks in hashed IO code.
David Roundy <[EMAIL PROTECTED]>**20080123173022] 
[output nicer progress in convert.
David Roundy <[EMAIL PROTECTED]>**20080123170428] 
[output timings when --timings is specified.
David Roundy <[EMAIL PROTECTED]>**20080123170314] 
[remove inaccurate message in convert.
David Roundy <[EMAIL PROTECTED]>**20080123170243] 
[use debugMessage in HashedIO.
David Roundy <[EMAIL PROTECTED]>**20080123160835] 
[add --timings flag (that as yet does nothing).
David Roundy <[EMAIL PROTECTED]>**20080123154931] 
[Major Perl test suite clean-up.
Mark Stosberg <[EMAIL PROTECTED]>**20080120035651
 The primary purpose of this patch was make sure all the tests are executed in
 randomly named directories, which allows us to run Perl tests in parallel, 
 without the directory names collided. 
 
 This isn't enabled by default for "make test", but it is there to play with.
 In the test directory, you can now do:
 
 ./bin/prove -j9 *.pl 
 
 to run 9 tests in parallel. There is also "--fork"
 option which should be a win on multi-CPU computers. 
 See "perldoc ./bin/prove" for details. 
 
 As part of this, a lot of boiler-plate code at the top and bottom of the
 scripts could be eliminated, and I made few other minor style clean-ups
 while I had the files open. 
 
 There should be no functional changes to the tests. 
] 
[Take advantage of new Perl testing infrastructure by eliminating needless --ignore-time mentions
Mark Stosberg <[EMAIL PROTECTED]>**20080120005242] 
[Take advantage of updated Perl testing infrastructure by removing needless author mentions in tests
Mark Stosberg <[EMAIL PROTECTED]>**20080120004503] 
[use --ignore-time in tests instead of "sleep", for faster, more reliable results
Mark Stosberg <[EMAIL PROTECTED]>**20080118030241] 
[Issue395: avoid single letter patch names in the test suite.  
Mark Stosberg <[EMAIL PROTECTED]>**20080118020634] 
[add regression test for amend-record removed file
Tommy Pettersson <[EMAIL PROTECTED]>**2008013231] 
[use UTC in date matching test untill match handles time zones
Tommy Pettersson <[EMAIL PROTECTED]>**20080122134322] 
[fix bug with timestamps and obliterate.
David Roundy <[EMAIL PROTECTED]>**2008014607] 
[Test: unpull may hide

Re: [darcs-devel] darcs patch: Make verbosity flags advanced options un... (and 2 more)

[David Roundy]

On Fri, Jan 25, 2008 at 06:40:59PM +, [EMAIL PROTECTED] wrote:
> Argh.  Sorry for the resend.  Forgot about the dependencies.
> Note: neither issue227 nor issue492 are tested.
> 
> Fri Jan 25 18:10:05 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
>   * Make verbosity flags advanced options universally.
> 
> Fri Jan 25 18:17:02 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
>   * [issue227] Platform-independent absolute paths in get --context
> 
> Fri Jan 25 18:37:41 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
>   * [issue492] Check that context file actually exists in darcs get.

I've applied these, and will push them pending tests.  Thanks, Eric!
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: adding File::Temp 0.20 to tree for more ... (and 2 more)

[David Roundy]

On Fri, Jan 25, 2008 at 12:57:46PM -0500, David Roundy wrote:
> On Wed, Jan 23, 2008 at 11:43:53PM -0500, Mark Stosberg wrote:
> > Wed Jan 23 22:30:49 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
> >   * adding File::Temp 0.20 to tree for more consistent test results. It is 
> > GPL-licensed.
> > 
> > Wed Jan 23 23:41:37 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
> >   * improved tempdir handling-- include the script name in the directory 
> > name, and better auto-cleanup
> > 
> > Wed Jan 23 23:42:45 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
> >   * make sure we are outside of the tempdir so the auto-cleanup can happen.
> 
> Thanks, Mark! Applied.

I take this back.  The improved tempdir handling causes add_in_subdir.pl to
fail, so I've only applied the first of these three.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] darcs patch: Make verbosity flags advanced options un... (and 2 more)

[E . Y . Kow]

Argh.  Sorry for the resend.  Forgot about the dependencies.
Note: neither issue227 nor issue492 are tested.

Fri Jan 25 18:10:05 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
  * Make verbosity flags advanced options universally.

Fri Jan 25 18:17:02 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
  * [issue227] Platform-independent absolute paths in get --context

Fri Jan 25 18:37:41 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
  * [issue492] Check that context file actually exists in darcs get.


New patches:

[Make verbosity flags advanced options universally.
Eric Kow <[EMAIL PROTECTED]>**20080125181005] {
hunk ./src/Darcs/Commands.lhs 157
 command_alloptions DarcsCommand { command_basic_options = opts1
 , command_advanced_options = opts2 }
 = (opts1 ++ [disable, help],
-   opts2 ++ [posthook_cmd, posthook_prompt
+   any_verbosity ++ opts2 ++
+[posthook_cmd, posthook_prompt
 ,prehook_cmd, prehook_prompt])
 
 --  Supercommands cannot be disabled.
hunk ./src/Darcs/Commands/Add.lhs 26
 
 import Darcs.Commands
 import Darcs.Arguments (noskip_boring, allow_caseonly,
-   any_verbosity, fancy_move_add,
+   fancy_move_add,
recursive, working_repo_dir, dry_run, umask_option,
list_files, list_unregistered_files,
DarcsFlag (AllowCaseOnly, Boring, Recursive,
hunk ./src/Darcs/Commands/Add.lhs 85
 command_basic_options =
 [noskip_boring, allow_caseonly,
  recursive "add contents of subdirectories",
- fancy_move_add]++ any_verbosity++[
+ fancy_move_add,
  working_repo_dir, dry_run]}
 \end{code}
 
hunk ./src/Darcs/Commands/Changes.lhs 37
 possibly_remote_repo_dir, get_repourl,
 working_repo_dir,
 summary, changes_reverse,
-match_several_or_range, any_verbosity,
+match_several_or_range,
 all_interactive, showFriendly,
 ssh_cm
   )
hunk ./src/Darcs/Commands/Changes.lhs 91
 command_advanced_options = [ssh_cm],
 command_basic_options = [match_several_or_range,
 changes_format,
-summary]++ any_verbosity++[
+summary,
 changes_reverse,
 possibly_remote_repo_dir,
 working_repo_dir,
hunk ./src/Darcs/Commands/Check.lhs 29
 
 import Darcs.Commands ( DarcsCommand(..), nodefaults )
 import Darcs.Arguments ( DarcsFlag( Quiet, Verbose, NoTest, LeaveTestDir ),
-partial_check, any_verbosity, notest,
+partial_check, notest,
 verify_hash,
 leave_test_dir, working_repo_dir,
   )
hunk ./src/Darcs/Commands/Check.lhs 90
   command_argdefaults = nodefaults,
   command_advanced_options = [],
   command_basic_options = [partial_check,
-  verify_hash]++
-  any_verbosity++[notest,
+  verify_hash,
+  notest,
   leave_test_dir,
   working_repo_dir
  ]}
hunk ./src/Darcs/Commands/Convert.lhs 35
 import Darcs.Arguments ( DarcsFlag( AllowConflicts, WorkDir,
 SetScriptsExecutable, UseFormat2, NoUpdateWorking,
 Verbose, Quiet, Context ),
-any_verbosity, reponame,
+reponame,
 set_scripts_executable,
 ssh_cm )
 import Darcs.Repository ( withRepoLock, ($-), withRepositoryDirectory, read_repo,
hunk ./src/Darcs/Commands/Convert.lhs 82
 command_get_arg_possibilities = return [],
 command_argdefaults = nodefaults,
 command_advanced_options = [ssh_cm],
-command_basic_options = reponame: any_verbosity++[set_scripts_executable]}
+command_basic_options = [reponame,set_scripts_executable]}
 \end{code}
 \begin{code}
 convert_cmd :: [DarcsFlag] -> [String] -> IO ()
hunk ./src/Darcs/Commands/Get.lhs 33
 UseFormat2, UseOldFashionedInventory, UseHashedInventory,
SetS

Re: [darcs-devel] darcs patch: Make verbosity flags advanced options universally.

[David Roundy]

On Fri, Jan 25, 2008 at 06:10:59PM +, [EMAIL PROTECTED] wrote:
> Fri Jan 25 18:10:05 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
>   * Make verbosity flags advanced options universally.

Applied, thanks!
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] testing: tmp repo names and parallel testing

[David Roundy]

On Wed, Jan 23, 2008 at 11:51:56PM -0500, Mark Stosberg wrote:
> >> Sat Jan 19 22:56:51 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
> >>   * Major Perl test suite clean-up.
> >>   The primary purpose of this patch was make sure all the tests are 
> >> executed in
> >>   randomly named directories, which allows us to run Perl tests in 
> >> parallel,
> >>   without the directory names collided.
> >>
> >>   This isn't enabled by default for "make test", but it is there to 
> >> play with.
> >>   In the test directory, you can now do:
> >>
> >>   ./bin/prove -j9 *.pl
> >>
> >>   to run 9 tests in parallel. There is also "--fork"
> >>   option which should be a win on multi-CPU computers.
> >>   See "perldoc ./bin/prove" for details.
> >>
> >>   As part of this, a lot of boiler-plate code at the top and bottom of 
> >> the
> >>   scripts could be eliminated, and I made few other minor style 
> >> clean-ups
> >>   while I had the files open.
> >>
> >>   There should be no functional changes to the tests.
> >
> > I'm not sure I like this idea, as it seems likely to lead to greater
> > difficulty in understanding and debugging failing test results.  We can
> > already run up to six tests in parallel just by running make -j6 test.
> 
> Unless I misunderstand how this would work, it is quite likely both
> tests would have tried to create a repo named "temp1" in the same
> directory at the same time, and there would be a conflict.

It just works by using separate directories for the tests_shell,
test_shell_hashed etc.  It's all done in GNUMakefile, so you can read about
it there:

# $1 - description
# $2 - setup
# $3 - discard
# $4 - type
# $5 - extension
# $6 - harness
define harness
@echo Running $4 tests on $1 repositories...
@rm -rf tests-$4-$1.dir && cp -R tests tests-$4-$1.dir
@cd tests-$4-$1.dir && rm -rf .darcs; mkdir .darcs;\
 $2;\
 if [ -r $(TEST_FILTER_FILE) ];\
 then grep "\.$5" $(TEST_FILTER_FILE) | grep -v '^#' | $3 | xargs $6; \
 else ls *.$5 | sort -r | $3 | xargs $6; fi
@echo Done running tests on $1 repositories...
endef

It's been refactored recently by Eric, so it's got less duplication of code
and is more complicated.

> > Also, why make the directory names random?
> 
> It's the standard tempfile/tempdir way to name temporary files and
> directories.

Right, but that's because it's not standard to expect users to investigate
temporary files and directories manually, but that's precisely how I use
the test suite (when tests fail).

> > Couldn't we just generate a unique directory name from the test name
> > itself? So then init_tmp_repo() in test foo.pl would generate a
> > repository named temp1-foo.
> 
> I have now sent a patch for something like this, that, so that the
> directory name is like "add.pl-tmpdir-", where the  become
> random characters.

That's definitely better, except that it causes add_in_subdir.pl to fail
for me...  :(
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] darcs patch: Make verbosity flags advanced options universally.

[E . Y . Kow]

Fri Jan 25 18:10:05 GMT 2008  Eric Kow <[EMAIL PROTECTED]>
  * Make verbosity flags advanced options universally.

New patches:

[Make verbosity flags advanced options universally.
Eric Kow <[EMAIL PROTECTED]>**20080125181005] {
hunk ./src/Darcs/Commands.lhs 157
 command_alloptions DarcsCommand { command_basic_options = opts1
 , command_advanced_options = opts2 }
 = (opts1 ++ [disable, help],
-   opts2 ++ [posthook_cmd, posthook_prompt
+   any_verbosity ++ opts2 ++
+[posthook_cmd, posthook_prompt
 ,prehook_cmd, prehook_prompt])
 
 --  Supercommands cannot be disabled.
hunk ./src/Darcs/Commands/Add.lhs 26
 
 import Darcs.Commands
 import Darcs.Arguments (noskip_boring, allow_caseonly,
-   any_verbosity, fancy_move_add,
+   fancy_move_add,
recursive, working_repo_dir, dry_run, umask_option,
list_files, list_unregistered_files,
DarcsFlag (AllowCaseOnly, Boring, Recursive,
hunk ./src/Darcs/Commands/Add.lhs 85
 command_basic_options =
 [noskip_boring, allow_caseonly,
  recursive "add contents of subdirectories",
- fancy_move_add]++ any_verbosity++[
+ fancy_move_add,
  working_repo_dir, dry_run]}
 \end{code}
 
hunk ./src/Darcs/Commands/Changes.lhs 37
 possibly_remote_repo_dir, get_repourl,
 working_repo_dir,
 summary, changes_reverse,
-match_several_or_range, any_verbosity,
+match_several_or_range,
 all_interactive, showFriendly,
 ssh_cm
   )
hunk ./src/Darcs/Commands/Changes.lhs 91
 command_advanced_options = [ssh_cm],
 command_basic_options = [match_several_or_range,
 changes_format,
-summary]++ any_verbosity++[
+summary,
 changes_reverse,
 possibly_remote_repo_dir,
 working_repo_dir,
hunk ./src/Darcs/Commands/Check.lhs 29
 
 import Darcs.Commands ( DarcsCommand(..), nodefaults )
 import Darcs.Arguments ( DarcsFlag( Quiet, Verbose, NoTest, LeaveTestDir ),
-partial_check, any_verbosity, notest,
+partial_check, notest,
 verify_hash,
 leave_test_dir, working_repo_dir,
   )
hunk ./src/Darcs/Commands/Check.lhs 90
   command_argdefaults = nodefaults,
   command_advanced_options = [],
   command_basic_options = [partial_check,
-  verify_hash]++
-  any_verbosity++[notest,
+  verify_hash,
+  notest,
   leave_test_dir,
   working_repo_dir
  ]}
hunk ./src/Darcs/Commands/Convert.lhs 35
 import Darcs.Arguments ( DarcsFlag( AllowConflicts, WorkDir,
 SetScriptsExecutable, UseFormat2, NoUpdateWorking,
 Verbose, Quiet, Context ),
-any_verbosity, reponame,
+reponame,
 set_scripts_executable,
 ssh_cm )
 import Darcs.Repository ( withRepoLock, ($-), withRepositoryDirectory, read_repo,
hunk ./src/Darcs/Commands/Convert.lhs 82
 command_get_arg_possibilities = return [],
 command_argdefaults = nodefaults,
 command_advanced_options = [ssh_cm],
-command_basic_options = reponame: any_verbosity++[set_scripts_executable]}
+command_basic_options = [reponame,set_scripts_executable]}
 \end{code}
 \begin{code}
 convert_cmd :: [DarcsFlag] -> [String] -> IO ()
hunk ./src/Darcs/Commands/Get.lhs 33
 UseFormat2, UseOldFashionedInventory, UseHashedInventory,
SetScriptsExecutable,
Verbose, Quiet, Context ),
-any_verbosity, partial, reponame,
+partial, reponame,
 match_one_context, set_default, set_scripts_executable, nolinks,
 ssh_cm )
 import Darcs.Repository ( Repository, withRepository, ($-),

Re: [darcs-devel] darcs patch: adding File::Temp 0.20 to tree for more ... (and 2 more)

[David Roundy]

On Wed, Jan 23, 2008 at 11:43:53PM -0500, Mark Stosberg wrote:
> Wed Jan 23 22:30:49 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * adding File::Temp 0.20 to tree for more consistent test results. It is 
> GPL-licensed.
> 
> Wed Jan 23 23:41:37 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * improved tempdir handling-- include the script name in the directory 
> name, and better auto-cleanup
> 
> Wed Jan 23 23:42:45 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * make sure we are outside of the tempdir so the auto-cleanup can happen.

Thanks, Mark! Applied.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: More error messages for libwww.

[David Roundy]

On Thu, Jan 24, 2008 at 12:54:21PM +0300, Dmitry Kurochkin wrote:
> Thu Jan 24 12:26:00 MSK 2008  Dmitry Kurochkin <[EMAIL PROTECTED]>
>   * More error messages for libwww.

Thanks Dmitry! Applied.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: update some ChangeLog entries to also cr... (and 1 more)

[David Roundy]

On Wed, Jan 23, 2008 at 08:57:39PM -0500, Mark Stosberg wrote:
> Tue Jan 22 18:54:35 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * update some ChangeLog entries to also credit those who contributed 
> through bug reporting, test writing or feedback. 
> 
> Tue Jan 22 19:04:17 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * update restrictive perms test to run a temporary directory and clean up 
> after itself.
>   Running in a tru temporary directory allows the potential to run tests 
> in parallel.

Applied.  Thanks!
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: amend-recorded patches for issue602

[David Roundy]

On Wed, Jan 23, 2008 at 08:56:17PM -0500, Mark Stosberg wrote:
> Tue Jan 22 20:36:42 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   *  issue602, part 2: freshen our versions of Test::More and Test::Builder
> 
> Wed Jan 23 19:54:07 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   *  issue602: part 1: Always prefer our private copy of Test::More over the 
> system-wide one for more consistent results

Applied.  Thanks!
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] darcs patch: issue608: a new test for 'mv', following Zooko's bug r...

[David Roundy]

On Wed, Jan 23, 2008 at 08:46:36PM -0500, Mark Stosberg wrote:
> Wed Jan 23 20:38:56 EST 2008  Mark Stosberg <[EMAIL PROTECTED]>
>   * issue608: a new test for 'mv', following Zooko's bug report

Applied, thanks!
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] announcing darcs 2.0.0pre3

[David Roundy]

On Wed, Jan 23, 2008 at 03:26:51PM +, Simon Marlow wrote:
> There are still times when I see nothing happening, for example in the 
> unpull test on the GHC repo (see previous messages), the last progress 
> message I get is
> 
> Reading patches in /64playpen/simonmar/ghc-darcs2 17040
> 
> and it sits there for 7-8 seconds before completing.  Does this maybe shed 
> any light on why this unpull is 2 times slower than darcs1?

I've finally got an idea what's causing this.  Actually, there are three
operations in which the hashed repositories don't scale quite as well as
the old format--all of which are "pretty fast", so it's only visible on
huge repositories like yours.

The first two scale with the number of files and directories in the
repository, and both are essentially garbage collecting of the pristine
cache.

One goes through the _darcs/pristine.hashed/ directory and compares each
file there with a list of files that should be there, deleting those that
aren't present.  This is currently O(N^2) in the number of files and
directories in the repository, because we use a list to do this set
comparison.  Using Data.Set should make this O(NlogN) which probably is
more than enough to make this negligible.  It's already pretty fast, even
on the ghc repo, so this may not even be noticeable.

The second is similar to the first.  It's when we go through the global
cache to remove any unused files.  Here we use the link count, and remove
any that have a link count of 1.  This means running stat on each file to
get the link count, so this is only O(N) where N is the total number of
distinct files and directories (i.e. having different hashes) present in
*all* repositories that you use.  It scales better than the previous one,
but if stat is slow on the cache, or if you've got very many different
large repositories, it could be slow.

The third (and possibly largest) issue is the writing of the inventory
files.  This is (thankfully) independent of the number or size of files in
the repository, and only depends on the number of patches and tags.  It's
roughly O(N+M) where N is the number of patches and tags (with a different
prefactor on the two, but who cares?), and M is the number of new or
modified patches.  This isn't *bad* scaling, but when you've got 17k
patches, O(N) adds up.  This is most likely the primary culprit, but is
tricky to fix, since as far as I can imagine, we'd need to change the
PatchSet data type (currently just a nested RL list) to cache the hash
values, and change all functions manipulating them to invalidate the cache
when they make changes.  :(

I've added progress reporting to all three of these (and it seems like it's
working).  All three could be sped up in some way (the second, perhaps just
by avoiding writing "pristine" files to the global cache, or failing to
clean them up).  But I'd rather hear from you where the pain is before
going ahead, since I've got more work right now than I can handle.

Also, if you want (way!) more information when tracking down timings, the
progress reporting now interacts with the --debug flag to generate enough
data to kill a horse.  You could also add the --timings flag, which will
add some timestamps (alas, with only 1s resolution) that might be helpful.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue619] darcs optimize --uncompress" doesn't uncompress inventories, patches, or pristine.hashed

[Zooko]

New submission from Zooko <[EMAIL PROTECTED]>:

There's no particularly important reason to uncompress these things,  
so this issue could be solved by removing the --uncompress option  
from darcs optimize.

I am uncompressing these things in order to pack a complete repo  
tighter:

column 1: tar the whole repo, 2: tar and gzip -9 the whole repo,  
column 3: 7z the whole repo
row A: standard darcs-2 format, row B: run "gunzip -r" on it first to  
uncompress any gzipped files

1   2   3
A   40 M32 M27 M
B   83 M32 M18 M

size of resulting tarball

Regards,

Zooko

--
messages: 2743
nosy: beschmi, droundy, kowey, tommy, zooko
status: unread
title: darcs optimize --uncompress" doesn't uncompress inventories, patches, or 
pristine.hashed

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue618] rm repository and then optimize --uncompress sibling repo yields "failed to read patch:"..."user error"

[Zooko]

New submission from Zooko <[EMAIL PROTECTED]>:

Here is a bash script which, on my Mac OS X machine, with darcs-2  
(recent vintage), yields the following output:

darcs: failed to read patch:
Tue Jan  8 11:47:49 MST 2008  [EMAIL PROTECTED]
   tagged allmydata-tahoe-0.7.0
user error (Couldn't fetch `cc623980b78aec5413b56d51f7fba803672e6194'
in subdir patches from sources:

thisrepo:/Users/wonwinmcbrootles/playground/allmydata/tahoe/d2b/trunk- 
d2f-gotten
repo:/Users/wonwinmcbrootles/playground/allmydata/tahoe/d2b/trunk-d2f
)

$ time darcs get http://allmydata.org/source/tahoe/trunk
$ darcs convert trunk trunk-d2f
$ /bin/rm -rf ./trunk
$ cd trunk-d2f
$ darcs optimize --modernize-patches --reorder-patches --uncompress

--
messages: 2740
nosy: beschmi, droundy, kowey, tommy, zooko
status: unread
title: rm repository and then optimize --uncompress sibling repo yields "failed 
to read patch:"..."user error"

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue227] get --context fails with absolute path in win32

[Mark Stosberg]

Is there anyone on darcs-devel that could help with what looks an easy bug fix
for Windows? See the issue tracker link online for more context:

On Thu, Jan 24, 2008 at 02:32:01 -, Mark Stosberg wrote:
>   let opts = fix_context orig_opts
>   fix_context o@(Context ('/':_):_) = o
>   fix_context (Context f:os) = Context (former_dir++"/"++f):os
>   fix_context (o:os) = o : fix_context os
>   fix_context [] = []

Hmm, darcs does have a more platform-independent is_absolute function
(Darcs.Url, I think) we could use.  Want to try plugging it in?

--
nosy: +darcs-devel, jaredj, rgm
topic: +ProbablyEasy

__
Darcs bug tracker <[EMAIL PROTECTED]>

__
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] [issue589] buggy error message: Impossible case in darcs PatchRead.lhs or Patch/Read.lhs

[Michael Abbott]

On Fri, 25 Jan 2008, Mark Stosberg wrote:
> If this is just a string change to create a better error message, perhaps this
> issue could be resolved sooner?
>
> I still don't understand the issue well enough to even suggest what a better
> error message might be.

If we're just talking about improving the error message, then can I 
suggest something along the lines of:

Patch blah blah corrupted: unable to apply

where the blah blah bit gives the proper filename of the patch.  Then at 
least I can hunt it out and look inside it to see -- aha! my damned proxy 
again.

The current error message says (1) panic panic darcs is broken and (2) 
just identifies the patch by number.  The problem with this is that 
finding the patch file requires a little guesswork and the lucky fact that 
`ls` lists the patches in the same order as darcs was reading them.
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel