[darcs-devel] [issue408] Slow Push and Pull with Remote Repository

[Thomas David Baker]

New submission from Thomas David Baker <[EMAIL PROTECTED]>:

Hello,

I'm trying to solve a problem with a repository.  Each pull and push
takes about 3 minutes.

It is a large-ish repository (109MB total, _darcs folder is 61MB, 2370
changes according to "darcs changes").

I set up ssh/scp/sftp logging to see what it was doing, as described here:

http://wiki.darcs.net/index.html/DeveloperTips

It turns out that every time a pull or a push happens, hundreds of
commands like this are issued:

--

scp -o ControlPath=/tmp//darcs-ssh/[EMAIL PROTECTED]
[EMAIL 
PROTECTED]:ersmithers_darcs_test/_darcs/patches/20061024182332-db8d4-6e9902a182892864b23efc86fa31c3352b7b9e4e.gz
/home/bakert/testdarcs/ersmithers_darcs_test/darcsHhasMC

--

[EMAIL PROTECTED] is where the repository I am push/pull-ing from/to is.

As each one takes up to a second this is why it is so slow.  Even if I
check out a new repository, make a one line change and then push it
still does hundreds of these.

I've tried issuing "darcs checkpoint" on the remote copy to no avail.
Probably unrelatedly if I try and do "darcs optimize --reorder-patches
--checkpoint" on my local copy I get:

--

darcs: bug in darcs!
fromJust error at DarcsRepo.lhs:525 compiled 06:01:36 Sep 13 2006

--

I'm not a darcs expert by any means and I'm really seeking any advice.
 Is the scp stuff normal?  Is there anything I can do to stop it or
speed it up?

Please let me know if any other information would help diagnose the problem.

The output of darcs --exact-version is:

--

darcs compiled on Sep 13 2006, at 06:04:11
# configured Fri Jun 16 14:55:21 EDT 2006
./configure --no-create --no-recursion

Context:

[TAG 1.0.8
Tommy Pettersson <[EMAIL PROTECTED]>**20060616160213]

--

Any help you can give greatly appreciated.

Thanks,

Tom

--
messages: 1466
nosy: EricKow, anonymous, bakert, beschmi, droundy, tommy
status: unread
title: Slow Push and Pull with Remote Repository


Darcs issue tracker <[EMAIL PROTECTED]>



___
darcs-devel mailing list
darcs-devel@darcs.net
http://www.abridgegame.org/cgi-bin/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[Dylan Thurston]

On Wed, Feb 07, 2007 at 04:48:33PM -0500, Zachary P. Landau wrote:
> Maybe this afternoon I'll do a little searching about and see what
> people (other than that HOWTO) are saying about using mkdtemp.

Out of curiosity, I looked at the Debian tmpreaper package and what it
says; it has a README.security, where the maintainer disagrees with
Michal Zalewski about the practical implications.  See
http://ursine.ca/cgi-bin/dwww/usr/share/doc/tmpreaper/README.security.gz

Peace,
Dylan


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[Zachary P. Landau]

> I'm not sure, but I've gotten the impression that if tmp cleaners are used
> then /tmp is pretty much irredeemably insecure, and there's not much point
> worrying about that case.  But maybe there are folks who use tmp cleaners
> and also care about security.  After all, one reason to use tmp cleaners is
> simply to avoid DOS attacks that fill up tmp--which itself is a security
> issue.

Maybe this afternoon I'll do a little searching about and see what
people (other than that HOWTO) are saying about using mkdtemp.  The
non-security issue is: what do we for operating systems that don't
provide mkdtemp?  Maybe haskell.org should sponsor a compile farm for
every OS that runs GHC.  Then we could see if all of them support it.

> Wouldn't it be nice if everybody would just agree to be good?

Maybe nobody ever asked.  Let's change the world, right here on
darcs-devel:

I hereby promise to be a good person, in all aspects of my life,
including but not exclusively with regard to security issues.
-- Zachary P. Landau, 2007-02-07

If we can get 6 billion people to sign the above statement, we can just
use mktemp() in /tmp.

--
Zachary P. Landau <[EMAIL PROTECTED]>
GPG: gpg --recv-key 0xC9F82052 | http://divineinvasion.net/kapheine.asc


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[David Roundy]

On Wed, Feb 07, 2007 at 03:56:36PM -0500, Zachary P. Landau wrote:
> On Wed, Feb 07, 2007 at 03:29:06PM -0500, Dylan Thurston wrote:
> > On Wed, Feb 07, 2007 at 11:27:30AM -0800, David Roundy wrote:
> > > > I wonder if instead we should be using tmpfile().
> > > > ...
> > > 
> > > tmpfile() would be nice, but the file is deleted automatically when the
> > > handle is closed (perhaps deleted before the handle is returned?), so we
> > > can't use it when we want a filename that we can pass to emacs.
> > 
> > The standard solution is to use mkdtemp to create a dirctory (with mode
> > 0700) and then create a file in that directory with your choice of
> > name.  mkdtemp is not in POSIX currently, but it seems like it will be
> > added soon.
> 
> The Secure Programs HOWTO link does mention mkdtemp.  They only mention
> that tmp cleaners can cause a security issue with mkdtemp, as Thomas
> brought up last week.  I feel like tmp cleaners could cause other issues
> in general, but maybe if people are using them we still need to worry?

I'm not sure, but I've gotten the impression that if tmp cleaners are used
then /tmp is pretty much irredeemably insecure, and there's not much point
worrying about that case.  But maybe there are folks who use tmp cleaners
and also care about security.  After all, one reason to use tmp cleaners is
simply to avoid DOS attacks that fill up tmp--which itself is a security
issue.

Wouldn't it be nice if everybody would just agree to be good?
-- 
David Roundy
Department of Physics
Oregon State University


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[Zachary P. Landau]

On Wed, Feb 07, 2007 at 03:29:06PM -0500, Dylan Thurston wrote:
> On Wed, Feb 07, 2007 at 11:27:30AM -0800, David Roundy wrote:
> > > I wonder if instead we should be using tmpfile().
> > > ...
> > 
> > tmpfile() would be nice, but the file is deleted automatically when the
> > handle is closed (perhaps deleted before the handle is returned?), so we
> > can't use it when we want a filename that we can pass to emacs.
> 
> The standard solution is to use mkdtemp to create a dirctory (with mode
> 0700) and then create a file in that directory with your choice of
> name.  mkdtemp is not in POSIX currently, but it seems like it will be
> added soon.

The Secure Programs HOWTO link does mention mkdtemp.  They only mention
that tmp cleaners can cause a security issue with mkdtemp, as Thomas
brought up last week.  I feel like tmp cleaners could cause other issues
in general, but maybe if people are using them we still need to worry?

--
Zachary P. Landau <[EMAIL PROTECTED]>
GPG: gpg --recv-key 0xC9F82052 | http://divineinvasion.net/kapheine.asc


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[Dylan Thurston]

On Wed, Feb 07, 2007 at 11:27:30AM -0800, David Roundy wrote:
> > I wonder if instead we should be using tmpfile().
> > ...
> 
> tmpfile() would be nice, but the file is deleted automatically when the
> handle is closed (perhaps deleted before the handle is returned?), so we
> can't use it when we want a filename that we can pass to emacs.

The standard solution is to use mkdtemp to create a dirctory (with mode
0700) and then create a file in that directory with your choice of
name.  mkdtemp is not in POSIX currently, but it seems like it will be
added soon.

Peace,
Dylan Thurston


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[Zachary P. Landau]

> My concern is just that my understanding of how to properly use mkstemp()
> and tmpfile() is that you can't close the file or rely on its name, both of
> which we need to do in order to interface with external programs.
> 
> > I don't mean to sound like I'm set on using /tmp (or tmpfile()).  But it
> > would provide a clean solution to the temp file problem, so I don't want
> > to disregard it unless there really is a valid reason to avoid it.
> 
> A clean solution would certainly be nice... I just don't see a solution,
> unless it's just that "once you've created a file in /tmp, you can do
> whatever you like with it, as long as you don't delete it, and you're still
> safe from attack".  If that sentence were true, we'd be fine using /tmp
> with mkstemp() using the $TMPDIR (or just /tmp)---I still don't see how we
> could use tmpfile().

I've just skimmed over
http://tldp.org/HOWTO/Secure-Programs-HOWTO/avoid-race.html a bit.
There do appear to be reasons why you can't securely use mkstemp and
then close the file.  I'll read more details into that tonight.

But the general idea seems to be that that are a lot more issues with
/tmp then just the basic race condition with regard to the filename.
That page does start going into ways you can avoid using mkstemp and
still be secure, but it looks tricky.  And tricky usually means that we
will make a mistake somewhere and there will be a security flaw.

So I think you are justified in avoiding /tmp.  I hereby concede.  I do
still like the fallback of using $HOME/.darcs/tmp if we fail every other
method we have of getting a valid directory.  If we can all agree on
that, I'll whip up a patch to implement that.

--
Zachary P. Landau <[EMAIL PROTECTED]>
GPG: gpg --recv-key 0xC9F82052 | http://divineinvasion.net/kapheine.asc


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] [issue407] darcs send: Improvements on error message

[Markus Herhoffer]

New submission from Markus Herhoffer <[EMAIL PROTECTED]>:

If "darc send" fails the error message is absolutely no help for finding the
error. There should be some information, why sending failed or what to do to
make it work. Furthermore darcs should ask if the user wants to save the patch
locally instead (i.e. darcs send -o)

--
messages: 1465
nosy: EricKow, beschmi, d135-1r43, droundy, tommy
priority: feature
status: unread
title: darcs send: Improvements on error message


Darcs issue tracker <[EMAIL PROTECTED]>



___
darcs-devel mailing list
darcs-devel@darcs.net
http://www.abridgegame.org/cgi-bin/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[David Roundy]

On Wed, Feb 07, 2007 at 02:12:03PM -0500, Zachary P. Landau wrote:
> On Wed, Feb 07, 2007 at 07:20:13PM +0100, Juliusz Chroboczek wrote:
> > >> I think most /tmp dirs have the t-flag set, which means you must
> > >> be the owner of a file to delete it from the directory. In those
> > >> cases it seems safe, but I don't know for certain.
> > 
> > I'd formulate it in a different manner.  Using /tmp is most certainly
> > safe on sane Linux and BSD systems.  It's anyone guess what happens on
> > other OSes.
> > 
> > In other words -- unless there's someone here who fully understands
> > the semantics of the sticky bit on Solaris and HP/UX, it's not a can
> > of worms we want to open.
> 
> I wonder if instead we should be using tmpfile().  The Linux manpage for
> it says that it conforms to POSIX.1-2001, so I would hope the other OSes
> will implement it correctly.  The other benefit is that tmpfile() will
> determine the directory to use.  If some OS for some reason doesn't have
> a secure /tmp, then it might provide another directory for tmpfile to
> use.

tmpfile() would be nice, but the file is deleted automatically when the
handle is closed (perhaps deleted before the handle is returned?), so we
can't use it when we want a filename that we can pass to emacs.

> I guess my main issue is that if calls like mkstemp() and tmpfile() are
> meant for securely creating temporary files and an OS does not implement
> it correctly, darcs will only be one of the many applications that will
> be susceptible to attack.  The issue at that point would be the security
> of the OS, not of an application.  Sure a workaround for applications
> could be made, but that feels like a hack around a bigger problem.

My concern is just that my understanding of how to properly use mkstemp()
and tmpfile() is that you can't close the file or rely on its name, both of
which we need to do in order to interface with external programs.

> I don't mean to sound like I'm set on using /tmp (or tmpfile()).  But it
> would provide a clean solution to the temp file problem, so I don't want
> to disregard it unless there really is a valid reason to avoid it.

A clean solution would certainly be nice... I just don't see a solution,
unless it's just that "once you've created a file in /tmp, you can do
whatever you like with it, as long as you don't delete it, and you're still
safe from attack".  If that sentence were true, we'd be fine using /tmp
with mkstemp() using the $TMPDIR (or just /tmp)---I still don't see how we
could use tmpfile().
-- 
David Roundy
Department of Physics
Oregon State University


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[David Roundy]

On Wed, Feb 07, 2007 at 07:20:13PM +0100, Juliusz Chroboczek wrote:
> >> I think most /tmp dirs have the t-flag set, which means you must
> >> be the owner of a file to delete it from the directory. In those
> >> cases it seems safe, but I don't know for certain.
> 
> I'd formulate it in a different manner.  Using /tmp is most certainly
> safe on sane Linux and BSD systems.  It's anyone guess what happens on
> other OSes.
> 
> In other words -- unless there's someone here who fully understands
> the semantics of the sticky bit on Solaris and HP/UX, it's not a can
> of worms we want to open.

I'm mostly hesitant about using /tmp simply because I don't fully
understand the possible attacks.  If we are confident that (in the absence
of /tmp cleaners?) using /tmp (as we plan to use it) is safe on linux and
BSD systems, then I'd say we should go ahead and do this (obeying $TEMP,
etc, of course).  But I'd want someone we trust (e.g. you, Juliusz) who
knows about such questions to tell us that it is safe.  I've seen just
enough security-related statements to believe that I don't understand well
enough to predict the security implications involving use of /tmp.
-- 
David Roundy
Department of Physics
Oregon State University
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[Zachary P. Landau]

On Wed, Feb 07, 2007 at 07:20:13PM +0100, Juliusz Chroboczek wrote:
> >> I think most /tmp dirs have the t-flag set, which means you must
> >> be the owner of a file to delete it from the directory. In those
> >> cases it seems safe, but I don't know for certain.
> 
> I'd formulate it in a different manner.  Using /tmp is most certainly
> safe on sane Linux and BSD systems.  It's anyone guess what happens on
> other OSes.
> 
> In other words -- unless there's someone here who fully understands
> the semantics of the sticky bit on Solaris and HP/UX, it's not a can
> of worms we want to open.

I wonder if instead we should be using tmpfile().  The Linux manpage for
it says that it conforms to POSIX.1-2001, so I would hope the other OSes
will implement it correctly.  The other benefit is that tmpfile() will
determine the directory to use.  If some OS for some reason doesn't have
a secure /tmp, then it might provide another directory for tmpfile to
use.

I guess my main issue is that if calls like mkstemp() and tmpfile() are
meant for securely creating temporary files and an OS does not implement
it correctly, darcs will only be one of the many applications that will
be susceptible to attack.  The issue at that point would be the security
of the OS, not of an application.  Sure a workaround for applications
could be made, but that feels like a hack around a bigger problem.

I don't mean to sound like I'm set on using /tmp (or tmpfile()).  But it
would provide a clean solution to the temp file problem, so I don't want
to disregard it unless there really is a valid reason to avoid it.

--
Zachary P. Landau <[EMAIL PROTECTED]>
GPG: gpg --recv-key 0xC9F82052 | http://divineinvasion.net/kapheine.asc


signature.asc
Description: Digital signature
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

Re: [darcs-devel] Temporary files issue

[Juliusz Chroboczek]

>> I think most /tmp dirs have the t-flag set, which means you must
>> be the owner of a file to delete it from the directory. In those
>> cases it seems safe, but I don't know for certain.

I'd formulate it in a different manner.  Using /tmp is most certainly
safe on sane Linux and BSD systems.  It's anyone guess what happens on
other OSes.

In other words -- unless there's someone here who fully understands
the semantics of the sticky bit on Solaris and HP/UX, it's not a can
of worms we want to open.

Juliusz
___
darcs-devel mailing list
darcs-devel@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-devel

[darcs-devel] Re: Building darcs

[Simon Marlow]

Zachary P. Landau wrote:


When I'm working on the darcs code, I'm generally rebuilding the code
fairly often.  Unfortunately this seems to take a long time.  Certain
files, because of dependencies, have to build a lot of files again.  A
clean build on my machine, a 3.4ghz machine with 1 gig of ram, takes
somewhere around 15 minutes.  Each individual file takes a certain
amount of time, and then the final link at the end takes quite a bit by
itself.


When GHC compiles files with optimisation on, the .hi files contain a lot of 
information required for cross-module optimisation (the definitions of small 
functions, strictness properties, etc.).  This has the effect that more 
recompilation will be required when something changes: it's very likely that 
small changes to a module will cause the optimisation annotations in the .hi 
file to change, forcing more recompilation.  However when you compile without 
optimisation, the .hi file only contains the types of exported functions and 
datatypes, which is much less likely to change when you modify the file.  So for 
fast edit/compile cycles, you really want to compile with optimisation off.



The tips under 'Faster edit-compile cycle' on
http://darcs.net/DarcsWiki/DeveloperTips don't seem to be valid anymore.
I don't see O2 under GHCFLAGS.  I did try the ghci line but it fails
with an error when trying to link in curl.  I may try to fight with this
later, as it seems like a good option.  I'll update that page once I'm
sure I'm not just doing something stupid.

I also tried playing with the -j option with make, but that did a fairly
good job of bringing my system to its knees.


I would try -j2, or more depending on how many cores in your machine.  Reduce 
the number if you're running out of memory.


Cheers,
Simon


___
darcs-devel mailing list
darcs-devel@darcs.net
http://www.abridgegame.org/cgi-bin/mailman/listinfo/darcs-devel