Re: [PATCH] Documentation/git-clean.txt: don't mention deletion of .git/modules/*

[Mikko Rapeli]

On Tue, Feb 09, 2016 at 12:47:39AM +0200, Mikko Rapeli wrote:
> On Mon, Feb 08, 2016 at 05:36:50PM -0500, Matt McCutchen wrote:
> > On Mon, 2016-02-08 at 14:22 -0800, Junio C Hamano wrote:
> > > Matt McCutchen <m...@mattmccutchen.net> writes:
> > > 
> > > > I found no evidence of such behavior in the source code.
> > > > 
> > > > Signed-off-by: Matt McCutchen <m...@mattmccutchen.net>
> > > > ---
> > > 
> > > That was added last year at bcd57cb9 (Documentation/git-clean.txt:
> > > document that -f may need to be given twice, 2015-02-26).  It would
> > > be better to know what got changed since then--that is, was the
> > > additional text unnecessary even back then, or we made changes to
> > > the system since then and forgot to remove the added text.
> > > 
> > > Mikko, is this need to give -f twice still the case?
> > 
> > I know you probably want confirmation from Mikko, but I'll offer my
> > understanding.  There were two statements added in bcd57cb9:
> > 
> > 1. -f may need to be given twice to delete nested worktrees and
> > embedded repositories.  This is still true.
> >
> > 2. Deletion of submodule repositories under .git/modules is conditional
> > on -f being given twice.  AFAICT, this was wrong even back then: "git
> > clean" has never deleted such repositories under any conditions.
> 
> This is the use case which I've used double -f at work with several build
> jobs but with older 1:1.7.9.5-1ubuntu0.2 (Ubuntu 12.04) and 1:1.9.1-1ubuntu0.2
> (Ubuntu 14.04) versions of git.

Sorry, can't reproduce the problem where submodules stayed in the tree until
git clean was called with two -f's.

You are right in removing the second part.

> But I can confirm that git version 1:2.7.0~rc3-1 (Debian unstable) is no
> longer removing the git submodule trees from .git/modules with double -f.
> 
> At work, we really want to remove the .git/modules subtrees since we want to
> test changes to .git/modules structure via normal commits to the git trees.
> Thus we need a way removing all non-tracked files from the git tree
> which includes obsolete (or for test only) git submodule trees.

This usecase does not exists in git then. Using gerrit topics git branches
to test changes to submodule structure does not work since there is no way of
undoing these changes from the working tree.

I should not have started using submodules in the first place, sigh.

-Mikko
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Documentation/git-clean.txt: don't mention deletion of .git/modules/*

[Mikko Rapeli]

On Mon, Feb 08, 2016 at 05:36:50PM -0500, Matt McCutchen wrote:
> On Mon, 2016-02-08 at 14:22 -0800, Junio C Hamano wrote:
> > Matt McCutchen  writes:
> > 
> > > I found no evidence of such behavior in the source code.
> > > 
> > > Signed-off-by: Matt McCutchen 
> > > ---
> > 
> > That was added last year at bcd57cb9 (Documentation/git-clean.txt:
> > document that -f may need to be given twice, 2015-02-26).  It would
> > be better to know what got changed since then--that is, was the
> > additional text unnecessary even back then, or we made changes to
> > the system since then and forgot to remove the added text.
> > 
> > Mikko, is this need to give -f twice still the case?
> 
> I know you probably want confirmation from Mikko, but I'll offer my
> understanding.  There were two statements added in bcd57cb9:
> 
> 1. -f may need to be given twice to delete nested worktrees and
> embedded repositories.  This is still true.
>
> 2. Deletion of submodule repositories under .git/modules is conditional
> on -f being given twice.  AFAICT, this was wrong even back then: "git
> clean" has never deleted such repositories under any conditions.

This is the use case which I've used double -f at work with several build
jobs but with older 1:1.7.9.5-1ubuntu0.2 (Ubuntu 12.04) and 1:1.9.1-1ubuntu0.2
(Ubuntu 14.04) versions of git.

But I can confirm that git version 1:2.7.0~rc3-1 (Debian unstable) is no
longer removing the git submodule trees from .git/modules with double -f.

At work, we really want to remove the .git/modules subtrees since we want to
test changes to .git/modules structure via normal commits to the git trees.
Thus we need a way removing all non-tracked files from the git tree
which includes obsolete (or for test only) git submodule trees.

I will test the old versions tomorrow again. Maybe in older git versions
the code path from 1) deletes also the git modules from 2).

-Mikko
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Documentation/git-clean.txt: document that -f may need to be given twice

[Mikko Rapeli]

On Thu, Feb 26, 2015 at 02:56:40PM +0200, Mikko Rapeli wrote:
 This is needed in build automation where the tree really needs to
 be reset to known state.
 
 Signed-off-by: Mikko Rapeli mikko.rap...@iki.fi
 ---
  Documentation/git-clean.txt |8 ++--
  1 file changed, 6 insertions(+), 2 deletions(-)
 
 diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
 index 94b6d19..872ab45 100644
 --- a/Documentation/git-clean.txt
 +++ b/Documentation/git-clean.txt
 @@ -34,8 +34,12 @@ OPTIONS
  -f::
  --force::
   If the Git configuration variable clean.requireForce is not set
 - to false, 'git clean' will refuse to run unless given -f, -n or
 - -i.
 + to false, 'git clean' will refuse to delete files or directories
 + unless given -f, -n or -i. Git will refuse to delete directories
 + with .git sub directory or file unless a second -f
 + is given. This affects also git submodules where the storage area of

Oops, of is here twice.

 + of the removed submodule under .git/modules/ is not removed until
 + -f is give twice.
  
  -i::
  --interactive::
 -- 
 1.7.10.4
 
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Documentation/git-clean.txt: document that -f may need to be given twice

[Mikko Rapeli]

This is needed in build automation where the tree really needs to
be reset to known state.

Signed-off-by: Mikko Rapeli mikko.rap...@iki.fi
---
 Documentation/git-clean.txt |8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
index 94b6d19..bd4e93d 100644
--- a/Documentation/git-clean.txt
+++ b/Documentation/git-clean.txt
@@ -34,8 +34,12 @@ OPTIONS
 -f::
 --force::
If the Git configuration variable clean.requireForce is not set
-   to false, 'git clean' will refuse to run unless given -f, -n or
-   -i.
+   to false, 'git clean' will refuse to delete files or directories
+   unless given -f, -n or -i. Git will refuse to delete directories
+   with .git sub directory or file unless a second -f
+   is given. This affects also git submodules where the storage area
+   of the removed submodule under .git/modules/ is not removed until
+   -f is give twice.
 
 -i::
 --interactive::
-- 
1.7.10.4

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Documentation/git-clean.txt: document that -f may need to be given twice

[Mikko Rapeli]

This is needed in build automation where the tree really needs to
be reset to known state.

Signed-off-by: Mikko Rapeli mikko.rap...@iki.fi
---
 Documentation/git-clean.txt |8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
index 94b6d19..872ab45 100644
--- a/Documentation/git-clean.txt
+++ b/Documentation/git-clean.txt
@@ -34,8 +34,12 @@ OPTIONS
 -f::
 --force::
If the Git configuration variable clean.requireForce is not set
-   to false, 'git clean' will refuse to run unless given -f, -n or
-   -i.
+   to false, 'git clean' will refuse to delete files or directories
+   unless given -f, -n or -i. Git will refuse to delete directories
+   with .git sub directory or file unless a second -f
+   is given. This affects also git submodules where the storage area of
+   of the removed submodule under .git/modules/ is not removed until
+   -f is give twice.
 
 -i::
 --interactive::
-- 
1.7.10.4

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Documentation/git-clean.txt: document that -f may need to be given twice

[Mikko Rapeli]

This is needed in build automation where the tree really needs to
be reset to known state.

Signed-off-by: Mikko Rapeli mikko.rap...@iki.fi
---
 Documentation/git-clean.txt |8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
index 94b6d19..641681f 100644
--- a/Documentation/git-clean.txt
+++ b/Documentation/git-clean.txt
@@ -34,8 +34,12 @@ OPTIONS
 -f::
 --force::
If the Git configuration variable clean.requireForce is not set
-   to false, 'git clean' will refuse to run unless given -f, -n or
-   -i.
+   to false, 'git clean' will refuse to delete files or directories
+   unless given -f, -n or -i. Git will refuse to delete directories
+   with .git sub directory or file unless a second -f
+   is given. This affects also git submodules where the storage area
+   of the removed submodule under .git/modules/ is not removed until
+   -f is given twice.
 
 -i::
 --interactive::
-- 
1.7.10.4

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Cygwin, git and x: directory

[Mikko Rapeli]

(please Cc: me in replies, not subscribed to the lists)

Hi Cygwin and git developers,

Does following scenario show signs of bugs in Cygwin and/or git?

# setup git repo
$ cd /tmp
$ mkdir foo  cd foo
$ git init

# create x: directory
$ mkdir x:
$ ls
x:

# create Windows X: drive, cygwin utils can work with both unix and dos style
# path names
$ mkdir c:/temp/bar
$ subst x: c:/temp/bar
$ touch x:/file.txt
$ ls x:/
file.txt

# clean git tree from non-tracked files
$ git clean -d -x -f
Removing x:/

# observe results, git did rm -rf on the X drive instead of the local
# directory named x:
$ ls
x:
$ file x\:
x:: directory
$ ls x:/
ls: cannot access x:/: No such file or directory
$ ls c:/temp/bar
ls: cannot access c:/temp/bar: No such file or directory
$ subst
X:\: = C:\temp\bar

In real life CMake created C: file in a build tree -- which is also a bug
but a separate one -- which resulted in obviously catastrophic results.

-Mikko
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git https transport and wrong password

[Mikko Rapeli]

Maybe my git installation was incomplete before when running from ~/bin since
I was not able to set break points to http_request() and some debug code
was not there until I ran git through bin-wrappers in the source tree.

I added some debug prints to http.c functions http_request() and
handle_curl_result(), and now I see this chain of events:

 http_request_reauth()
 http_request()
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 401 Authorization Required
* Ignoring the response-body
* Issue another request to this URL: '...'
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 401 Authorization Required
 handle_curl_result: res = 22, http_code = 401, user = ..., pass = (null)
Password for '...': (enter valid password)
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 200 OK

So, for some reason the first GET request is issued twice and first 401
is ignored. I'll try to debug run_active_slot() next...

-Mikko
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git https transport and wrong password

[Mikko Rapeli]

On Wed, Apr 03, 2013 at 10:12:12AM -0400, Jeff King wrote:
 I would expect without the username in the URL for it to make only two
 requests: one to get the first 401, then git collects the credentials,
 then a follow-up with the credentials. But instead we get:
 
   $ GIT_CURL_VERBOSE=1 git ls-remote https://github.com/requires/auth \
   21 /dev/null | egrep '^|^ HTTP|^Authorization|requested URL'
GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
   * The requested URL returned error: 401 Authorization Required
   Username for 'https://github.com': foo
   Password for 'https://f...@github.com': 
GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
HTTP/1.1 401 Authorization Required
GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
   Authorization: Basic Zm9vOmJhcg==
HTTP/1.1 401 Authorization Required
   * The requested URL returned error: 401
 
 So we get a 401, as expected, git prompts for the credentials and feeds
 them directly to curl, but then we still get _two_ requests: we trigger
 another 401, and only then does curl provide the authorization header to
 the server.
 
 I'm not sure if that extra auth is intended or not.

git uses CURLAUTH_ANY which means: first try without authentication
(CURLAUTH_NONE), if that fails it will try (I guess) CURLAUTH_BASIC|DIGEST|
GSS|NTML and so on, and only then it will fail with the 401.

It seems that skipping CURLAUTH_NONE try is not possible even if it's
not a good idea when a username and possibly password is available.
Changing CURLAUTH_ANY to skip CURLAUTH_NONE could also break other
users.

Since netrc support really needs this one try from git to curl before
password prompt I guess in our case using HTTPS with git is simply not
feasible. Changing the corporate single sign-on policies is also hard
so I will now try to get SSH transport running on the server.

Account locking will still be quite easy but hopefully only after
multiple false passwords to the SSH promp.

-Mikko
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

git https transport and wrong password

[Mikko Rapeli]

Hi,

I have a problem with git (1.7.9 and 1.8.2.357.gcc3e4eb) and https transport
to gerrit server (2.5.1-3-g719dfc7). I'm producing the problem on Cygwin but my
colleagues have same issue on Linux as well.

Gerrit server is matching corporate policies with single sign on, so after
three failed login attempts the account gets locked until a password reset.

Git amplifies this problem by asking for users password only once, and if
user made a typo git is still re-using the wrong password enough times to
get an account immediately locked.

I have client side logs with GIT_CURL_VERBOSE=1 but from intranet so can't
publish them directly. Here's roughly what the log shows:

---

$ GIT_CURL_VERBOSE=1 git fetch
...
 GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
...
 HTTP/1.1 401 Authorization Required
...

-- I guess git prompts for password here. --

* Issue another request to this URL: 
'https://..info/refs?service=git-upload-pack'
...
* Re-using existing connection! ...
...
* Server auth using Basic with user '...'
 GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ...
...
 HTTP/1.1 401 Authorization Required
 Date: ...
* Authentication problem. Ignoring this.
...
* The requested URL returned error: 401
* Closing connection 0
...
* About to connect() to ...
...
* Connected to ...
...
* STATE: PROTOCONNECT = DO handle...
* Server auth using Basic with user '...'
 GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ...
...
* STATE: DO = DO_DONE handle...
* STATE: DO_DONE = WAITPERFORM handle...
* STATE: WAITPERFORM = PERFORM handle...
...
 HTTP/1.1 302 Found
...
 Location: ...funnylongurl
...
* Ignoring the response-body
* Connection #1 to host ... left intact
* Issue another request to this URL: '...funnylongurl'
...
* Server auth using Basic with user '...'
 GET ...funnylongurl
Authorization: Basic ...
...
* The requested URL returned error: 500 Internal Server Error
* Closing connection 1
...
* About to connect()...
...
* Server auth using Basic with user '...'
 GET /gerrit/.../info/refs HTTP/1.1
Authorization: Basic ...
...
 HTTP/1.1 302 Found
 Date...
 Set-Cookie...
 Cache-Control: no-store
 Location: ...funnylongurl
...
* Re-using existing connection! (#2)...
 GET ...funnylongurl
...
* The requested URL returned error: 500 Internal Server Error
* Closing connection 2
...
error: The requested URL returned error: 500 Internal Server Error while 
accessing ...
fatal: HTTP request failed

---

Any idea what could be wrong here? Is git client really retrying with the
bad password?

Regards,

-Mikko
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git https transport and wrong password

[Mikko Rapeli]

On Tue, Apr 02, 2013 at 06:54:40PM +0300, Mikko Rapeli wrote:
 I have client side logs with GIT_CURL_VERBOSE=1 but from intranet so can't
 publish them directly. Here's roughly what the log shows:

Maybe this is simpler summary:

$ grep HTTP\/1.1 log.txt
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 401 Authorization required

password prompt here, and ctrl-c does not work in Cygwin, sigh.

 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 401 Authorization required
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 302 Found

account locked I presume

 GET longredirecturl
 GET ...info/refs
 HTTP/1.1 302 Found
 GET longredirecturl

I was not able reproduce this issue using curl directly to get the info/refs
page.

-Mikko
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git https transport and wrong password

[Mikko Rapeli]

On Tue, Apr 02, 2013 at 03:28:45PM -0400, Jeff King wrote:
 We get redirected somewhere where we provide the (presumably wrong)
 credential again. I do not think that is git's fault; the server asked
 us to make the extra request. Is that part of the lockout procedure? If
 it is not, it seems odd that the server would issue a redirect for a
 bogus auth (shouldn't it just keep giving us 401?).

I think it is supposed to be a catch all failure mode without any
authentication but is just wrong/buggy. I'll try to debug these by
issuing curl commands step by step.

 I do not know what is going on with the redirection there, but I have a
 hunch on the extra auth round-trip.  What does your remote URL look
 like? Does it have your username (e.g., https://user@host/project.git)?

Yes, that's the giturl format I have.

 I have noticed that if curl sees such a URL, it attempts to do a
 password-less authentication itself, before even handing control back to
 git. So my above sequence would become:
 
   1. git feeds URL to curl, who makes request
   2. we get a 401
   3. curl says Oh, I have a username; let me try that and re-requests
   4. we get another 401, because we need a password
   5. curl says that didn't work and hands control back to git
   6. git requests a password from the user and gives it to curl
   7. curl retries with the password, but it's wrong, so that results in
  a 401, too
 
 At the end of it, we've now made _two_ failed requests for user X,
 rather than one. I don't know if there's a way to tell curl not to try
 the extra user-only round-trip. But you can strip the username out of
 your URL to avoid it.

It did seem like there was just one GET and 401 return before password
was promptet. I'll tripple check that.

Played around with command line curl a bit and at least it did the right
thing with a URL without username -- failed with 401 after single try --
and with URL without username but username provided -u 'username' which
succeeded or failed on single try based on password.

Don't know anything about curl but maybe git could parse the url for a
username and prompt for the password before the first 401 failure roundtrip
that's now in place. I guess most of this logic is in http.c.

-Mikko
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git https transport and wrong password

[Mikko Rapeli]

On Tue, Apr 02, 2013 at 04:05:51PM -0400, Jeff King wrote:
 On Tue, Apr 02, 2013 at 10:47:51PM +0300, Mikko Rapeli wrote:
 
  Don't know anything about curl but maybe git could parse the url for a
  username and prompt for the password before the first 401 failure roundtrip
  that's now in place. I guess most of this logic is in http.c.
 
 We used to do that but stopped, as curl might also be able to retrieve
 the password from .netrc; the extra prompt was an annoyance to users
 in this situation.

Ok, I think I've seen this before and ended up storing passwords in .netrc.

 Now that we have the credential subsystem, I would recommend dropping
 usernames from all git-over-http URLs, and either:
 
   1. Using a credential helper that supports secure long-term storage
  (osxkeychain, wincred, etc).
 
   2. Specifying the username to the credential subsystem explicitly, by
  putting something like:
 
[credential https://yourhost/;]
   username = yourusername
 
  in your git config.
 
 Obviously (1) is nicer, but you may have corporate policies against
 storing credentials. Or you may have a complicated single sign-on
 procedure, where the password changes. In that case, I would still say
 it is worth writing a custom helper script that can feed the temporary
 credential to git.

Thanks, I'll have a look at these helpers. Policies we may have but in
practice I think many just store plaintext passwords in giturls, which
is obviously the worst case, but it works against accidental typos in
the password prompt (though blows up when the mandatory password change
comes along).

-Mikko
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html