Re: git ls-files -o seems to ignore .gitignore

[Matthieu Moy]

Charles Bailey  writes:

> On Mon, Oct 27, 2014 at 07:16:49AM +0100, Richard PALO wrote:
>> Hash: SHA1
>> 
>> I'm having an issue in that 'git ls-files -o' seems to ignore
>> [parts of] .gitignore whereas other commands, such as 'git status'
>> seem fine.
>
> This is, as far as I am aware, by design.

I would not call that "by design", but indeed "git ls" was written this
way, and it's a plumbing command for which we do not want
backward-incompatible changes. So, --exclude-standard is available, but
not activated by default.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Editing git changelog automatically

[Matthieu Moy]

Dennis Kaarsemaker  writes:

> On zo, 2014-10-26 at 22:27 -0700, Cong Wang wrote:
>> 
>> My question is how to edit dozens of git commit changelogs
>> automatically?
>
> You can use git filter-branch in --msg-filter mode.

Note that in any case, you'll rewrite the history hence change commit
identifiers. If the history is already published, it's probably better
to live with the typo than to try to fix it now.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git ls-files -o seems to ignore .gitignore

[richard . palo]

Well, ok.  Not very intuitive, so I added an alias in .gitconfig... 
'ls = ls-files --exclude-standard' and that seems to work now alright for me.

Thanks

- Mail original -
> De: "Matthieu Moy" 
> À: "Charles Bailey" 
> Cc: "Richard PALO" , git@vger.kernel.org
> Envoyé: Lundi 27 Octobre 2014 09:31:56
> Objet: Re: git ls-files -o seems to ignore .gitignore
> 
> Charles Bailey  writes:
> 
> > On Mon, Oct 27, 2014 at 07:16:49AM +0100, Richard PALO wrote:
> >> Hash: SHA1
> >> 
> >> I'm having an issue in that 'git ls-files -o' seems to ignore
> >> [parts of] .gitignore whereas other commands, such as 'git status'
> >> seem fine.
> >
> > This is, as far as I am aware, by design.
> 
> I would not call that "by design", but indeed "git ls" was written
> this
> way, and it's a plumbing command for which we do not want
> backward-incompatible changes. So, --exclude-standard is available,
> but
> not activated by default.
> 
> --
> Matthieu Moy
> http://www-verimag.imag.fr/~moy/
> 
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 00/19] Untracked cache to speed up "git status"

[Nguyễn Thái Ngọc Duy]

The last post was five months ago [1]. It's probably time for a resend
in case I'm hit by a bus. Numbers are in 17/19, saving about 40% time
on "git status". Details are in 02/19 and 06/19. Still on the table:

 - index-helper series [2] probably helps save about 10-15% in total
 - watchman support to lower the numbers even more, but not for Windows

I still need to see how watchman can be made on top of this.

[1] http://thread.gmane.org/gmane.comp.version-control.git/248306
[2] http://thread.gmane.org/gmane.comp.version-control.git/254314/focus=254318
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 01/19] dir.c: optionally compute sha-1 of a .gitignore file

[Nguyễn Thái Ngọc Duy]

This is not used anywhere yet. But the goal is to compare quickly if a
.gitignore file has changed when we have the SHA-1 of both old (cached
somewhere) and new (from index or a tree) versions.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 50 +++---
 dir.h |  5 +
 2 files changed, 48 insertions(+), 7 deletions(-)

diff --git a/dir.c b/dir.c
index bd274a7..33a35c1 100644
--- a/dir.c
+++ b/dir.c
@@ -466,7 +466,8 @@ void add_exclude(const char *string, const char *base,
x->el = el;
 }
 
-static void *read_skip_worktree_file_from_index(const char *path, size_t *size)
+static void *read_skip_worktree_file_from_index(const char *path, size_t *size,
+   struct sha1_stat *ss)
 {
int pos, len;
unsigned long sz;
@@ -485,6 +486,10 @@ static void *read_skip_worktree_file_from_index(const char 
*path, size_t *size)
return NULL;
}
*size = xsize_t(sz);
+   if (ss) {
+   memset(&ss->stat, 0, sizeof(ss->stat));
+   hashcpy(ss->sha1, active_cache[pos]->sha1);
+   }
return data;
 }
 
@@ -529,11 +534,18 @@ static void trim_trailing_spaces(char *buf)
*last_space = '\0';
 }
 
-int add_excludes_from_file_to_list(const char *fname,
-  const char *base,
-  int baselen,
-  struct exclude_list *el,
-  int check_index)
+/*
+ * Given a file with name "fname", read it (either from disk, or from
+ * the index if "check_index" is non-zero), parse it and store the
+ * exclude rules in "el".
+ *
+ * If "ss" is not NULL, compute SHA-1 of the exclude file and fill
+ * stat data from disk (only valid if add_excludes returns zero). If
+ * ss_valid is non-zero, "ss" must contain good value as input.
+ */
+static int add_excludes(const char *fname, const char *base, int baselen,
+   struct exclude_list *el, int check_index,
+   struct sha1_stat *ss, int ss_valid)
 {
struct stat st;
int fd, i, lineno = 1;
@@ -547,7 +559,7 @@ int add_excludes_from_file_to_list(const char *fname,
if (0 <= fd)
close(fd);
if (!check_index ||
-   (buf = read_skip_worktree_file_from_index(fname, &size)) == 
NULL)
+   (buf = read_skip_worktree_file_from_index(fname, &size, 
ss)) == NULL)
return -1;
if (size == 0) {
free(buf);
@@ -560,6 +572,10 @@ int add_excludes_from_file_to_list(const char *fname,
} else {
size = xsize_t(st.st_size);
if (size == 0) {
+   if (ss) {
+   fill_stat_data(&ss->stat, &st);
+   hashcpy(ss->sha1, EMPTY_BLOB_SHA1_BIN);
+   }
close(fd);
return 0;
}
@@ -571,6 +587,19 @@ int add_excludes_from_file_to_list(const char *fname,
}
buf[size++] = '\n';
close(fd);
+   if (ss) {
+   int pos;
+   if (ss_valid && !match_stat_data(&ss->stat, &st))
+   ; /* no content change, ss->sha1 still good */
+   else if (check_index &&
+(pos = cache_name_pos(fname, strlen(fname))) 
>= 0 &&
+!ce_stage(active_cache[pos]) &&
+ce_uptodate(active_cache[pos]))
+   hashcpy(ss->sha1, active_cache[pos]->sha1);
+   else
+   hash_sha1_file(buf, size, "blob", ss->sha1);
+   fill_stat_data(&ss->stat, &st);
+   }
}
 
el->filebuf = buf;
@@ -589,6 +618,13 @@ int add_excludes_from_file_to_list(const char *fname,
return 0;
 }
 
+int add_excludes_from_file_to_list(const char *fname, const char *base,
+  int baselen, struct exclude_list *el,
+  int check_index)
+{
+   return add_excludes(fname, base, baselen, el, check_index, NULL, 0);
+}
+
 struct exclude_list *add_exclude_list(struct dir_struct *dir,
  int group_type, const char *src)
 {
diff --git a/dir.h b/dir.h
index 6c45e9d..032d197 100644
--- a/dir.h
+++ b/dir.h
@@ -73,6 +73,11 @@ struct exclude_list_group {
struct exclude_list *el;
 };
 
+struct sha1_stat {
+   struct stat_data stat;
+   unsigned char sha1[20];
+};
+
 struct dir_struct {
int nr, alloc;
int ignored_nr, ignored_alloc;
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message 

[PATCH 04/19] untracked cache: invalidate dirs recursively if .gitignore changes

[Nguyễn Thái Ngọc Duy]

It's easy to see that if an existing .gitignore changes, its SHA-1
would be different and invalidate_gitignore() is called.

If .gitignore is removed, add_excludes() will treat it like an empty
.gitignore, which again should invalidate the cached directory data.

if .gitignore is added, lookup_untracked() already fills initial
.gitignore SHA-1 as "empty file", so again invalidate_gitignore() is
called.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 17 -
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/dir.c b/dir.c
index 5161f74..a0a0fa8 100644
--- a/dir.c
+++ b/dir.c
@@ -1005,7 +1005,22 @@ static void prep_exclude(struct dir_struct *dir, const 
char *base, int baselen)
add_excludes(el->src, el->src, stk->baselen, el, 1,
 untracked ? &ss : NULL, 0);
}
-   if (untracked) {
+   /*
+* NEEDSWORK: when untracked cache is enabled, prep_exclude()
+* will first be called in valid_cached_dir() then maybe many
+* times more in last_exclude_matching(). When the cache is
+* used, last_exclude_matching() will not be called and
+* reading .gitignore content will be a waste.
+*
+* So when it's called by valid_cached_dir() and we can get
+* .gitignore SHA-1 from the index (i.e. .gitignore is not
+* modified on work tree), we could delay reading the
+* .gitignore content until we absolutely need it in
+* last_exclude_matching(). Be careful about ignore rule
+* order, though, if you do that.
+*/
+   if (untracked && hashcmp(ss.sha1, untracked->exclude_sha1)) {
+   invalidate_gitignore(dir->untracked, untracked);
hashcpy(untracked->exclude_sha1, ss.sha1);
}
dir->exclude_stack = stk;
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 03/19] untracked cache: initial untracked cache validation

[Nguyễn Thái Ngọc Duy]

Make sure the starting conditions and all global exclude files are
good to go. If not, either disable untracked cache completely, or wipe
out the cache and start fresh.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 113 --
 dir.h |   4 +++
 2 files changed, 114 insertions(+), 3 deletions(-)

diff --git a/dir.c b/dir.c
index 25d8c5d..5161f74 100644
--- a/dir.c
+++ b/dir.c
@@ -581,6 +581,22 @@ static struct untracked_cache_dir *lookup_untracked(struct 
untracked_cache *uc,
return d;
 }
 
+static void do_invalidate_gitignore(struct untracked_cache_dir *dir)
+{
+   int i;
+   dir->valid = 0;
+   dir->untracked_nr = 0;
+   for (i = 0; i < dir->dirs_nr; i++)
+   do_invalidate_gitignore(dir->dirs[i]);
+}
+
+static void invalidate_gitignore(struct untracked_cache *uc,
+struct untracked_cache_dir *dir)
+{
+   uc->gitignore_invalidated++;
+   do_invalidate_gitignore(dir);
+}
+
 /*
  * Given a file with name "fname", read it (either from disk, or from
  * the index if "check_index" is non-zero), parse it and store the
@@ -693,6 +709,13 @@ static void add_excludes_from_file_1(struct dir_struct 
*dir, const char *fname,
 struct sha1_stat *ss, int ss_valid)
 {
struct exclude_list *el;
+   /*
+* catch setup_standard_excludes() that's called before
+* dir->untracked is assigned. That function behaves
+* differently when dir->untracked is non-NULL.
+*/
+   if (!dir->untracked)
+   dir->unmanaged_exclude_files++;
el = add_exclude_list(dir, EXC_FILE, fname);
if (add_excludes(fname, "", 0, el, 0, ss, ss_valid) < 0)
die("cannot use %s as an exclude file", fname);
@@ -700,6 +723,7 @@ static void add_excludes_from_file_1(struct dir_struct 
*dir, const char *fname,
 
 void add_excludes_from_file(struct dir_struct *dir, const char *fname)
 {
+   dir->unmanaged_exclude_files++; /* see validate_untracked_cache() */
add_excludes_from_file_1(dir, fname, NULL, 0);
 }
 
@@ -1567,9 +1591,87 @@ static int treat_leading_path(struct dir_struct *dir,
return rc;
 }
 
+static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct 
*dir,
+ int base_len,
+ const struct pathspec 
*pathspec)
+{
+   struct untracked_cache_dir *root;
+
+   if (!dir->untracked)
+   return NULL;
+
+   /*
+* We only support $GIT_DIR/info/exclude and core.excludesfile
+* as the global ignore rule files. Any other additions
+* (e.g. from command line) invalidate the cache. This
+* condition also catches running setup_standard_excludes()
+* before setting dir->untracked!
+*/
+   if (dir->unmanaged_exclude_files)
+   return NULL;
+
+   /*
+* Optimize for the main use case only: whole-tree git
+* status. More work involved in treat_leading_path() if we
+* use cache on just a subset of the worktree. pathspec
+* support could make the matter even worse.
+*/
+   if (base_len || (pathspec && pathspec->nr))
+   return NULL;
+
+   /* Different set of flags may produce different results */
+   if (dir->flags != dir->untracked->dir_flags ||
+   /*
+* See treat_directory(), case index_nonexistent. Without
+* this flag, we may need to also cache .git file content
+* for the resolve_gitlink_ref() call, which we don't.
+*/
+   !(dir->flags & DIR_SHOW_OTHER_DIRECTORIES) ||
+   /* We don't support collecting ignore files */
+   (dir->flags & (DIR_SHOW_IGNORED | DIR_SHOW_IGNORED_TOO |
+  DIR_COLLECT_IGNORED)))
+   return NULL;
+
+   /*
+* If we use .gitignore in the cache and now you change it to
+* .gitexclude, everything will go wrong.
+*/
+   if (dir->exclude_per_dir != dir->untracked->exclude_per_dir &&
+   strcmp(dir->exclude_per_dir, dir->untracked->exclude_per_dir))
+   return NULL;
+
+   /*
+* EXC_CMDL is not considered in the cache. If people set it,
+* skip the cache.
+*/
+   if (dir->exclude_list_group[EXC_CMDL].nr)
+   return NULL;
+
+   if (!dir->untracked->root) {
+   const int len = sizeof(*dir->untracked->root);
+   dir->untracked->root = xmalloc(len);
+   memset(dir->untracked->root, 0, len);
+   }
+
+   /* Validate $GIT_DIR/info/exclude and core.excludesfile */
+   root = dir->untracked->root;
+   if (hashcmp(dir->ss_info_exclude.sha1,
+   dir->untracked->ss_info_exclude.sha1)) {
+   invalidate_gitignore(dir->untracked, roo

[PATCH 02/19] untracked cache: record .gitignore information and dir hierarchy

[Nguyễn Thái Ngọc Duy]

The idea is if we can capture all input and (non-rescursive) output of
read_directory_recursive(), and can verify later that all the input is
the same, then the second r_d_r() should produce the same output as in
the first run.

The requirement for this to work is stat info of a directory MUST
change if an entry is added to or removed from that directory (and
should not change often otherwise). If your OS and filesytem do not
meet this requirement, untracked cache is not for you. Most file
systems on *nix should be fine. On Windows, NTFS is fine while FAT may
be not [1] even though FAT on Linux seems to be fine.

The list of input of r_d_r() is in the big comment block in dir.h. In
short, the output of a directory (not counting subdirs) mainly depends
on stat info of the directory in question, all .gitignore leading to
it and the check_only flag when r_d_r() is called recursively. This
patch records all this info (and the output) as r_d_r() runs.

Two hash_sha1_file() are required for $GIT_DIR/info/exclude and
core.excludesfile unless their stat data matches. hash_sha1_file() is
only needed when .gitignore files in the worktree are modified,
otherwise their SHA-1 in index is used (see the previous patch).

We could store stat data for .gitignore files so we don't have to
rehash them if their content is different from index, but I think
.gitignore files are rarely modified, so not worth extra cache data
(and hashing penalty read-cache.c:verify_hdr(), as we will be storing
this as an index extension).

The implication is, if you change .gitignore, you better add it to the
index soon or you lose all the benefit of untracked cache because a
modified .gitignore invalidates all subdirs recursively. This is
especially bad for .gitignore at root.

This cached output is about untracked files only, not ignored files
because the number of tracked files is usually small, so small cache
overhead, while the number of ignored files could go really high
(e.g. *.o files mixing with source code).

[1] "Description of NTFS date and time stamps for files and folders"
http://support.microsoft.com/kb/299648

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 150 --
 dir.h |  60 +++
 2 files changed, 189 insertions(+), 21 deletions(-)

diff --git a/dir.c b/dir.c
index 33a35c1..25d8c5d 100644
--- a/dir.c
+++ b/dir.c
@@ -32,7 +32,7 @@ enum path_treatment {
 };
 
 static enum path_treatment read_directory_recursive(struct dir_struct *dir,
-   const char *path, int len,
+   const char *path, int len, struct untracked_cache_dir *untracked,
int check_only, const struct path_simplify *simplify);
 static int get_dtype(struct dirent *de, const char *path, int len);
 
@@ -535,6 +535,53 @@ static void trim_trailing_spaces(char *buf)
 }
 
 /*
+ * Given a subdirectory name and "dir" of the current directory,
+ * search the subdir in "dir" and return it, or create a new one if it
+ * does not exist in "dir".
+ *
+ * If "name" has the trailing slash, it'll be excluded in the search.
+ */
+static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,
+   struct untracked_cache_dir 
*dir,
+   const char *name, int len)
+{
+   int first, last;
+   struct untracked_cache_dir *d;
+   if (!dir)
+   return NULL;
+   if (len && name[len - 1] == '/')
+   len--;
+   first = 0;
+   last = dir->dirs_nr;
+   while (last > first) {
+   int cmp, next = (last + first) >> 1;
+   d = dir->dirs[next];
+   cmp = strncmp(name, d->name, len);
+   if (!cmp && strlen(d->name) > len)
+   cmp = -1;
+   if (!cmp)
+   return d;
+   if (cmp < 0) {
+   last = next;
+   continue;
+   }
+   first = next+1;
+   }
+
+   uc->dir_created++;
+   d = xmalloc(sizeof(*d) + len);
+   memset(d, 0, sizeof(*d) + len);
+   memcpy(d->name, name, len);
+
+   ALLOC_GROW(dir->dirs, dir->dirs_nr + 1, dir->dirs_alloc);
+   memmove(dir->dirs + first + 1, dir->dirs + first,
+   (dir->dirs_nr - first) * sizeof(*dir->dirs));
+   dir->dirs_nr++;
+   dir->dirs[first] = d;
+   return d;
+}
+
+/*
  * Given a file with name "fname", read it (either from disk, or from
  * the index if "check_index" is non-zero), parse it and store the
  * exclude rules in "el".
@@ -642,14 +689,20 @@ struct exclude_list *add_exclude_list(struct dir_struct 
*dir,
 /*
  * Used to set up core.excludesfile and .git/info/exclude lists.
  */
-void add_excludes_from_file(struct dir_struct *dir, const char *fname)
+static void add_excludes_from_file_1(struct dir_struct *dir, const char *fname,
+struct 

[PATCH 05/19] untracked cache: make a wrapper around {open,read,close}dir()

[Nguyễn Thái Ngọc Duy]

This allows us to feed different info to read_directory_recursive()
based on untracked cache in the next patch.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 55 +++
 1 file changed, 47 insertions(+), 8 deletions(-)

diff --git a/dir.c b/dir.c
index a0a0fa8..2793e57 100644
--- a/dir.c
+++ b/dir.c
@@ -31,6 +31,15 @@ enum path_treatment {
path_untracked
 };
 
+/*
+ * Support data structure for our opendir/readdir/closedir wrappers
+ */
+struct cached_dir {
+   DIR *fdir;
+   struct untracked_cache_dir *untracked;
+   struct dirent *de;
+};
+
 static enum path_treatment read_directory_recursive(struct dir_struct *dir,
const char *path, int len, struct untracked_cache_dir *untracked,
int check_only, const struct path_simplify *simplify);
@@ -1411,12 +1420,13 @@ static enum path_treatment treat_one_path(struct 
dir_struct *dir,
 
 static enum path_treatment treat_path(struct dir_struct *dir,
  struct untracked_cache_dir *untracked,
- struct dirent *de,
+ struct cached_dir *cdir,
  struct strbuf *path,
  int baselen,
  const struct path_simplify *simplify)
 {
int dtype;
+   struct dirent *de = cdir->de;
 
if (is_dot_or_dotdot(de->d_name) || !strcmp(de->d_name, ".git"))
return path_none;
@@ -1438,6 +1448,37 @@ static void add_untracked(struct untracked_cache_dir 
*dir, const char *name)
dir->untracked[dir->untracked_nr++] = xstrdup(name);
 }
 
+static int open_cached_dir(struct cached_dir *cdir,
+  struct dir_struct *dir,
+  struct untracked_cache_dir *untracked,
+  struct strbuf *path,
+  int check_only)
+{
+   memset(cdir, 0, sizeof(*cdir));
+   cdir->untracked = untracked;
+   cdir->fdir = opendir(path->len ? path->buf : ".");
+   if (!cdir->fdir)
+   return -1;
+   return 0;
+}
+
+int read_cached_dir(struct cached_dir *cdir)
+{
+   if (cdir->fdir) {
+   cdir->de = readdir(cdir->fdir);
+   if (!cdir->de)
+   return -1;
+   return 0;
+   }
+   return -1;
+}
+
+static void close_cached_dir(struct cached_dir *cdir)
+{
+   if (cdir->fdir)
+   closedir(cdir->fdir);
+}
+
 /*
  * Read a directory tree. We currently ignore anything but
  * directories, regular files and symlinks. That's because git
@@ -1454,23 +1495,21 @@ static enum path_treatment 
read_directory_recursive(struct dir_struct *dir,
struct untracked_cache_dir *untracked, int 
check_only,
const struct path_simplify *simplify)
 {
-   DIR *fdir;
+   struct cached_dir cdir;
enum path_treatment state, subdir_state, dir_state = path_none;
-   struct dirent *de;
struct strbuf path = STRBUF_INIT;
 
strbuf_add(&path, base, baselen);
 
-   fdir = opendir(path.len ? path.buf : ".");
-   if (!fdir)
+   if (open_cached_dir(&cdir, dir, untracked, &path, check_only))
goto out;
 
if (untracked)
untracked->check_only = !!check_only;
 
-   while ((de = readdir(fdir)) != NULL) {
+   while (!read_cached_dir(&cdir)) {
/* check how the file or directory should be treated */
-   state = treat_path(dir, untracked, de, &path, baselen, 
simplify);
+   state = treat_path(dir, untracked, &cdir, &path, baselen, 
simplify);
 
if (state > dir_state)
dir_state = state;
@@ -1523,7 +1562,7 @@ static enum path_treatment 
read_directory_recursive(struct dir_struct *dir,
break;
}
}
-   closedir(fdir);
+   close_cached_dir(&cdir);
  out:
strbuf_release(&path);
 
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 07/19] untracked cache: mark what dirs should be recursed/saved

[Nguyễn Thái Ngọc Duy]

If we redo this thing in a functional style, we would have one struct
untracked_dir as input tree and another as output. The input is used
for verification. The output is a brand new tree, reflecting current
worktree.

But that means recreate a lot of dir nodes even if a lot could be
shared between input and output trees in good cases. So we go with the
messy but efficient way, combining both input and output trees into
one. We need a way to know which node in this combined tree belongs to
the output. This is the purpose of this "recurse" flag.

"valid" bit can't be used for this because it's about data of the node
except the subdirs. When we invalidate a directory, we want to keep
cached data of the subdirs intact even though we don't really know
what subdir still exists (yet). Then we check worktree to see what
actual subdir remains on disk. Those will have 'recurse' bit set
again. If cached data for those are still valid, we may be able to
avoid computing exclude files for them. Those subdirs that are deleted
will have 'recurse' remained clear and their 'valid' bits do not
matter.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 14 +-
 dir.h |  3 ++-
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/dir.c b/dir.c
index 55780a7..d9675c8 100644
--- a/dir.c
+++ b/dir.c
@@ -614,9 +614,12 @@ static void invalidate_gitignore(struct untracked_cache 
*uc,
 static void invalidate_directory(struct untracked_cache *uc,
 struct untracked_cache_dir *dir)
 {
+   int i;
uc->dir_invalidated++;
dir->valid = 0;
dir->untracked_nr = 0;
+   for (i = 0; i < dir->dirs_nr; i++)
+   dir->dirs[i]->recurse = 0;
 }
 
 /*
@@ -1573,6 +1576,10 @@ int read_cached_dir(struct cached_dir *cdir)
}
while (cdir->nr_dirs < cdir->untracked->dirs_nr) {
struct untracked_cache_dir *d = 
cdir->untracked->dirs[cdir->nr_dirs];
+   if (!d->recurse) {
+   cdir->nr_dirs++;
+   continue;
+   }
cdir->ucd = d;
cdir->nr_dirs++;
return 0;
@@ -1594,8 +1601,10 @@ static void close_cached_dir(struct cached_dir *cdir)
 * We have gone through this directory and found no untracked
 * entries. Mark it valid.
 */
-   if (cdir->untracked && !cdir->untracked->valid)
+   if (cdir->untracked) {
cdir->untracked->valid = 1;
+   cdir->untracked->recurse = 1;
+   }
 }
 
 /*
@@ -1838,6 +1847,9 @@ static struct untracked_cache_dir 
*validate_untracked_cache(struct dir_struct *d
invalidate_gitignore(dir->untracked, root);
dir->untracked->ss_excludes_file = dir->ss_excludes_file;
}
+
+   /* Make sure this directory is not dropped out at saving phase */
+   root->recurse = 1;
return root;
 }
 
diff --git a/dir.h b/dir.h
index 1fefd4e..c302dcb 100644
--- a/dir.h
+++ b/dir.h
@@ -113,8 +113,9 @@ struct untracked_cache_dir {
/* null SHA-1 means this directory does not have .gitignore */
unsigned char exclude_sha1[20];
struct stat_data stat_data;
+   unsigned int recurse : 1;
unsigned int check_only : 1;
-   /* all data in this struct are good */
+   /* all data except 'dirs' in this struct are good */
unsigned int valid : 1;
unsigned int untracked_nr : 29;
unsigned int untracked_alloc, dirs_nr, dirs_alloc;
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 06/19] untracked cache: record/validate dir mtime and reuse cached output

[Nguyễn Thái Ngọc Duy]

The main readdir loop in read_directory_recursive() is replaced with a
new one that checks if cached results of a directory is still valid.

If a file is added or removed from the index, the containing directory
is invalidated (but not its subdirs). If directory's mtime is changed,
the same happens. If a .gitignore is updated, the containing directory
and all subdirs are invalidated recursively. If dir_struct#flags or
other conditions change, the cache is ignored.

If a directory is invalidated, we opendir/readdir/closedir and run the
exclude machinery on that directory listing as usual. If untracked
cache is also enabled, we'll update the cache along the way. If a
directory is validated, we simply pull the untracked listing out from
the cache. The cache also records the list of direct subdirs that we
have to recurse in. Fully excluded directories are seen as "untracked
files".

In the best case when no dirs are invalidated, read_directory()
becomes a series of

  stat(dir), open(.gitignore), fstat(), read(), close() and optionally
  hash_sha1_file()

For comparison, standard read_directory() is a sequence of

  opendir(), readdir(), open(.gitignore), fstat(), read(), close(), the
  expensive last_exclude_matching() and closedir().

We already try not to open(.gitignore) if we know it does not exist,
so open/fstat/read/close sequence does not apply to every
directory. The sequence could be reduced further, as noted in
prep_exclude() in another patch. So in theory, the entire best-case
read_directory sequence could be reduced to a series of stat() and
nothing else.

This is not a silver bullet approach. When you compile a C file, for
example, the old .o file is removed and a new one with the same name
created, effectively invalidating the containing directory's cache
(but not its subdirectories). If your build process touches every
directory, this cache adds extra overhead for nothing, so it's a good
idea to separate generated files from tracked files.. Editors may use
the same strategy for saving files. And of course you're out of luck
running your repo on an unsupported filesytem and/or operating system.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 123 --
 dir.h |   2 ++
 2 files changed, 123 insertions(+), 2 deletions(-)

diff --git a/dir.c b/dir.c
index 2793e57..55780a7 100644
--- a/dir.c
+++ b/dir.c
@@ -37,7 +37,12 @@ enum path_treatment {
 struct cached_dir {
DIR *fdir;
struct untracked_cache_dir *untracked;
+   int nr_files;
+   int nr_dirs;
+
struct dirent *de;
+   const char *file;
+   struct untracked_cache_dir *ucd;
 };
 
 static enum path_treatment read_directory_recursive(struct dir_struct *dir,
@@ -606,6 +611,14 @@ static void invalidate_gitignore(struct untracked_cache 
*uc,
do_invalidate_gitignore(dir);
 }
 
+static void invalidate_directory(struct untracked_cache *uc,
+struct untracked_cache_dir *dir)
+{
+   uc->dir_invalidated++;
+   dir->valid = 0;
+   dir->untracked_nr = 0;
+}
+
 /*
  * Given a file with name "fname", read it (either from disk, or from
  * the index if "check_index" is non-zero), parse it and store the
@@ -1418,6 +1431,41 @@ static enum path_treatment treat_one_path(struct 
dir_struct *dir,
}
 }
 
+static enum path_treatment treat_path_fast(struct dir_struct *dir,
+  struct untracked_cache_dir 
*untracked,
+  struct cached_dir *cdir,
+  struct strbuf *path,
+  int baselen,
+  const struct path_simplify *simplify)
+{
+   if (!cdir->ucd) {
+   strbuf_setlen(path, baselen);
+   strbuf_addstr(path, cdir->file);
+   return path_untracked;
+   }
+   strbuf_setlen(path, baselen);
+   strbuf_addstr(path, cdir->ucd->name);
+   /* treat_one_path() does this before it calls treat_directory() */
+   if (path->buf[path->len - 1] != '/')
+   strbuf_addch(path, '/');
+   if (cdir->ucd->check_only)
+   /*
+* check_only is set as a result of treat_directory() getting
+* to its bottom. Verify again the same set of directories
+* with check_only set.
+*/
+   return read_directory_recursive(dir, path->buf, path->len,
+
+   cdir->ucd, 1, simplify);
+   /*
+* We get path_recurse in the first run when
+* directory_exists_in_index() returns index_nonexistent. We
+* are sure that new changes in the index does not impact the
+* outcome. Return now.
+*/
+   return path_recurse;
+}
+
 static enum path_treatment treat_path(struct dir_struct *dir,
  s

[PATCH 08/19] untracked cache: don't open non-existent .gitignore

[Nguyễn Thái Ngọc Duy]

This cuts down a signficant number of open(.gitignore) because most
directories usually don't have .gitignore files.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 26 +-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/dir.c b/dir.c
index d9675c8..d4f9830 100644
--- a/dir.c
+++ b/dir.c
@@ -1014,7 +1014,21 @@ static void prep_exclude(struct dir_struct *dir, const 
char *base, int baselen)
 
/* Try to read per-directory file */
hashclr(ss.sha1);
-   if (dir->exclude_per_dir) {
+   if (dir->exclude_per_dir &&
+   /*
+* If we know that no files have been added in
+* this directory (i.e. valid_cached_dir() has
+* been executed and set untracked->valid) ..
+*/
+   (!untracked || !untracked->valid ||
+/*
+ * .. and .gitignore does not exist before
+ * (i.e. null exclude_sha1 and skip_worktree is
+ * not set). Then we can skip loading .gitignore,
+ * which would result in ENOENT anyway.
+ * skip_worktree is taken care in read_directory()
+ */
+!is_null_sha1(untracked->exclude_sha1))) {
/*
 * dir->basebuf gets reused by the traversal, but we
 * need fname to remain unchanged to ensure the src
@@ -1778,6 +1792,7 @@ static struct untracked_cache_dir 
*validate_untracked_cache(struct dir_struct *d
  const struct pathspec 
*pathspec)
 {
struct untracked_cache_dir *root;
+   int i;
 
if (!dir->untracked)
return NULL;
@@ -1829,6 +1844,15 @@ static struct untracked_cache_dir 
*validate_untracked_cache(struct dir_struct *d
if (dir->exclude_list_group[EXC_CMDL].nr)
return NULL;
 
+   /*
+* An optimization in prep_exclude() does not play well with
+* CE_SKIP_WORKTREE. It's a rare case anyway, if a single
+* entry has that bit set, disable the whole untracked cache.
+*/
+   for (i = 0; i < active_nr; i++)
+   if (ce_skip_worktree(active_cache[i]))
+   return NULL;
+
if (!dir->untracked->root) {
const int len = sizeof(*dir->untracked->root);
dir->untracked->root = xmalloc(len);
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 11/19] untracked cache: invalidate at index addition or removal

[Nguyễn Thái Ngọc Duy]

Ideally we should implement untracked_cache_remove_from_index() and
untracked_cache_add_to_index() so that they update untracked cache
right away instead of invalidating it and wait for read_directory()
next time to deal with it. But that may need some more work in
unpack-trees.c. So stay simple as the first step.

The new call in add_index_entry_with_check() may look strange because
new calls usually stay close to cache_tree_invalidate_path(). We do it
a bit later than c_t_i_p() in this function because if it's about
replacing the entry with the same name, we don't care (but cache-tree
does).

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c  | 31 +++
 dir.h  |  4 
 read-cache.c   |  4 
 unpack-trees.c |  7 +--
 4 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/dir.c b/dir.c
index c97b0c3..ddc549c 100644
--- a/dir.c
+++ b/dir.c
@@ -2369,3 +2369,34 @@ struct untracked_cache *read_untracked_extension(const 
void *data, unsigned long
free_untracked_cache(uc);
return NULL;
 }
+
+void untracked_cache_invalidate_path(struct index_state *istate,
+const char *path)
+{
+   const char *sep;
+   struct untracked_cache_dir *d;
+   if (!istate->untracked || !istate->untracked->root)
+   return;
+   sep = strrchr(path, '/');
+   if (sep)
+   d = lookup_untracked(istate->untracked,
+istate->untracked->root,
+path, sep - path);
+   else
+   d = istate->untracked->root;
+   istate->untracked->dir_invalidated++;
+   d->valid = 0;
+   d->untracked_nr = 0;
+}
+
+void untracked_cache_remove_from_index(struct index_state *istate,
+  const char *path)
+{
+   untracked_cache_invalidate_path(istate, path);
+}
+
+void untracked_cache_add_to_index(struct index_state *istate,
+ const char *path)
+{
+   untracked_cache_invalidate_path(istate, path);
+}
diff --git a/dir.h b/dir.h
index 014f3ed..8c29324 100644
--- a/dir.h
+++ b/dir.h
@@ -298,6 +298,10 @@ static inline int dir_path_match(const struct dir_entry 
*ent,
  has_trailing_dir);
 }
 
+void untracked_cache_invalidate_path(struct index_state *, const char *);
+void untracked_cache_remove_from_index(struct index_state *, const char *);
+void untracked_cache_add_to_index(struct index_state *, const char *);
+
 void free_untracked_cache(struct untracked_cache *);
 struct untracked_cache *read_untracked_extension(const void *data, unsigned 
long sz);
 void write_untracked_extension(struct strbuf *out, struct untracked_cache 
*untracked);
diff --git a/read-cache.c b/read-cache.c
index 60baeaf..feb10b0 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -79,6 +79,7 @@ void rename_index_entry_at(struct index_state *istate, int 
nr, const char *new_n
memcpy(new->name, new_name, namelen + 1);
 
cache_tree_invalidate_path(istate, old->name);
+   untracked_cache_remove_from_index(istate, old->name);
remove_index_entry_at(istate, nr);
add_index_entry(istate, new, 
ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
 }
@@ -538,6 +539,7 @@ int remove_file_from_index(struct index_state *istate, 
const char *path)
if (pos < 0)
pos = -pos-1;
cache_tree_invalidate_path(istate, path);
+   untracked_cache_remove_from_index(istate, path);
while (pos < istate->cache_nr && !strcmp(istate->cache[pos]->name, 
path))
remove_index_entry_at(istate, pos);
return 0;
@@ -969,6 +971,8 @@ static int add_index_entry_with_check(struct index_state 
*istate, struct cache_e
}
pos = -pos-1;
 
+   untracked_cache_add_to_index(istate, ce->name);
+
/*
 * Inserting a merged entry ("stage 0") into the index
 * will always replace all non-merged entries..
diff --git a/unpack-trees.c b/unpack-trees.c
index 629c658..e5ddb0c 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -9,6 +9,7 @@
 #include "refs.h"
 #include "attr.h"
 #include "split-index.h"
+#include "dir.h"
 
 /*
  * Error messages expected by scripts out of plumbing commands such as
@@ -1255,8 +1256,10 @@ static int verify_uptodate_sparse(const struct 
cache_entry *ce,
 static void invalidate_ce_path(const struct cache_entry *ce,
   struct unpack_trees_options *o)
 {
-   if (ce)
-   cache_tree_invalidate_path(o->src_index, ce->name);
+   if (!ce)
+   return;
+   cache_tree_invalidate_path(o->src_index, ce->name);
+   untracked_cache_invalidate_path(o->src_index, ce->name);
 }
 
 /*
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 09/19] untracked cache: save to an index extension

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 cache.h  |  3 +++
 dir.c| 85 
 dir.h|  1 +
 read-cache.c | 12 +
 4 files changed, 101 insertions(+)

diff --git a/cache.h b/cache.h
index 5b86065..4f903fa 100644
--- a/cache.h
+++ b/cache.h
@@ -298,6 +298,8 @@ static inline unsigned int canon_mode(unsigned int mode)
 #define SPLIT_INDEX_ORDERED(1 << 6)
 
 struct split_index;
+struct untracked_cache;
+
 struct index_state {
struct cache_entry **cache;
unsigned int version;
@@ -311,6 +313,7 @@ struct index_state {
struct hashmap name_hash;
struct hashmap dir_hash;
unsigned char sha1[20];
+   struct untracked_cache *untracked;
 };
 
 extern struct index_state the_index;
diff --git a/dir.c b/dir.c
index d4f9830..d696388 100644
--- a/dir.c
+++ b/dir.c
@@ -12,6 +12,7 @@
 #include "refs.h"
 #include "wildmatch.h"
 #include "pathspec.h"
+#include "varint.h"
 
 struct path_simplify {
int len;
@@ -2145,3 +2146,87 @@ void clear_directory(struct dir_struct *dir)
}
strbuf_release(&dir->basebuf);
 }
+
+struct ondisk_untracked_cache {
+   struct stat_data info_exclude_stat;
+   struct stat_data excludes_file_stat;
+   uint32_t dir_flags;
+   unsigned char info_exclude_sha1[20];
+   unsigned char excludes_file_sha1[20];
+   char exclude_per_dir[1];
+};
+
+static void stat_data_to_disk(struct stat_data *to, const struct stat_data 
*from)
+{
+   to->sd_ctime.sec  = htonl(from->sd_ctime.sec);
+   to->sd_ctime.nsec = htonl(from->sd_ctime.nsec);
+   to->sd_mtime.sec  = htonl(from->sd_mtime.sec);
+   to->sd_mtime.nsec = htonl(from->sd_mtime.nsec);
+   to->sd_dev= htonl(from->sd_dev);
+   to->sd_ino= htonl(from->sd_ino);
+   to->sd_uid= htonl(from->sd_uid);
+   to->sd_gid= htonl(from->sd_gid);
+   to->sd_size   = htonl(from->sd_size);
+}
+
+static void write_one_dir(struct strbuf *out, struct untracked_cache_dir 
*untracked)
+{
+   struct stat_data stat_data;
+   unsigned char intbuf[16];
+   unsigned int intlen, value;
+   int i;
+
+   stat_data_to_disk(&stat_data, &untracked->stat_data);
+   strbuf_add(out, &stat_data, sizeof(stat_data));
+   strbuf_add(out, untracked->exclude_sha1, 20);
+
+   /*
+* untracked_nr should be reset whenever valid is clear, but
+* for safety..
+*/
+   if (!untracked->valid) {
+   untracked->untracked_nr = 0;
+   untracked->check_only = 0;
+   }
+
+   value  = untracked->valid;
+   value |= untracked->check_only   << 1;
+   value |= untracked->untracked_nr << 2;
+   intlen = encode_varint(value, intbuf);
+   strbuf_add(out, intbuf, intlen);
+
+   /* skip non-recurse directories */
+   for (i = 0, value = 0; i < untracked->dirs_nr; i++)
+   if (untracked->dirs[i]->recurse)
+   value++;
+   intlen = encode_varint(value, intbuf);
+   strbuf_add(out, intbuf, intlen);
+
+   strbuf_add(out, untracked->name, strlen(untracked->name) + 1);
+
+   for (i = 0; i < untracked->untracked_nr; i++)
+   strbuf_add(out, untracked->untracked[i],
+  strlen(untracked->untracked[i]) + 1);
+
+   for (i = 0; i < untracked->dirs_nr; i++)
+   if (untracked->dirs[i]->recurse)
+   write_one_dir(out, untracked->dirs[i]);
+}
+
+void write_untracked_extension(struct strbuf *out, struct untracked_cache 
*untracked)
+{
+   struct ondisk_untracked_cache *ouc;
+   int len = 0;
+   if (untracked->exclude_per_dir)
+   len = strlen(untracked->exclude_per_dir);
+   ouc = xmalloc(sizeof(*ouc) + len);
+   stat_data_to_disk(&ouc->info_exclude_stat, 
&untracked->ss_info_exclude.stat);
+   stat_data_to_disk(&ouc->excludes_file_stat, 
&untracked->ss_excludes_file.stat);
+   hashcpy(ouc->info_exclude_sha1, untracked->ss_info_exclude.sha1);
+   hashcpy(ouc->excludes_file_sha1, untracked->ss_excludes_file.sha1);
+   ouc->dir_flags = htonl(untracked->dir_flags);
+   memcpy(ouc->exclude_per_dir, untracked->exclude_per_dir, len + 1);
+   strbuf_add(out, ouc, sizeof(*ouc) + len);
+   if (untracked->root)
+   write_one_dir(out, untracked->root);
+}
diff --git a/dir.h b/dir.h
index c302dcb..7ef0f63 100644
--- a/dir.h
+++ b/dir.h
@@ -297,4 +297,5 @@ static inline int dir_path_match(const struct dir_entry 
*ent,
  has_trailing_dir);
 }
 
+void write_untracked_extension(struct strbuf *out, struct untracked_cache 
*untracked);
 #endif
diff --git a/read-cache.c b/read-cache.c
index 8f3e9eb..efff4e2 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -38,6 +38,7 @@ static struct cache_entry *refresh_cache_entry(struct 
cache_entry *ce,
 #define CACHE_EXT_TREE 0x54524545  /* "TREE" */
 #d

[PATCH 12/19] read-cache.c: split racy stat test to a separate function

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 read-cache.c | 24 +++-
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/read-cache.c b/read-cache.c
index feb10b0..a14646b 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -271,20 +271,26 @@ static int ce_match_stat_basic(const struct cache_entry 
*ce, struct stat *st)
return changed;
 }
 
-static int is_racy_timestamp(const struct index_state *istate,
-const struct cache_entry *ce)
+static int is_racy_stat(const struct index_state *istate,
+   const struct stat_data *sd)
 {
-   return (!S_ISGITLINK(ce->ce_mode) &&
-   istate->timestamp.sec &&
+   return (istate->timestamp.sec &&
 #ifdef USE_NSEC
 /* nanosecond timestamped files can also be racy! */
-   (istate->timestamp.sec < ce->ce_stat_data.sd_mtime.sec ||
-(istate->timestamp.sec == ce->ce_stat_data.sd_mtime.sec &&
- istate->timestamp.nsec <= ce->ce_stat_data.sd_mtime.nsec))
+   (istate->timestamp.sec < sd->sd_mtime.sec ||
+(istate->timestamp.sec == sd->sd_mtime.sec &&
+ istate->timestamp.nsec <= sd->sd_mtime.nsec))
 #else
-   istate->timestamp.sec <= ce->ce_stat_data.sd_mtime.sec
+   istate->timestamp.sec <= sd->sd_mtime.sec
 #endif
-);
+   );
+}
+
+static int is_racy_timestamp(const struct index_state *istate,
+const struct cache_entry *ce)
+{
+   return (!S_ISGITLINK(ce->ce_mode) &&
+   is_racy_stat(istate, &ce->ce_stat_data));
 }
 
 int ie_match_stat(const struct index_state *istate,
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 10/19] untracked cache: load from UNTR index extension

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c| 147 +--
 dir.h|   3 ++
 read-cache.c |   5 ++
 3 files changed, 151 insertions(+), 4 deletions(-)

diff --git a/dir.c b/dir.c
index d696388..c97b0c3 100644
--- a/dir.c
+++ b/dir.c
@@ -2085,15 +2085,27 @@ void setup_standard_excludes(struct dir_struct *dir)
}
if (!access_or_warn(path, R_OK, 0)) {
struct sha1_stat *ss = NULL;
-   if (dir->untracked)
+   int ss_valid = 0;
+   if (dir->untracked) {
ss = &dir->ss_info_exclude;
-   add_excludes_from_file_1(dir, path, ss, 0);
+   if (dir->untracked->loaded) {
+   *ss = dir->untracked->ss_info_exclude;
+   ss_valid = 1;
+   }
+   }
+   add_excludes_from_file_1(dir, path, ss, ss_valid);
}
if (excludes_file && !access_or_warn(excludes_file, R_OK, 0)) {
struct sha1_stat *ss = NULL;
-   if (dir->untracked)
+   int ss_valid = 0;
+   if (dir->untracked) {
ss = &dir->ss_excludes_file;
-   add_excludes_from_file_1(dir, excludes_file, ss, 0);
+   if (dir->untracked->loaded) {
+   *ss = dir->untracked->ss_excludes_file;
+   ss_valid = 1;
+   }
+   }
+   add_excludes_from_file_1(dir, excludes_file, ss, ss_valid);
}
 }
 
@@ -2230,3 +2242,130 @@ void write_untracked_extension(struct strbuf *out, 
struct untracked_cache *untra
if (untracked->root)
write_one_dir(out, untracked->root);
 }
+
+static void free_untracked(struct untracked_cache_dir *ucd)
+{
+   int i;
+   if (!ucd)
+   return;
+   for (i = 0; i < ucd->dirs_nr; i++)
+   free_untracked(ucd->dirs[i]);
+   for (i = 0; i < ucd->untracked_nr; i++)
+   free(ucd->untracked[i]);
+   free(ucd->untracked);
+   free(ucd->dirs);
+   free(ucd);
+}
+
+void free_untracked_cache(struct untracked_cache *uc)
+{
+   if (uc)
+   free_untracked(uc->root);
+   free(uc);
+}
+
+static void stat_data_from_disk(struct stat_data *to, const struct stat_data 
*from)
+{
+   to->sd_ctime.sec  = get_be32(&from->sd_ctime.sec);
+   to->sd_ctime.nsec = get_be32(&from->sd_ctime.nsec);
+   to->sd_mtime.sec  = get_be32(&from->sd_mtime.sec);
+   to->sd_mtime.nsec = get_be32(&from->sd_mtime.nsec);
+   to->sd_dev= get_be32(&from->sd_dev);
+   to->sd_ino= get_be32(&from->sd_ino);
+   to->sd_uid= get_be32(&from->sd_uid);
+   to->sd_gid= get_be32(&from->sd_gid);
+   to->sd_size   = get_be32(&from->sd_size);
+}
+
+static int read_one_dir(struct untracked_cache_dir **untracked_,
+   const unsigned char *data_, unsigned long sz)
+{
+#define NEXT(x) \
+   next = data + (x); \
+   if (next > data_ + sz) \
+   return -1;
+
+   struct untracked_cache_dir ud, *untracked;
+   const unsigned char *next, *data = data_;
+   unsigned int value;
+   int i, len;
+
+   memset(&ud, 0, sizeof(ud));
+
+   NEXT(sizeof(struct stat_data));
+   stat_data_from_disk(&ud.stat_data, (struct stat_data *)data);
+   data = next;
+
+   NEXT(20);
+   hashcpy(ud.exclude_sha1, data);
+   data = next;
+
+   next = data;
+   value = decode_varint(&next);
+   if (next > data_ + sz)
+   return -1;
+   ud.recurse = 1;
+   ud.valid = value & 1;
+   ud.check_only = (value >> 1) & 1;
+   ud.untracked_alloc = ud.untracked_nr = value >> 2;
+   if (ud.untracked_nr)
+   ud.untracked = xmalloc(sizeof(*ud.untracked) * ud.untracked_nr);
+   data = next;
+
+   next = data;
+   ud.dirs_alloc = ud.dirs_nr = decode_varint(&next);
+   if (next > data_ + sz)
+   return -1;
+   ud.dirs = xmalloc(sizeof(*ud.dirs) * ud.dirs_nr);
+   data = next;
+
+   len = strlen((const char *)data);
+   NEXT(len + 1);
+   *untracked_ = untracked = xmalloc(sizeof(*untracked) + len);
+   memcpy(untracked, &ud, sizeof(ud));
+   memcpy(untracked->name, data, len + 1);
+   data = next;
+
+   for (i = 0; i < untracked->untracked_nr; i++) {
+   len = strlen((const char *)data);
+   NEXT(len + 1);
+   untracked->untracked[i] = xstrdup((const char*)data);
+   data = next;
+   }
+
+   for (i = 0; i < untracked->dirs_nr; i++) {
+   len = read_one_dir(untracked->dirs + i, data, sz - (data - 
data_));
+   if (len < 0)
+   return -1;
+   data += len;
+   }
+   return data - data_;
+}
+

[PATCH 15/19] untracked cache: mark index dirty if untracked cache is updated

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 cache.h  | 1 +
 dir.c| 9 +
 read-cache.c | 2 +-
 3 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/cache.h b/cache.h
index 2b93217..dcbdc3c 100644
--- a/cache.h
+++ b/cache.h
@@ -296,6 +296,7 @@ static inline unsigned int canon_mode(unsigned int mode)
 #define RESOLVE_UNDO_CHANGED   (1 << 4)
 #define CACHE_TREE_CHANGED (1 << 5)
 #define SPLIT_INDEX_ORDERED(1 << 6)
+#define UNTRACKED_CHANGED   (1 << 7)
 
 struct split_index;
 struct untracked_cache;
diff --git a/dir.c b/dir.c
index 57b49f7..d373d9a 100644
--- a/dir.c
+++ b/dir.c
@@ -1929,6 +1929,15 @@ int read_directory(struct dir_struct *dir, const char 
*path, int len, const stru
 dir->untracked->gitignore_invalidated,
 dir->untracked->dir_invalidated,
 dir->untracked->dir_opened);
+   if (dir->untracked == the_index.untracked &&
+   (dir->untracked->dir_opened ||
+dir->untracked->gitignore_invalidated ||
+dir->untracked->dir_invalidated))
+   the_index.cache_changed |= UNTRACKED_CHANGED;
+   if (dir->untracked != the_index.untracked) {
+   free(dir->untracked);
+   dir->untracked = NULL;
+   }
}
return dir->nr;
 }
diff --git a/read-cache.c b/read-cache.c
index 177cbae..779c080 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -43,7 +43,7 @@ static struct cache_entry *refresh_cache_entry(struct 
cache_entry *ce,
 /* changes that can be kept in $GIT_DIR/index (basically all extensions) */
 #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \
 CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \
-SPLIT_INDEX_ORDERED)
+SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)
 
 struct index_state the_index;
 static const char *alternate_index_output;
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 18/19] update-index: test the system before enabling untracked cache

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 Documentation/git-update-index.txt |   6 ++
 builtin/update-index.c | 146 +
 2 files changed, 152 insertions(+)

diff --git a/Documentation/git-update-index.txt 
b/Documentation/git-update-index.txt
index 16f2686..fab1fea 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -180,6 +180,12 @@ may not support it yet.
system must change `st_mtime` field of a directory if files
are added or deleted in that directory.
 
+--force-untracked-cache::
+   For safety, `--untracked-cache` performs tests on the working
+   directory to make sure untracked cache can be used. These
+   tests can take a few seconds. `--force-untracked-cache` can be
+   used to skip the tests.
+
 \--::
Do not interpret any more arguments as options.
 
diff --git a/builtin/update-index.c b/builtin/update-index.c
index e57e2d7..471c0b4 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -48,6 +48,145 @@ static void report(const char *fmt, ...)
va_end(vp);
 }
 
+static void remove_test_directory(void)
+{
+   struct strbuf sb = STRBUF_INIT;
+   strbuf_addstr(&sb, "dir-mtime-test");
+   remove_dir_recursively(&sb, 0);
+   strbuf_release(&sb);
+}
+
+static void xmkdir(const char *path)
+{
+   if (mkdir(path, 0700))
+   die_errno(_("failed to create directory %s"), path);
+}
+
+static int xstat(const char *path, struct stat *st)
+{
+   if (stat(path, st))
+   die_errno(_("failed to stat %s"), path);
+   return 0;
+}
+
+static int create_file(const char *path)
+{
+   int fd = open(path, O_CREAT | O_RDWR, 0644);
+   if (fd < 0)
+   die_errno(_("failed to create file %s"), path);
+   return fd;
+}
+
+static void xunlink(const char *path)
+{
+   if (unlink(path))
+   die_errno(_("failed to delete file %s"), path);
+}
+
+static void xrmdir(const char *path)
+{
+   if (rmdir(path))
+   die_errno(_("failed to delete directory %s"), path);
+}
+
+static void avoid_racy(void)
+{
+   /*
+* not use if we could usleep(10) if USE_NSEC is defined. The
+* field nsec could be there, but the OS could choose to
+* ignore it?
+*/
+   sleep(1);
+}
+
+static int test_if_untracked_cache_is_supported(void)
+{
+   struct stat st;
+   struct stat_data base;
+   int fd;
+
+   fprintf(stderr, _("Testing "));
+   xmkdir("dir-mtime-test");
+   atexit(remove_test_directory);
+   xstat("dir-mtime-test", &st);
+   fill_stat_data(&base, &st);
+   fputc('.', stderr);
+
+   avoid_racy();
+   fd = create_file("dir-mtime-test/newfile");
+   xstat("dir-mtime-test", &st);
+   if (!match_stat_data(&base, &st)) {
+   fputc('\n', stderr);
+   fprintf_ln(stderr,_("directory stat info does not "
+   "change after adding a new file"));
+   return 0;
+   }
+   fill_stat_data(&base, &st);
+   fputc('.', stderr);
+
+   avoid_racy();
+   xmkdir("dir-mtime-test/new-dir");
+   xstat("dir-mtime-test", &st);
+   if (!match_stat_data(&base, &st)) {
+   fputc('\n', stderr);
+   fprintf_ln(stderr, _("directory stat info does not change "
+"after adding a new directory"));
+   return 0;
+   }
+   fill_stat_data(&base, &st);
+   fputc('.', stderr);
+
+   avoid_racy();
+   write_or_die(fd, "data", 4);
+   close(fd);
+   xstat("dir-mtime-test", &st);
+   if (match_stat_data(&base, &st)) {
+   fputc('\n', stderr);
+   fprintf_ln(stderr, _("directory stat info changes "
+"after updating a file"));
+   return 0;
+   }
+   fputc('.', stderr);
+
+   avoid_racy();
+   close(create_file("dir-mtime-test/new-dir/new"));
+   xstat("dir-mtime-test", &st);
+   if (match_stat_data(&base, &st)) {
+   fputc('\n', stderr);
+   fprintf_ln(stderr, _("directory stat info changes after "
+"adding a file inside subdirectory"));
+   return 0;
+   }
+   fputc('.', stderr);
+
+   avoid_racy();
+   xunlink("dir-mtime-test/newfile");
+   xstat("dir-mtime-test", &st);
+   if (!match_stat_data(&base, &st)) {
+   fputc('\n', stderr);
+   fprintf_ln(stderr, _("directory stat info does not "
+"change after deleting a file"));
+   return 0;
+   }
+   fill_stat_data(&base, &st);
+   fputc('.', stderr);
+
+   avoid_racy();
+   xunlink("dir-mtime-test/new-dir/new");
+   xrmdir("dir-mtime-test/new-dir");
+   xstat("dir-mtime-test", &st);
+   if (!match_stat_data(&base, 

[PATCH 19/19] t7063: tests for untracked cache

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 .gitignore |   1 +
 Makefile   |   1 +
 t/t7063-status-untracked-cache.sh (new +x) | 353 +
 test-dump-untracked-cache.c (new)  |  61 +
 4 files changed, 416 insertions(+)
 create mode 100755 t/t7063-status-untracked-cache.sh
 create mode 100644 test-dump-untracked-cache.c

diff --git a/.gitignore b/.gitignore
index 9ec40fa..fde1143 100644
--- a/.gitignore
+++ b/.gitignore
@@ -183,6 +183,7 @@
 /test-delta
 /test-dump-cache-tree
 /test-dump-split-index
+/test-dump-untracked-cache
 /test-scrap-cache-tree
 /test-genrandom
 /test-hashmap
diff --git a/Makefile b/Makefile
index 356feb5..8070e56 100644
--- a/Makefile
+++ b/Makefile
@@ -552,6 +552,7 @@ TEST_PROGRAMS_NEED_X += test-date
 TEST_PROGRAMS_NEED_X += test-delta
 TEST_PROGRAMS_NEED_X += test-dump-cache-tree
 TEST_PROGRAMS_NEED_X += test-dump-split-index
+TEST_PROGRAMS_NEED_X += test-dump-untracked-cache
 TEST_PROGRAMS_NEED_X += test-genrandom
 TEST_PROGRAMS_NEED_X += test-hashmap
 TEST_PROGRAMS_NEED_X += test-index-version
diff --git a/t/t7063-status-untracked-cache.sh 
b/t/t7063-status-untracked-cache.sh
new file mode 100755
index 000..2b2ffd7
--- /dev/null
+++ b/t/t7063-status-untracked-cache.sh
@@ -0,0 +1,353 @@
+#!/bin/sh
+
+test_description='test untracked cache'
+
+. ./test-lib.sh
+
+avoid_racy() {
+   sleep 1
+}
+
+git update-index --untracked-cache
+# It's fine if git update-index returns an error code other than one,
+# it'll be caught in the first test.
+if test $? -eq 1; then
+   skip_all='This system does not support untracked cache'
+   test_done
+fi
+
+test_expect_success 'setup' '
+   git init worktree &&
+   cd worktree &&
+   mkdir done dtwo dthree &&
+   touch one two three done/one dtwo/two dthree/three &&
+   git add one two done/one &&
+   : >.git/info/exclude &&
+   git update-index --untracked-cache
+'
+
+test_expect_success 'untracked cache is empty' '
+   test-dump-untracked-cache >../actual &&
+   cat >../expect <../status.expect <../dump.expect <../trace &&
+   GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
+   git status --porcelain >../actual &&
+   test_cmp ../status.expect ../actual &&
+   cat >../trace.expect <../actual &&
+   test_cmp ../dump.expect ../actual
+'
+
+test_expect_success 'status second time (fully populated cache)' '
+   avoid_racy &&
+   : >../trace &&
+   GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
+   git status --porcelain >../actual &&
+   test_cmp ../status.expect ../actual &&
+   cat >../trace.expect <../actual &&
+   test_cmp ../dump.expect ../actual
+'
+
+test_expect_success 'modify in root directory, one dir invalidation' '
+   avoid_racy &&
+   : >four &&
+   : >../trace &&
+   GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
+   git status --porcelain >../actual &&
+   cat >../status.expect <../trace.expect <../actual &&
+   cat >../expect <.gitignore &&
+   : >../trace &&
+   GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
+   git status --porcelain >../actual &&
+   cat >../status.expect <../trace.expect <../actual &&
+   cat >../expect <>.git/info/exclude &&
+   : >../trace &&
+   GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
+   git status --porcelain >../actual &&
+   cat >../status.expect <../trace.expect <../actual &&
+   cat >../expect <../actual &&
+   cat >../expect <../trace &&
+   GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
+   git status --porcelain >../actual &&
+   cat >../status.expect <../trace.expect <../actual &&
+   cat >../expect <../actual &&
+   cat >../expect <../trace &&
+   GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
+   git status --porcelain >../actual &&
+   cat >../status.expect <../trace.expect <../actual &&
+   cat >../expect name);
+}
+
+static void dump(struct untracked_cache_dir *ucd, struct strbuf *base)
+{
+   int i, len;
+   qsort(ucd->untracked, ucd->untracked_nr, sizeof(*ucd->untracked),
+ compare_untracked);
+   qsort(ucd->dirs, ucd->dirs_nr, sizeof(*ucd->dirs),
+ compare_dir);
+   len = base->len;
+   strbuf_addf(base, "%s/", ucd->name);
+   printf("%s %s", base->buf,
+  sha1_to_hex(ucd->exclude_sha1));
+   if (ucd->recurse)
+   fputs(" recurse", stdout);
+   if (ucd->check_only)
+   fputs(" check_only", stdout);
+   if (ucd->valid)
+   fputs(" valid", stdout);
+   printf("\n");
+   for (i = 0; i < ucd->untracked_nr; i++)
+   printf("%s\n", ucd->untracked[i]);
+   for (i = 0; i < ucd->dirs_nr; i++)
+   dump(ucd->dirs[i], base);
+   strbuf_setlen(base, len);
+}
+
+int main(int ac, char **av)
+{
+   st

[PATCH 17/19] update-index: manually enable or disable untracked cache

[Nguyễn Thái Ngọc Duy]

Overall time saving on "git status" is about 40% in the best case
scenario, removing ..collect_untracked() as the most time consuming
function. read and refresh index operations are now at the top (which
should drop when index-helper and/or watchman support is added). More
numbers and analysis below.

webkit.git
==

169k files. 6k dirs. Lots of test data (i.e. not touched most of the
time)

Base status
---

Index version 4 in split index mode and cache-tree populated. No
untracked cache. It shows how time is consumed by "git status". The
same settings are used for other repos below.

18:28:10.199679 builtin/commit.c:1394   performance: 0.00451 s: 
cmd_status:setup
18:28:10.474847 read-cache.c:1407   performance: 0.274873831 s: read_index
18:28:10.475295 read-cache.c:1407   performance: 0.00656 s: read_index
18:28:10.728443 preload-index.c:131 performance: 0.253147487 s: 
read_index_preload
18:28:10.741422 read-cache.c:1254   performance: 0.012868340 s: 
refresh_index
18:28:10.752300 wt-status.c:623 performance: 0.010421357 s: 
wt_status_collect_changes_worktree
18:28:10.762069 wt-status.c:629 performance: 0.009644748 s: 
wt_status_collect_changes_index
18:28:11.601019 wt-status.c:632 performance: 0.838859547 s: 
wt_status_collect_untracked
18:28:11.605939 builtin/commit.c:1421   performance: 0.004835004 s: 
cmd_status:update_index
18:28:11.606580 trace.c:415 performance: 1.407878388 s: git 
command: 'git' 'status'

Populating status
-

This is after enabling untracked cache and the cache is still empty.
We see a slight increase in .._collect_untracked() and update_index
(because new cache has to be written to $GIT_DIR/index).

18:28:18.915213 builtin/commit.c:1394   performance: 0.00326 s: 
cmd_status:setup
18:28:19.197364 read-cache.c:1407   performance: 0.281901416 s: read_index
18:28:19.197754 read-cache.c:1407   performance: 0.00546 s: read_index
18:28:19.451355 preload-index.c:131 performance: 0.253599607 s: 
read_index_preload
18:28:19.464400 read-cache.c:1254   performance: 0.012935336 s: 
refresh_index
18:28:19.475115 wt-status.c:623 performance: 0.010236920 s: 
wt_status_collect_changes_worktree
18:28:19.486022 wt-status.c:629 performance: 0.010801685 s: 
wt_status_collect_changes_index
18:28:20.362660 wt-status.c:632 performance: 0.876551366 s: 
wt_status_collect_untracked
18:28:20.396199 builtin/commit.c:1421   performance: 0.033447969 s: 
cmd_status:update_index
18:28:20.396939 trace.c:415 performance: 1.482695902 s: git 
command: 'git' 'status'

Populated status


After the cache is populated, wt_status_collect_untracked() drops 82%
from 0.838s to 0.144s. Overall time drops 45%. Top offenders are now
read_index() and read_index_preload().

18:28:20.408605 builtin/commit.c:1394   performance: 0.00457 s: 
cmd_status:setup
18:28:20.692864 read-cache.c:1407   performance: 0.283980458 s: read_index
18:28:20.693273 read-cache.c:1407   performance: 0.00661 s: read_index
18:28:20.958814 preload-index.c:131 performance: 0.265540254 s: 
read_index_preload
18:28:20.972375 read-cache.c:1254   performance: 0.013437429 s: 
refresh_index
18:28:20.983959 wt-status.c:623 performance: 0.011146646 s: 
wt_status_collect_changes_worktree
18:28:20.993948 wt-status.c:629 performance: 0.009879094 s: 
wt_status_collect_changes_index
18:28:21.138125 wt-status.c:632 performance: 0.144084737 s: 
wt_status_collect_untracked
18:28:21.173678 builtin/commit.c:1421   performance: 0.035463949 s: 
cmd_status:update_index
18:28:21.174251 trace.c:415 performance: 0.766707355 s: git 
command: 'git' 'status'

gentoo-x86.git
==

This repository is a strange one with a balanced, wide and shallow
worktree (about 100k files and 23k dirs) and no .gitignore in
worktree. .._collect_untracked() time drops 88%, total time drops 56%.

Base status
---
18:20:40.828642 builtin/commit.c:1394   performance: 0.00496 s: 
cmd_status:setup
18:20:41.027233 read-cache.c:1407   performance: 0.198130532 s: read_index
18:20:41.027670 read-cache.c:1407   performance: 0.00581 s: read_index
18:20:41.171716 preload-index.c:131 performance: 0.144045594 s: 
read_index_preload
18:20:41.179171 read-cache.c:1254   performance: 0.007320424 s: 
refresh_index
18:20:41.185785 wt-status.c:623 performance: 0.006144638 s: 
wt_status_collect_changes_worktree
18:20:41.192701 wt-status.c:629 performance: 0.006780184 s: 
wt_status_collect_changes_index
18:20:41.991723 wt-status.c:632 performance: 0.798927029 s: 
wt_status_collect_untracked
18:20:41.994664 builtin/commit.c:1421   performance: 0.002852772 s: 
cmd_status:update_index
18:20:41.995458 trace.c:415 performance: 1.168427502 s: git 
command: 'git' 'status'
Populating status
-
18:20:48.968848 builtin

[PATCH 13/19] untracked cache: avoid racy timestamps

[Nguyễn Thái Ngọc Duy]

When a directory is updated within the same second that its timestamp
is last saved, we cannot realize the directory has been updated by
checking timestamps. Assume the worst (something is update). See
29e4d36 (Racy GIT - 2005-12-20) for more information.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 cache.h  | 2 ++
 dir.c| 5 +++--
 read-cache.c | 8 
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/cache.h b/cache.h
index 4f903fa..2b93217 100644
--- a/cache.h
+++ b/cache.h
@@ -562,6 +562,8 @@ extern void fill_stat_data(struct stat_data *sd, struct 
stat *st);
  * INODE_CHANGED, and DATA_CHANGED.
  */
 extern int match_stat_data(const struct stat_data *sd, struct stat *st);
+extern int match_stat_data_racy(const struct index_state *istate,
+   const struct stat_data *sd, struct stat *st);
 
 extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);
 
diff --git a/dir.c b/dir.c
index ddc549c..3eb1a12 100644
--- a/dir.c
+++ b/dir.c
@@ -678,7 +678,8 @@ static int add_excludes(const char *fname, const char 
*base, int baselen,
close(fd);
if (ss) {
int pos;
-   if (ss_valid && !match_stat_data(&ss->stat, &st))
+   if (ss_valid &&
+   !match_stat_data_racy(&the_index, &ss->stat, &st))
; /* no content change, ss->sha1 still good */
else if (check_index &&
 (pos = cache_name_pos(fname, strlen(fname))) 
>= 0 &&
@@ -1533,7 +1534,7 @@ static int valid_cached_dir(struct dir_struct *dir,
return 0;
}
if (!untracked->valid ||
-   match_stat_data(&untracked->stat_data, &st)) {
+   match_stat_data_racy(&the_index, &untracked->stat_data, &st)) {
if (untracked->valid)
invalidate_directory(dir->untracked, untracked);
fill_stat_data(&untracked->stat_data, &st);
diff --git a/read-cache.c b/read-cache.c
index a14646b..177cbae 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -293,6 +293,14 @@ static int is_racy_timestamp(const struct index_state 
*istate,
is_racy_stat(istate, &ce->ce_stat_data));
 }
 
+int match_stat_data_racy(const struct index_state *istate,
+const struct stat_data *sd, struct stat *st)
+{
+   if (is_racy_stat(istate, sd))
+   return MTIME_CHANGED;
+   return match_stat_data(sd, st);
+}
+
 int ie_match_stat(const struct index_state *istate,
  const struct cache_entry *ce, struct stat *st,
  unsigned int options)
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 14/19] untracked cache: print stats with $GIT_TRACE_UNTRACKED_STATS

[Nguyễn Thái Ngọc Duy]

This could be used to verify correct behavior in tests

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 dir.c | 12 
 1 file changed, 12 insertions(+)

diff --git a/dir.c b/dir.c
index 3eb1a12..57b49f7 100644
--- a/dir.c
+++ b/dir.c
@@ -1918,6 +1918,18 @@ int read_directory(struct dir_struct *dir, const char 
*path, int len, const stru
free_simplify(simplify);
qsort(dir->entries, dir->nr, sizeof(struct dir_entry *), cmp_name);
qsort(dir->ignored, dir->ignored_nr, sizeof(struct dir_entry *), 
cmp_name);
+   if (dir->untracked) {
+   static struct trace_key trace_untracked_stats = 
TRACE_KEY_INIT(UNTRACKED_STATS);
+   trace_printf_key(&trace_untracked_stats,
+"node creation: %u\n"
+"gitignore invalidation: %u\n"
+"directory invalidation: %u\n"
+"opendir: %u\n",
+dir->untracked->dir_created,
+dir->untracked->gitignore_invalidated,
+dir->untracked->dir_invalidated,
+dir->untracked->dir_opened);
+   }
return dir->nr;
 }
 
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 16/19] status: enable untracked cache

[Nguyễn Thái Ngọc Duy]

update_index_if_able() is moved down so that the updated untracked
cache could be written out.

Signed-off-by: Nguyễn Thái Ngọc Duy 
---
 builtin/commit.c | 5 +++--
 wt-status.c  | 2 ++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/builtin/commit.c b/builtin/commit.c
index 81dc622..08e2964 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -1408,13 +1408,14 @@ int cmd_status(int argc, const char **argv, const char 
*prefix)
refresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, 
NULL, NULL);
 
fd = hold_locked_index(&index_lock, 0);
-   if (0 <= fd)
-   update_index_if_able(&the_index, &index_lock);
 
s.is_initial = get_sha1(s.reference, sha1) ? 1 : 0;
s.ignore_submodule_arg = ignore_submodule_arg;
wt_status_collect(&s);
 
+   if (0 <= fd)
+   update_index_if_able(&the_index, &index_lock);
+
if (s.relative_paths)
s.prefix = prefix;
 
diff --git a/wt-status.c b/wt-status.c
index 1bf5d72..42aeebe 100644
--- a/wt-status.c
+++ b/wt-status.c
@@ -585,6 +585,8 @@ static void wt_status_collect_untracked(struct wt_status *s)
DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES;
if (s->show_ignored_files)
dir.flags |= DIR_SHOW_IGNORED_TOO;
+   else
+   dir.untracked = the_index.untracked;
setup_standard_excludes(&dir);
 
fill_directory(&dir, &s->pathspec);
-- 
2.1.0.rc0.78.gc0d8480

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: difftool--helper: exit when reading a prompt answer fails

[Michael J Gruber]

David Aguilar schrieb am 27.10.2014 um 02:10:
> On Sun, Oct 26, 2014 at 05:41:49PM -0700, David Aguilar wrote:
>> On Sun, Oct 26, 2014 at 09:09:20AM +0100, Johannes Sixt wrote:
>>> An attempt to quit difftool by hitting Ctrl-D (EOF) at its prompt does
>>> not quit it, but is treated as if 'yes' was answered to the prompt and
>>> all following prompts, which is contrary to the user's intent. Fix the
>>> error check.
>>>
>>> Signed-off-by: Johannes Sixt 
>>> ---
>>>  Found while reviewing your latest patch.
>>
>>
>> Thanks for the careful review.
>> I have one small question about the test below.
> 
> [snip]
> 
>>> diff --git a/t/t7800-difftool.sh b/t/t7800-difftool.sh
>>> index dc30a51..9cf5dc9 100755
>>> --- a/t/t7800-difftool.sh
>>> +++ b/t/t7800-difftool.sh
>>> @@ -301,6 +301,14 @@ test_expect_success PERL 'say no to the second file' '
>>> ! grep br2 output
>>>  '
>>>  
>>> +test_expect_success PERL 'ending prompt input with EOF' '
>>> +   git difftool -x cat branch output &&
>>> +   ! grep master output &&
>>> +   ! grep branch output &&
>>> +   ! grep m2 output &&
>>> +   ! grep br2 output
>>> +'
>>
>> Should we use "test_must_fail grep ..." instead of "! grep ..." here?
> 
> 
> Nevermind, this is good as-is.
> Using "! grep" is consistent with the rest of the tests in t7800.
> 
> What I'll do is add a follow-up patch in my upcoming reroll
> that swaps all the "! grep" lines to "test_must_fail grep"
> in one step.

Don't do that ;)

test_must_fail is meant for testing (git) commands such that a "failure
return code" is marked as "success", whereas a failure to run the
command is still capturd and marked as a failed test.

For non-git commands like grep sed etc. which we do not perform tests
*on* (but only *with*), the simple negator "!" is fine and preferable.

Michael who has sinned in the past, but repented
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: life cycle documentation

[Ben Harper]

On 10/24/2014 04:56 PM, brian m. carlson wrote:

On Thu, Oct 23, 2014 at 05:15:21PM -0500, Ben Harper wrote:

Greetings,

I am unable to find any documentation regarding the life cycle regarding the
various versions of git.  Is only the current version supported?  What about
older minor/major versions?  At what point does a version go EOL? Currently,
is only 2.1.2 supported?  I would entertain the thought on creating a
RELEASES document if the information is provided.

The development process is outlined at [0].  Usually there's just one
supported release (at the moment, 2.1.2), although I have occasionally
seen releases out of order (from different branches).

Having said that, Git developers try very hard not to break things, so
many people are fine using an older version, such as the ones their
distros provide.  It is considered courteous to try the latest version
before reporting a bug, however.

[0] https://github.com/git/git/blob/master/Documentation/howto/maintain-git.txt

Hey Brian,

Thanks for the clarification.  Someone in IRC mentioned the 
maintain-git.txt file.  I skimmed it and searched for some keywords, but 
was unable to find the information I needed.


Do you feel a RELEASES document is needed or is the maintain-git.txt 
file sufficient?


Ben Harper
OS Deployment Services, RPMDEV
Rackspace Hosting & IUS Community
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

git documentation webpage is unavailable

[Voynar, Stephen S CIV NSWCDD, Z24]

Several weblinks located on your git website (http://git-scm.com/) have 
broken links to the Documentation page (i.e. the links to 
http://git-scm.com/documentation and http://git-scm.com/doc are yielding an 
Internal Server Error).





Stephen Voynar
Computer Scientist
Naval Surface Warfare Center Dahlgren
CBR Analysis, Testing, and Systems Engineering Branch (Z24)
ATTN Kaitlan Phillips
18372 Frontage Road, Suite 318, Dahlgren, VA 22448-5160
(540) 653-8199, DSN 249-8199
stephen.voy...@navy.mil




--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: git documentation webpage is unavailable

[Voynar, Stephen S CIV NSWCDD, Z24]

   Thanks for your rapid response - the weblink now works.





Stephen Voynar
Computer Scientist
Naval Surface Warfare Center Dahlgren
CBR Analysis, Testing, and Systems Engineering Branch (Z24)
ATTN Kaitlan Phillips
18372 Frontage Road, Suite 318, Dahlgren, VA 22448-5160
(540) 653-8199, DSN 249-8199
stephen.voy...@navy.mil






From: Scott Chacon [scha...@gmail.com]
Sent: Monday, October 27, 2014 12:06 PM
To: Voynar, Stephen S CIV NSWCDD, Z24
Cc: git@vger.kernel.org
Subject: Re: git documentation webpage is unavailable

This should be fixed now.

On Mon, Oct 27, 2014 at 3:56 PM, Voynar, Stephen S CIV NSWCDD, Z24 
mailto:stephen.voy...@navy.mil>> wrote:
Several weblinks located on your git website (http://git-scm.com/) have 
broken links to the Documentation page (i.e. the links to 
http://git-scm.com/documentation and http://git-scm.com/doc are yielding an 
Internal Server Error).





Stephen Voynar
Computer Scientist
Naval Surface Warfare Center Dahlgren
CBR Analysis, Testing, and Systems Engineering Branch (Z24)
ATTN Kaitlan Phillips
18372 Frontage Road, Suite 318, Dahlgren, VA 22448-5160
(540) 653-8199, DSN 249-8199
stephen.voy...@navy.mil




--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to 
majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anomaly with the new code - Re: git-svn performance

[Eric Wong]

Eric Wong  wrote:
> Which SVN version are you using?  I'm cloning (currently on r373xx)
> https://svn.r-project.org/R using --stdlayout and
> unable to see memory growth of the git-svn Perl process beyond 40M
> (on a 32-bit system).

git-svn hit 45M and took 11:44 to finish.   My ping times to
svn.r-project.org is around 150ms (I'm running this from a server in
Fremont, California).  I'll keep the repo around and periodically fetch
to see how it runs.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] merge & sequencer: turn "Conflicts:" hint into a comment

[Junio C Hamano]

Jeff King  writes:

> Just to play devil's advocate for a moment, though, are we hurting
> people who find it useful to record that information in the commit
> message?

I thought about it, but ultimately, they are using it wrong if they
did depend on them.  As you said, the paths themselves are not that
interesting, unless they are accompanied by description in the log
message explaining what caused conflicts and how they were resolved
at the semantic level.  The hints are to remind them what conflicts
in which paths to describe.

I do not mind "merge.commentConflictsHint = no" as a backward
compatibility toggle, if somebody cares deeply about it, though.

> If that is the only casualty, I think it is probably a net-win. We may
> want better tooling around viewing the merge later, but that can wait
> until somebody steps up with a real use case (because even that conflict
> list may not be completely what they want; they may also want the list
> of files that were auto-merged successfully, for example).

Yup.

Also Christian's "trailer" series may want to learn the same trick
we did to builtin/commit.c in this series, if it does not already
know about possible trailing comment and blank lines.

>> diff --git a/sequencer.c b/sequencer.c
>> index 0f84bbe..1d97da3 100644
>> --- a/sequencer.c
>> +++ b/sequencer.c
>> @@ -291,13 +291,12 @@ void append_conflicts_hint(struct strbuf *msgbuf)
>>  {
>>  int i;
>>  
>> -strbuf_addstr(msgbuf, "\nConflicts:\n");
>> +strbuf_addch(msgbuf, '\n');
>> +strbuf_commented_addf(msgbuf, "Conflicts:\n");
>>  for (i = 0; i < active_nr;) {
>>  const struct cache_entry *ce = active_cache[i++];
>>  if (ce_stage(ce)) {
>> -strbuf_addch(msgbuf, '\t');
>> -strbuf_addstr(msgbuf, ce->name);
>> -strbuf_addch(msgbuf, '\n');
>> +strbuf_commented_addf(msgbuf, "\t%s\n", ce->name);
>
> This ends up adding a space followed by a tab. Besides being redundant,
> it makes my editor highlight it as a whitespace error. I realize this is
> a pretty minor nit, though.

Interesting ;-)

I do not think it is too hard to teach strbuf_commented_addf() about
the leading HT, but that would be a separate topic; if squashing the
SP-HT to HT is worth doing for this codepath, doing it at that helper
would benefit all callers.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: difftool--helper: exit when reading a prompt answer fails

[Junio C Hamano]

David Aguilar  writes:

>> diff --git a/t/t7800-difftool.sh b/t/t7800-difftool.sh
>> index dc30a51..9cf5dc9 100755
>> --- a/t/t7800-difftool.sh
>> +++ b/t/t7800-difftool.sh
>> @@ -301,6 +301,14 @@ test_expect_success PERL 'say no to the second file' '
>>  ! grep br2 output
>>  '
>>  
>> +test_expect_success PERL 'ending prompt input with EOF' '
>> +git difftool -x cat branch output &&
>> +! grep master output &&
>> +! grep branch output &&
>> +! grep m2 output &&
>> +! grep br2 output
>> +'
>
> Should we use "test_must_fail grep ..." instead of "! grep ..." here?

NO.  We do not expect system-supplied "grep" to dump core and
declare it as a test failure.  test_must_fail is for catching an
expected non-zero status exit from git commands, i.e. when we expect
our binary to correctly notice some condition and report that fact
with non-zero exit status, we do not want to mistake the binary
segfaulting as working correctly.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/3] Makefile: Reorder linker flags in the git executable rule

[Junio C Hamano]

Jeff King  writes:

> On Sun, Oct 26, 2014 at 02:54:56PM -0400, David Michael wrote:
>
>> Yes, the compiler refuses to run by default when a "-L" option occurs
>> after a source/object file.  It tries to interpret it as another file
>> name and fails.
>
> Yeah, I think I have seen similar behavior before, but it has been long
> enough that I no longer remember the compiler in use.
>
>> I believe I can work around the error with an "export _C89_CCMODE=1",
>> but I thought I'd send the patch since this is the only occurrence of
>> the problem, and the argument order is inconsistent with other linker
>> commands in the file.
>
> I don't think working around it makes sense. That would fix your case,
> but nobody else's (though given how long it has been that way without
> complaints, I suspect any other compilers this picky may have died off).

I think you meant s/nobody else's/breaks &/;

With that, I agree with your assessment.  The diff itself is
probably fine as-is (I didn't look at it for more than 10 seconds,
though ;-).  And I agree that it needs to be better explained.


>> Do you want me to resend the patch and reference the IBM documentation
>> in the message?
>
> I don't think you need to. More interesting than documentation is the
> real-world breakage you experienced and the analysis of the situation.
> I'd be fine taking the patch as-is, or if changing anything, mentioning
> the failure mode in the commit message.
>
> -Peff
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH][RFC] grep: add color.grep.matchcontext and color.grep.matchselected

[René Scharfe]

The config option color.grep.match can be used to specify the highlighting
color for matching strings.  Add the options matchContext and matchSelected
to allow different colors to be specified for matching strings in the
context vs. in selected lines.  This is similar to the ms and mc specifiers
in GNU grep's environment variable GREP_COLORS.

Signed-off-by: Rene Scharfe 
---
Only *very* lightly tested, and a test for t/is missing anyway.  Just
wanted to quickly show what I meant.  You'd set color.grep.matchContext=""
to turn off highlighting in context lines.  What do you think?

 Documentation/config.txt |  6 +-
 grep.c   | 29 ++---
 grep.h   |  3 ++-
 3 files changed, 29 insertions(+), 9 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 8b49813..78832ae 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -885,7 +885,11 @@ color.grep.::
 `linenumber`;;
line number prefix (when using `-n`)
 `match`;;
-   matching text
+   matching text (same as setting `matchContext` and `matchSelected`)
+`matchContext`;;
+   matching text in context lines
+`matchSelected`;;
+   matching text in selected lines
 `selected`;;
non-matching text in selected lines
 `separator`;;
diff --git a/grep.c b/grep.c
index 4dc31ea..6e085f8 100644
--- a/grep.c
+++ b/grep.c
@@ -35,7 +35,8 @@ void init_grep_defaults(void)
strcpy(opt->color_filename, "");
strcpy(opt->color_function, "");
strcpy(opt->color_lineno, "");
-   strcpy(opt->color_match, GIT_COLOR_BOLD_RED);
+   strcpy(opt->color_match_context, GIT_COLOR_BOLD_RED);
+   strcpy(opt->color_match_selected, GIT_COLOR_BOLD_RED);
strcpy(opt->color_selected, "");
strcpy(opt->color_sep, GIT_COLOR_CYAN);
opt->color = -1;
@@ -101,12 +102,22 @@ int grep_config(const char *var, const char *value, void 
*cb)
color = opt->color_function;
else if (!strcmp(var, "color.grep.linenumber"))
color = opt->color_lineno;
-   else if (!strcmp(var, "color.grep.match"))
-   color = opt->color_match;
+   else if (!strcmp(var, "color.grep.matchcontext"))
+   color = opt->color_match_context;
+   else if (!strcmp(var, "color.grep.matchselected"))
+   color = opt->color_match_selected;
else if (!strcmp(var, "color.grep.selected"))
color = opt->color_selected;
else if (!strcmp(var, "color.grep.separator"))
color = opt->color_sep;
+   else if (!strcmp(var, "color.grep.match")) {
+   int rc = 0;
+   if (!value)
+   return config_error_nonbool(var);
+   rc |= color_parse(value, opt->color_match_context);
+   rc |= color_parse(value, opt->color_match_selected);
+   return rc;
+   }
 
if (color) {
if (!value)
@@ -144,7 +155,8 @@ void grep_init(struct grep_opt *opt, const char *prefix)
strcpy(opt->color_filename, def->color_filename);
strcpy(opt->color_function, def->color_function);
strcpy(opt->color_lineno, def->color_lineno);
-   strcpy(opt->color_match, def->color_match);
+   strcpy(opt->color_match_context, def->color_match_context);
+   strcpy(opt->color_match_selected, def->color_match_selected);
strcpy(opt->color_selected, def->color_selected);
strcpy(opt->color_sep, def->color_sep);
 }
@@ -1084,7 +1096,7 @@ static void show_line(struct grep_opt *opt, char *bol, 
char *eol,
  const char *name, unsigned lno, char sign)
 {
int rest = eol - bol;
-   char *line_color = NULL;
+   const char *match_color, *line_color = NULL;
 
if (opt->file_break && opt->last_shown == 0) {
if (opt->show_hunk_mark)
@@ -1123,6 +1135,10 @@ static void show_line(struct grep_opt *opt, char *bol, 
char *eol,
int eflags = 0;
 
if (sign == ':')
+   match_color = opt->color_match_selected;
+   else
+   match_color = opt->color_match_context;
+   if (sign == ':')
line_color = opt->color_selected;
else if (sign == '-')
line_color = opt->color_context;
@@ -1135,8 +1151,7 @@ static void show_line(struct grep_opt *opt, char *bol, 
char *eol,
 
output_color(opt, bol, match.rm_so, line_color);
output_color(opt, bol + match.rm_so,
-match.rm_eo - match.rm_so,
-opt->color_match);
+match.rm_eo - match.rm_so, match_color);
bol += match.rm_eo;
rest -= match.rm_eo;
eflags = REG_NOTBOL;
diff --git a/grep.h b/grep.h
index 

Re: [PATCH v3 2/2] difftool: add support for --trust-exit-code

[Junio C Hamano]

David Aguilar  writes:

> +write_script .git/fail-right-file <<\EOF
> +echo "$2"
> +exit 1
> +EOF

This should be inside the next one, no?

> +test_expect_success PERL 'difftool stops on error with --trust-exit-code' '
> + >for-diff &&
> + git add for-diff &&
> + echo file>expect &&
> + test_must_fail git difftool -y --trust-exit-code \
> + --extcmd .git/fail-right-file branch >actual &&
> + test_cmp expect actual &&
> + git reset -- for-diff &&
> + rm -f for-diff .git/fail-right-file
> +'

In other words, this squashed in.

 t/t7800-difftool.sh | 17 -
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/t/t7800-difftool.sh b/t/t7800-difftool.sh
index 4b2f611..69bde7a 100755
--- a/t/t7800-difftool.sh
+++ b/t/t7800-difftool.sh
@@ -104,20 +104,19 @@ test_expect_success PERL 'difftool ignores exit code with 
--no-trust-exit-code'
git difftool -y --no-trust-exit-code -t error branch
 '
 
-write_script .git/fail-right-file <<\EOF
-echo "$2"
-exit 1
-EOF
-
 test_expect_success PERL 'difftool stops on error with --trust-exit-code' '
+   test_when_finished "rm -f for-diff .git/fail-right-file" &&
+   test_when_finished "git reset -- for-diff" &&
+   write_script .git/fail-right-file <<-\EOF &&
+   echo "$2"
+   exit 1
+   EOF
>for-diff &&
git add for-diff &&
-   echo file>expect &&
+   echo file >expect &&
test_must_fail git difftool -y --trust-exit-code \
--extcmd .git/fail-right-file branch >actual &&
-   test_cmp expect actual &&
-   git reset -- for-diff &&
-   rm -f for-diff .git/fail-right-file
+   test_cmp expect actual
 '
 
 test_expect_success PERL 'difftool honors --gui' '
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: flatten-merge history

[Andreas Schwab]

Henning Moll  writes:

> Just a final question: Is it possible to keep the GIT_COMMITTER_DATE in
> all those rebases?

If you want that you need to work with git filter-branch.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Editing git changelog automatically

[Andreas Schwab]

Cong Wang  writes:

> Let's say I want to fix a stupid typo in all of these commits, as
> simply as s/foo/bar/. Usually I use`git rebase -i` and `git commit
> --amend`, but both of them are interactive, apparently I don't want to
> edit them one by one. :)

Both can be scripted, though.

> I know I can change $EDITOR to something like `sed -e 's/foo/bar/'`,
> but this seems pretty ugly. Is there a clean way to do that?

I don't see the ugliness.  There is no requirement that $EDITOR is
interactive.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH][RFC] grep: add color.grep.matchcontext and color.grep.matchselected

[Junio C Hamano]

René Scharfe  writes:

> The config option color.grep.match can be used to specify the highlighting
> color for matching strings.  Add the options matchContext and matchSelected
> to allow different colors to be specified for matching strings in the
> context vs. in selected lines.  This is similar to the ms and mc specifiers
> in GNU grep's environment variable GREP_COLORS.
>
> Signed-off-by: Rene Scharfe 
> ---
> Only *very* lightly tested, and a test for t/is missing anyway.  Just
> wanted to quickly show what I meant.  You'd set color.grep.matchContext=""
> to turn off highlighting in context lines.  What do you think?

I didn't realize that people wanted to see pieces on non-matching
lines highlighted.  It makes certain sense, e.g. it would allow you
to spot near-misses, but that is only true for lines that neighbour
real hits, so...

I like this approach better in that it makes those who want a
different behaviour to do the work without breaking the expectation
of those who are used to the established behaviour.

Zoltan?



>  Documentation/config.txt |  6 +-
>  grep.c   | 29 ++---
>  grep.h   |  3 ++-
>  3 files changed, 29 insertions(+), 9 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 8b49813..78832ae 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -885,7 +885,11 @@ color.grep.::
>  `linenumber`;;
>   line number prefix (when using `-n`)
>  `match`;;
> - matching text
> + matching text (same as setting `matchContext` and `matchSelected`)
> +`matchContext`;;
> + matching text in context lines
> +`matchSelected`;;
> + matching text in selected lines
>  `selected`;;
>   non-matching text in selected lines
>  `separator`;;
> diff --git a/grep.c b/grep.c
> index 4dc31ea..6e085f8 100644
> --- a/grep.c
> +++ b/grep.c
> @@ -35,7 +35,8 @@ void init_grep_defaults(void)
>   strcpy(opt->color_filename, "");
>   strcpy(opt->color_function, "");
>   strcpy(opt->color_lineno, "");
> - strcpy(opt->color_match, GIT_COLOR_BOLD_RED);
> + strcpy(opt->color_match_context, GIT_COLOR_BOLD_RED);
> + strcpy(opt->color_match_selected, GIT_COLOR_BOLD_RED);
>   strcpy(opt->color_selected, "");
>   strcpy(opt->color_sep, GIT_COLOR_CYAN);
>   opt->color = -1;
> @@ -101,12 +102,22 @@ int grep_config(const char *var, const char *value, 
> void *cb)
>   color = opt->color_function;
>   else if (!strcmp(var, "color.grep.linenumber"))
>   color = opt->color_lineno;
> - else if (!strcmp(var, "color.grep.match"))
> - color = opt->color_match;
> + else if (!strcmp(var, "color.grep.matchcontext"))
> + color = opt->color_match_context;
> + else if (!strcmp(var, "color.grep.matchselected"))
> + color = opt->color_match_selected;
>   else if (!strcmp(var, "color.grep.selected"))
>   color = opt->color_selected;
>   else if (!strcmp(var, "color.grep.separator"))
>   color = opt->color_sep;
> + else if (!strcmp(var, "color.grep.match")) {
> + int rc = 0;
> + if (!value)
> + return config_error_nonbool(var);
> + rc |= color_parse(value, opt->color_match_context);
> + rc |= color_parse(value, opt->color_match_selected);
> + return rc;
> + }
>  
>   if (color) {
>   if (!value)
> @@ -144,7 +155,8 @@ void grep_init(struct grep_opt *opt, const char *prefix)
>   strcpy(opt->color_filename, def->color_filename);
>   strcpy(opt->color_function, def->color_function);
>   strcpy(opt->color_lineno, def->color_lineno);
> - strcpy(opt->color_match, def->color_match);
> + strcpy(opt->color_match_context, def->color_match_context);
> + strcpy(opt->color_match_selected, def->color_match_selected);
>   strcpy(opt->color_selected, def->color_selected);
>   strcpy(opt->color_sep, def->color_sep);
>  }
> @@ -1084,7 +1096,7 @@ static void show_line(struct grep_opt *opt, char *bol, 
> char *eol,
> const char *name, unsigned lno, char sign)
>  {
>   int rest = eol - bol;
> - char *line_color = NULL;
> + const char *match_color, *line_color = NULL;
>  
>   if (opt->file_break && opt->last_shown == 0) {
>   if (opt->show_hunk_mark)
> @@ -1123,6 +1135,10 @@ static void show_line(struct grep_opt *opt, char *bol, 
> char *eol,
>   int eflags = 0;
>  
>   if (sign == ':')
> + match_color = opt->color_match_selected;
> + else
> + match_color = opt->color_match_context;
> + if (sign == ':')
>   line_color = opt->color_selected;
>   else if (sign == '-')
>   line_color = opt->color_context;
> @@ -1135,8 +1151,7 @@ static void show_line(struct grep_opt *opt, c

Re: [PATCH][RFC] grep: add color.grep.matchcontext and color.grep.matchselected

[Junio C Hamano]

Junio C Hamano  writes:

> René Scharfe  writes:
>
>> The config option color.grep.match can be used to specify the highlighting
>> color for matching strings.  Add the options matchContext and matchSelected
>> to allow different colors to be specified for matching strings in the
>> context vs. in selected lines.  This is similar to the ms and mc specifiers
>> in GNU grep's environment variable GREP_COLORS.
>>
>> Signed-off-by: Rene Scharfe 
>> ---
>> Only *very* lightly tested, and a test for t/is missing anyway.  Just
>> wanted to quickly show what I meant.  You'd set color.grep.matchContext=""
>> to turn off highlighting in context lines.  What do you think?
>
> I didn't realize that people wanted to see pieces on non-matching
> lines highlighted.  It makes certain sense, e.g. it would allow you
> to spot near-misses, but that is only true for lines that neighbour
> real hits, so...
>
> I like this approach better in that it makes those who want a
> different behaviour to do the work without breaking the expectation
> of those who are used to the established behaviour.

FWIW, here is a backport on top of maint-1.8.5 with Zoltan's tests,
not because I wanted to apply this as a bugfix to maintenance track,
but because I wanted to compare with what has been queued already.

To apply to post f6c5a296 (color_parse: do not mention variable name
in error message, 2014-10-07) codebase, the mangling I did for two
calls to color_parse() function needs to be undone, obviously.



 Documentation/config.txt |  6 +++-
 grep.c   | 29 +++
 grep.h   |  3 +-
 t/t7810-grep.sh  | 94 
 4 files changed, 123 insertions(+), 9 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index ab26963..aa881fc 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -860,7 +860,11 @@ color.grep.::
 `linenumber`;;
line number prefix (when using `-n`)
 `match`;;
-   matching text
+   matching text (same as setting `matchContext` and `matchSelected`)
+`matchContext`;;
+   matching text in context lines
+`matchSelected`;;
+   matching text in selected lines
 `selected`;;
non-matching text in selected lines
 `separator`;;
diff --git a/grep.c b/grep.c
index c668034..f950651 100644
--- a/grep.c
+++ b/grep.c
@@ -35,7 +35,8 @@ void init_grep_defaults(void)
strcpy(opt->color_filename, "");
strcpy(opt->color_function, "");
strcpy(opt->color_lineno, "");
-   strcpy(opt->color_match, GIT_COLOR_BOLD_RED);
+   strcpy(opt->color_match_context, GIT_COLOR_BOLD_RED);
+   strcpy(opt->color_match_selected, GIT_COLOR_BOLD_RED);
strcpy(opt->color_selected, "");
strcpy(opt->color_sep, GIT_COLOR_CYAN);
opt->color = -1;
@@ -96,12 +97,22 @@ int grep_config(const char *var, const char *value, void 
*cb)
color = opt->color_function;
else if (!strcmp(var, "color.grep.linenumber"))
color = opt->color_lineno;
-   else if (!strcmp(var, "color.grep.match"))
-   color = opt->color_match;
+   else if (!strcmp(var, "color.grep.matchcontext"))
+   color = opt->color_match_context;
+   else if (!strcmp(var, "color.grep.matchselected"))
+   color = opt->color_match_selected;
else if (!strcmp(var, "color.grep.selected"))
color = opt->color_selected;
else if (!strcmp(var, "color.grep.separator"))
color = opt->color_sep;
+   else if (!strcmp(var, "color.grep.match")) {
+   int rc = 0;
+   if (!value)
+   return config_error_nonbool(var);
+   color_parse(value, var, opt->color_match_context);
+   color_parse(value, var, opt->color_match_selected);
+   return rc;
+   }
 
if (color) {
if (!value)
@@ -139,7 +150,8 @@ void grep_init(struct grep_opt *opt, const char *prefix)
strcpy(opt->color_filename, def->color_filename);
strcpy(opt->color_function, def->color_function);
strcpy(opt->color_lineno, def->color_lineno);
-   strcpy(opt->color_match, def->color_match);
+   strcpy(opt->color_match_context, def->color_match_context);
+   strcpy(opt->color_match_selected, def->color_match_selected);
strcpy(opt->color_selected, def->color_selected);
strcpy(opt->color_sep, def->color_sep);
 }
@@ -1079,7 +1091,7 @@ static void show_line(struct grep_opt *opt, char *bol, 
char *eol,
  const char *name, unsigned lno, char sign)
 {
int rest = eol - bol;
-   char *line_color = NULL;
+   const char *match_color, *line_color = NULL;
 
if (opt->file_break && opt->last_shown == 0) {
if (opt->show_hunk_mark)
@@ -1118,6 +1130,10 @@ static void show_line(struct grep_opt *opt, char *bol, 
char *eol,
int eflags

Re: Sources for 3.18-rc1 not uploaded

[Junio C Hamano]

René Scharfe  writes:

> That's by design -- extended headers are meant to be extracted as
> plain files by implementations that do not understand them.
>
> http://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html
> says: "If a particular implementation does not recognize the type, or
> the user does not have appropriate privilege to create that type, the
> file shall be extracted as if it were a regular file if the file type
> is defined to have a meaning for the size field that could cause data
> logical records to be written on the medium [...]."

Ahh, thanks for digging this up.  I knew POSIX said something about
this somewhere when I responded (and that is why I said "even though
I wouldn't have minded if the original implementation were to apply
the same umask for these entries that look like "dummy files" to
them."), but I didn't have patience to read it through myself.

> It's surprising and sad to see *pax* implementations not supporting
> pax extended headers in 2014, though.  It seems long file names
> etc. are not common enough.  Or perhaps pax is simply not used that
> much.

I would say that if we really want strictness, the _right_ way
forward might be:

 - Use tar.paxumask patch from Linus, to allow those who are aware
   of and care about the older pax implementations (i.e. Brian), to
   optionally tweak umasks applied to those extended header entries,
   while keeping the traditional behaviour as the default;

 - Warn that the default will change to use tar.paxumask that is the
   same as tar.umask in some future version of Git;

 - In some future version, flip the default.

Given that it will be a race between us flipping the default and the
affected implementations of extraction tools going extinct, however,
I do not think such a transition would be of high priority.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] git-fetch.txt: Update over git fetch example on using `+`.

[Roberto Decurnex]

From: Roberto Decurnex 

Making clear that `+` before a branch name will force the update even for 
non-fast-forward branches.

Signed-off-by: Roberto Decurnex 
---
 Documentation/git-fetch.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
index 8deb614..b971a2b 100644
--- a/Documentation/git-fetch.txt
+++ b/Documentation/git-fetch.txt
@@ -124,7 +124,7 @@ This updates (or creates, as necessary) branches `pu` and 
`tmp` in
 the local repository by fetching from the branches (respectively)
 `pu` and `maint` from the remote repository.
 +
-The `pu` branch will be updated even if it is does not fast-forward,
+The `pu` branch will be updated even if it is non-fast-forward,
 because it is prefixed with a plus sign; `tmp` will not be.
 
 * Peek at a remote's branch, without configuring the remote in your local
-- 
2.0.4

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] merge & sequencer: turn "Conflicts:" hint into a comment

[Junio C Hamano]

Junio C Hamano  writes:

>>> diff --git a/sequencer.c b/sequencer.c
>>> index 0f84bbe..1d97da3 100644
>>> --- a/sequencer.c
>>> +++ b/sequencer.c
>>> @@ -291,13 +291,12 @@ void append_conflicts_hint(struct strbuf *msgbuf)
>>>  {
>>> int i;
>>>  
>>> -   strbuf_addstr(msgbuf, "\nConflicts:\n");
>>> +   strbuf_addch(msgbuf, '\n');
>>> +   strbuf_commented_addf(msgbuf, "Conflicts:\n");
>>> for (i = 0; i < active_nr;) {
>>> const struct cache_entry *ce = active_cache[i++];
>>> if (ce_stage(ce)) {
>>> -   strbuf_addch(msgbuf, '\t');
>>> -   strbuf_addstr(msgbuf, ce->name);
>>> -   strbuf_addch(msgbuf, '\n');
>>> +   strbuf_commented_addf(msgbuf, "\t%s\n", ce->name);
>>
>> This ends up adding a space followed by a tab. Besides being redundant,
>> it makes my editor highlight it as a whitespace error. I realize this is
>> a pretty minor nit, though.
>
> Interesting ;-)
>
> I do not think it is too hard to teach strbuf_commented_addf() about
> the leading HT, but that would be a separate topic; if squashing the
> SP-HT to HT is worth doing for this codepath, doing it at that helper
> would benefit all callers.

-- >8 --
Subject: [PATCH] strbuf_add_lines(): avoid SP-HT sequence

The strbuf_add_commented_lines() function passes a pair of prefixes,
one for a line that has some meat on it, and the other for an empty
line.  The former is set to a comment char followed by a SP, while
the latter is set to just the comment char.  This is to give a SP
after the comment character, e.g. "# \n" and to avoid
emitting an unsightly "# \n" in its output.

Teach the machinery to also use the latter space-less prefix when
the payload line begins with a tab; otherwise we will end up showing
"# \t\n" which is similarly unsightly.

Signed-off-by: Junio C Hamano 
---
 strbuf.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/strbuf.c b/strbuf.c
index 0346e74..88cafd4 100644
--- a/strbuf.c
+++ b/strbuf.c
@@ -229,7 +229,8 @@ static void add_lines(struct strbuf *out,
const char *next = memchr(buf, '\n', size);
next = next ? (next + 1) : (buf + size);
 
-   prefix = (prefix2 && buf[0] == '\n') ? prefix2 : prefix1;
+   prefix = ((prefix2 && (buf[0] == '\n' || buf[0] == '\t'))
+ ? prefix2 : prefix1);
strbuf_addstr(out, prefix);
strbuf_add(out, buf, next - buf);
size -= next - buf;
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] merge & sequencer: turn "Conflicts:" hint into a comment

[Jonathan Nieder]

Jeff King wrote:

> For the most part, combined-diff (and --cc) will show the interesting
> cases anyway. But if you take a whole file from one side of the merge,
> then there is nothing interesting for diff to show. Do people still want
> to get that more complete list of potentially interesting files? And if
> so, how do they do it?  I think there really isn't a great way besides
> repeating the merge.

If you have time to experiment with tr/remerge-diff from pu[1], that
would be welcome.

Maybe some day it can be the default behavior for 'log -p'.

> If that is the only casualty, I think it is probably a net-win.

Yes.

Thanks,
Jonathan

[1] http://thread.gmane.org/gmane.comp.version-control.git/256591
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 01/19] dir.c: optionally compute sha-1 of a .gitignore file

[Junio C Hamano]

Nguyễn Thái Ngọc Duy   writes:

> This is not used anywhere yet. But the goal is to compare quickly if a
> .gitignore file has changed when we have the SHA-1 of both old (cached
> somewhere) and new (from index or a tree) versions.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy 
> ---
>  dir.c | 50 +++---
>  dir.h |  5 +
>  2 files changed, 48 insertions(+), 7 deletions(-)
>
> diff --git a/dir.c b/dir.c
> index bd274a7..33a35c1 100644
> --- a/dir.c
> +++ b/dir.c
> @@ -466,7 +466,8 @@ void add_exclude(const char *string, const char *base,
>   x->el = el;
>  }
>  
> -static void *read_skip_worktree_file_from_index(const char *path, size_t 
> *size)
> +static void *read_skip_worktree_file_from_index(const char *path, size_t 
> *size,
> + struct sha1_stat *ss)
>  {
>   int pos, len;
>   unsigned long sz;
> @@ -485,6 +486,10 @@ static void *read_skip_worktree_file_from_index(const 
> char *path, size_t *size)
>   return NULL;
>   }
>   *size = xsize_t(sz);
> + if (ss) {
> + memset(&ss->stat, 0, sizeof(ss->stat));
> + hashcpy(ss->sha1, active_cache[pos]->sha1);
> + }
>   return data;
>  }
>  
> @@ -529,11 +534,18 @@ static void trim_trailing_spaces(char *buf)
>   *last_space = '\0';
>  }
>  
> -int add_excludes_from_file_to_list(const char *fname,
> -const char *base,
> -int baselen,
> -struct exclude_list *el,
> -int check_index)
> +/*
> + * Given a file with name "fname", read it (either from disk, or from
> + * the index if "check_index" is non-zero), parse it and store the
> + * exclude rules in "el".
> + *
> + * If "ss" is not NULL, compute SHA-1 of the exclude file and fill
> + * stat data from disk (only valid if add_excludes returns zero). If
> + * ss_valid is non-zero, "ss" must contain good value as input.
> + */

Hmm, do we want a separate parameter for that?  Wouldn't it be
sufficient and cleaner to pass is_null_sha1(ss->sha1[]), or
alternatively have an element ss->valid that is initialized to
false?

That is...

> @@ -571,6 +587,19 @@ int add_excludes_from_file_to_list(const char *fname,
>   }
>   buf[size++] = '\n';
>   close(fd);
> + if (ss) {
> + int pos;
> + if (ss_valid && !match_stat_data(&ss->stat, &st))

s/ss_valid/ss->valid/;

> + ; /* no content change, ss->sha1 still good */
> + else if (check_index &&
> +  (pos = cache_name_pos(fname, strlen(fname))) 
> >= 0 &&
> +  !ce_stage(active_cache[pos]) &&
> +  ce_uptodate(active_cache[pos]))
> + hashcpy(ss->sha1, active_cache[pos]->sha1);
> + else
> + hash_sha1_file(buf, size, "blob", ss->sha1);
> + fill_stat_data(&ss->stat, &st);

And toggle "ss->valid = 1"; here...

> @@ -589,6 +618,13 @@ int add_excludes_from_file_to_list(const char *fname,
>   return 0;
>  }
>  
> +int add_excludes_from_file_to_list(const char *fname, const char *base,
> +int baselen, struct exclude_list *el,
> +int check_index)
> +{
> + return add_excludes(fname, base, baselen, el, check_index, NULL, 0);
> +}
> +
>  struct exclude_list *add_exclude_list(struct dir_struct *dir,
> int group_type, const char *src)
>  {
> diff --git a/dir.h b/dir.h
> index 6c45e9d..032d197 100644
> --- a/dir.h
> +++ b/dir.h
> @@ -73,6 +73,11 @@ struct exclude_list_group {
>   struct exclude_list *el;
>  };
>  
> +struct sha1_stat {
> + struct stat_data stat;
> + unsigned char sha1[20];
> +};
> +
>  struct dir_struct {
>   int nr, alloc;
>   int ignored_nr, ignored_alloc;
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: life cycle documentation

[brian m. carlson]

On Mon, Oct 27, 2014 at 09:52:31AM -0500, Ben Harper wrote:
> Thanks for the clarification.  Someone in IRC mentioned the maintain-git.txt
> file.  I skimmed it and searched for some keywords, but was unable to find
> the information I needed.
> 
> Do you feel a RELEASES document is needed or is the maintain-git.txt file
> sufficient?

I don't personally feel one is needed.  However, I'm a semi-regular on
the list, so my opinion of what would be helpful to people less familiar
with the project is going to be skewed.  If you were to send a patch to
add a releases.txt file in the Documentation directory, it would
probably be welcome.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187


signature.asc
Description: Digital signature

Re: Anomaly with the new code - Re: git-svn performance

[Hin-Tak Leung]

--
On Mon, Oct 27, 2014 06:38 GMT Eric Wong wrote:

>Which SVN version are you using?  I'm cloning (currently on r373xx)
>https://svn.r-project.org/R using --stdlayout and
>unable to see memory growth of the git-svn Perl process beyond 40M
>(on a 32-bit system).
>
>I also tried http:// (not https), svn+ssh:// on my local (64-bit) system
>and did not see memory growth, either:
>
>    http://mid.gmane.org/20141027014033.ga4...@dcvr.yhbt.net
>
>I'm using svn 1.6.17 on Debian stable in all cases.

The memory consumption does seem to go up a good deal after r48xxx -ish (the 
total
being about 67xxx-ish now), when there are a fair number of branches. Seeing as
you seem to be able to make the memory consumption drops further,
I'll rebuild git with dropping/adding those patches now.

I also just realised "/usr/bin/time -v git svn fetch --all" also includes the 
periodic auto-
garbage collection from git itself if fetching more than a number of commits,
so may not be accurate once git svn's memory consumption drops below
a certain level. Is there any way of coping with that?

I made a 3rd clone yesterday - it took 8 hours 15 minutes, and 
Command being timed: "git svn fetch --all"
User time (seconds): 6897.80
System time (seconds): 18853.08
Percent of CPU this job got: 86%
Elapsed (wall clock) time (h:mm:ss or m:ss): 8:14:00
...
Maximum resident set size (kbytes): 675436

and fetching the next 8 commits:

$ /usr/bin/time -v git svn fetch --all
M   doc/NEWS.Rd
r66871 = 0a7f50fc04dee174229513a0d80fecfcd12975ca (refs/remotes/trunk)
...
M   doc/manual/R-exts.texi
r66879 = ede68f65df714c3ba283579d85105393c1eccc80 (refs/remotes/trunk)
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Command being timed: "git svn fetch --all"
User time (seconds): 856.82
System time (seconds): 29.78
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 15:03.39
...
Maximum resident set size (kbytes): 791088

and quite similar against the 2nd clone, but against the first clone (which 
were created
by fetching every few days over a few years):

Command being timed: "git svn fetch --all"
User time (seconds): 518.00
System time (seconds): 28.62
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 9:16.84
...
Maximum resident set size (kbytes): 403160

So it seems the first clone is rather different from the recent ones. I haven't 
got round to compare
the branches yet - it is actually easier than I thought, since I only need to 
compare
the branch HEADs. (I already mentioned that trunk is different, due to a blank 
vs 3 word
commit message about 2 years ago - I reckon I might see similar issues in the 
other branches
- I'll go and write a script to check that now).

All recent fetch were done with git 2.1.0 patched with the 6 patches I 
mentioned, on fedora 20
x86_64.

BTW, I have been meaning to ask - are you the same "Eric Wong" who maintained
some chinese packages on Debian some years ago? :-)
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anomaly with the new code - Re: git-svn performance

[Hin-Tak Leung]

--
On Mon, Oct 27, 2014 16:56 GMT Eric Wong wrote:

>Eric Wong  wrote:
>> Which SVN version are you using?  I'm cloning (currently on r373xx)
>> https://svn.r-project.org/R using --stdlayout and
>> unable to see memory growth of the git-svn Perl process beyond 40M
>> (on a 32-bit system).
>
>git-svn hit 45M and took 11:44 to finish.   My ping times to
>svn.r-project.org is around 150ms (I'm running this from a server in
>Fremont, California).  I'll keep the repo around and periodically fetch
>to see how it runs.

I'll apply the 10 patches against 2.1.0 and see then. As I wrote
in my last reply, my 3rd clone took about 8 hours to finish,
and the max resident size is about 700MB (according to GNU "time").

AFAIK the hosting server is in northern Europe (Copahagen?), I think,
so it is supposed to be faster for me fetching from UK.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH][RFC] grep: add color.grep.matchcontext and color.grep.matchselected

[Zoltan Klinger]

I like René's approach, too. It's more flexible, supports the old
behaviour and it scratches my itch as well.
Don't mind if you dropped my patch and used René's instead.

>> Only *very* lightly tested, and a test for t/is missing anyway.  Just
>> wanted to quickly show what I meant.  You'd set color.grep.matchContext=""
>> to turn off highlighting in context lines.  What do you think?
>
> I didn't realize that people wanted to see pieces on non-matching
> lines highlighted.  It makes certain sense, e.g. it would allow you
> to spot near-misses, but that is only true for lines that neighbour
> real hits, so...
>
> I like this approach better in that it makes those who want a
> different behaviour to do the work without breaking the expectation
> of those who are used to the established behaviour.
>
> Zoltan?
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 01/19] dir.c: optionally compute sha-1 of a .gitignore file

[Duy Nguyen]

On Tue, Oct 28, 2014 at 5:46 AM, Junio C Hamano  wrote:
> Nguyễn Thái Ngọc Duy   writes:
>
>> This is not used anywhere yet. But the goal is to compare quickly if a
>> .gitignore file has changed when we have the SHA-1 of both old (cached
>> somewhere) and new (from index or a tree) versions.
>>
>> Signed-off-by: Nguyễn Thái Ngọc Duy 
>> ---
>>  dir.c | 50 +++---
>>  dir.h |  5 +
>>  2 files changed, 48 insertions(+), 7 deletions(-)
>>
>> diff --git a/dir.c b/dir.c
>> index bd274a7..33a35c1 100644
>> --- a/dir.c
>> +++ b/dir.c
>> @@ -466,7 +466,8 @@ void add_exclude(const char *string, const char *base,
>>   x->el = el;
>>  }
>>
>> -static void *read_skip_worktree_file_from_index(const char *path, size_t 
>> *size)
>> +static void *read_skip_worktree_file_from_index(const char *path, size_t 
>> *size,
>> + struct sha1_stat *ss)
>>  {
>>   int pos, len;
>>   unsigned long sz;
>> @@ -485,6 +486,10 @@ static void *read_skip_worktree_file_from_index(const 
>> char *path, size_t *size)
>>   return NULL;
>>   }
>>   *size = xsize_t(sz);
>> + if (ss) {
>> + memset(&ss->stat, 0, sizeof(ss->stat));
>> + hashcpy(ss->sha1, active_cache[pos]->sha1);
>> + }
>>   return data;
>>  }
>>
>> @@ -529,11 +534,18 @@ static void trim_trailing_spaces(char *buf)
>>   *last_space = '\0';
>>  }
>>
>> -int add_excludes_from_file_to_list(const char *fname,
>> -const char *base,
>> -int baselen,
>> -struct exclude_list *el,
>> -int check_index)
>> +/*
>> + * Given a file with name "fname", read it (either from disk, or from
>> + * the index if "check_index" is non-zero), parse it and store the
>> + * exclude rules in "el".
>> + *
>> + * If "ss" is not NULL, compute SHA-1 of the exclude file and fill
>> + * stat data from disk (only valid if add_excludes returns zero). If
>> + * ss_valid is non-zero, "ss" must contain good value as input.
>> + */
>
> Hmm, do we want a separate parameter for that?  Wouldn't it be
> sufficient and cleaner to pass is_null_sha1(ss->sha1[]),

Hm.. no. If both ss->sha1 and ss->stat are valid, then we could try to
match stat on disk and reuse ss->sha1, so we can't use a special value
of ss->sha1[] to mark the validity.

> or alternatively have an element ss->valid that is initialized to false?

Yeah..
-- 
Duy
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

differences between old clone and new Re: git-svn performance

[Hin-Tak Leung]

To compare the old clone with the new, I did:

git branch -r | sort | xargs -n 1 git log --decorate=full -n 1

It turned out other than the empty vs 3 word commit messages
about two years ago on trunk (which are inherited in all the newer
branches), there are two other groups of differences.

One branch on the old clone has an extra merge from trunk (
and some extra trunk commits) listed in 'git log', while
another branch has the exact opposite - on the old clone
has one fewer merge.

I see the merge seem to be genuine - the subversion log
often says so e.g. "ported from rXXX from trunk", but
the extra/missing pattern isn't consistent.

So the histories are largely the same, except due to the 
extra merge, don't have the same sha1 sums.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] merge & sequencer: turn "Conflicts:" hint into a comment

[Christian Couder]

On Mon, Oct 27, 2014 at 6:32 PM, Junio C Hamano  wrote:
> Jeff King  writes:
>
>> If that is the only casualty, I think it is probably a net-win. We may
>> want better tooling around viewing the merge later, but that can wait
>> until somebody steps up with a real use case (because even that conflict
>> list may not be completely what they want; they may also want the list
>> of files that were auto-merged successfully, for example).
>
> Yup.
>
> Also Christian's "trailer" series may want to learn the same trick
> we did to builtin/commit.c in this series, if it does not already
> know about possible trailing comment and blank lines.

The trailer series already tries to ignore comments and blank lines.
This is the relevant function:

/*
 * Return the (0 based) index of the first trailer line or count if
 * there are no trailers. Trailers are searched only in the lines from
 * index (count - 1) down to index 0.
 */
static int find_trailer_start(struct strbuf **lines, int count)
{
int start, only_spaces = 1;

/*
 * Get the start of the trailers by looking starting from the end
 * for a line with only spaces before lines with one separator.
 */
for (start = count - 1; start >= 0; start--) {
if (lines[start]->buf[0] == comment_line_char)
continue;
if (contains_only_spaces(lines[start]->buf)) {
if (only_spaces)
continue;
return start + 1;
}
if (strcspn(lines[start]->buf, separators) < lines[start]->len) {
if (only_spaces)
only_spaces = 0;
continue;
}
return count;
}

return only_spaces ? count : 0;
}

But I am not sure sure that it does all of what you do to
builtin/commit.c in the above patch. I will have a look.
Anyway I would be happy to use an existing function or to refactor
some existing code into a shared function if possible.

Thanks,
Christian.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html