Re: [git-users] How to improve the Git status performance

[Philip Oakley]

- Original Message - 
  From: Stephen Morton 
  To: Git for human beings 
  Cc: philipoak...@iee.org 
  Sent: Wednesday, February 01, 2017 8:28 PM
  Subject: Re: [git-users] How to improve the Git status performance


  On Wednesday, 1 February 2017 00:42:51 UTC-5, Ravalika wrote:
Thank you Philip and Mark.


We are using Linux servers. We can't have local disk, as it is corporate 
setup. 
We have our project shares on build servers across LAN only


I have tried sparse checkout and other options but not much improvement on 
git status. 
git status -uno
git update-index --untracked-cache and git update-index --split-index




  Sounds to me like the issue is "git status performance when repo is on a 
network drive".
  What you will need to do is this
- Find out what sort of network mounting you have
- Find out what sort of caching you have, and what is available to you.
- You started to do some profiling on 'git status' performance that seems 
to indicate that caching plays a big part. You could confirm this by observing 
how subsequent 'git status' commands are faster, but that when you drop the 
cache, performance goes back down to the original slow time.
- Talk to your IT people to find out if anything can be sped up.
- Do some research on git's performance in this situation


  My gut feeling is that you are basically out of luck. Git does a lot of 
filesystem calls and I suspect that performance on a repo with lots and lots of 
files is going to be extra-slow when network mounted. (By contrast, my huge 
corporate repo on a locally-mounted SSD on linux can do a 'git status' in 
<0.1s.)



  Steve
Also, talk with your IT to determine if you can get some local machine space to 
host your local copy (then just use push to the server repo for one shot 
updates and 'backup'). Explain that it's the way Git was designed - to allow 
local working when off network. Failing that, use it as a lever to improve the 
network performance.

Philip

-- 
You received this message because you are subscribed to the Google Groups "Git 
for human beings" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to git-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [git-users] How to improve the Git status performance

[Stephen Morton]

On Wednesday, 1 February 2017 00:42:51 UTC-5, Ravalika wrote:
>
> Thank you Philip and Mark.
>
> We are using Linux servers. We can't have local disk, as it is corporate 
> setup. 
> We have our project shares on build servers across LAN only
>
> I have tried sparse checkout and other options but not much improvement on 
> git status. 
> git status -uno
> git update-index --untracked-cache and git update-index --split-index
>


Sounds to me like the issue is "git status performance when repo is on a 
network drive".
What you will need to do is this
  - Find out what sort of network mounting you have
  - Find out what sort of caching you have, and what is available to you.
  - You started to do some profiling on 'git status' performance that seems 
to indicate that caching plays a big part. You could confirm this by 
observing how subsequent 'git status' commands are faster, but that when 
you drop the cache, performance goes back down to the original slow time.
  - Talk to your IT people to find out if anything can be sped up.
  - Do some research on git's performance in this situation

My gut feeling is that you are basically out of luck. Git does a *lot* of 
filesystem calls and I suspect that performance on a repo with lots and 
lots of files is going to be extra-slow when network mounted. (By contrast, 
my huge corporate repo on a locally-mounted SSD on linux can do a 'git 
status' in <0.1s.)

Steve
 
 

-- 
You received this message because you are subscribed to the Google Groups "Git 
for human beings" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to git-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [git-users] How to improve the Git status performance

[Ravalika]

Thank you Philip and Mark.

We are using Linux servers. We can't have local disk, as it is corporate 
setup. 
We have our project shares on build servers across LAN only

I have tried sparse checkout and other options but not much improvement on 
git status. 
git status -uno
git update-index --untracked-cache and git update-index --split-index


Thanks
Ravalika


On Sunday, November 27, 2016 at 9:23:35 PM UTC+5:30, Philip Oakley wrote:
>
> Hi Mark, Ravalika,
>  
> We need to distinguish between URL style network connections (push, pull, 
> fetch, clone) and mapped drive local area network LAN style network 
> connections.
>  
> Mark is right that git status does not do any URL network style checks. 
> However, depending on Ravalika's local setup (which may be corporate), it 
> is quite possible for the local drive that the repository is held on to 
> actually be across the LAN - I often have that at my work where there are 
> lots of project shares, and one is expected to hold project data (even 
> personal workings) on those project drives. 
>  
> Obviously(?) the network is then setup to respond quickly most of the 
> time, though occassional I see slow downs if others are saturating a (my) 
> network switch with some large file transfer, such that my file transfer 
> goes slowly. Some corporate setups are not as reponsive...
>  
> The other aspect is the number of files checked out that need to be stat'd 
> and the total size of the worktree. There are philiosophical differences 
> between Linux and Windows regarding what parts of the stat process should 
> trusted meaning that Windows is generally slower, so for a large number of 
> files, all stored via a network share, a process delay is quite possible.
>  
> In addition to Mark's advice, it is worth determining how the repository 
> storage is organised with regard to internal vs network share storage. (and 
> OS / Git versions)
>  
> Philip
>
> - Original Message - 
> *From:* Mark Waite  
> *To:* Git for human beings  
> *Sent:* Saturday, November 26, 2016 1:48 PM
> *Subject:* Re: [git-users] How to improve the Git status performance
>
> Based on the description in the git-status manual page 
> <https://git-scm.com/docs/git-status>, I don't think that it is 
> performing any network operations.  Changing the location of the server 
> should not affect the performance of the "git status" command. 
>
> The manual page notes that if you have a repository which includes many 
> files, you can improve the performance of "git status" by not reporting 
> untracked files ("git status -uno" or "git status --untracked-files=no"). 
>  The specific quote is
>
> Because it takes extra work to find untracked files in the filesystem, 
>> this mode may take some time in a large working tree. Consider enabling 
>> untracked cache and split index if supported (see git update-index 
>> --untracked-cache and git update-index --split-index), Otherwise you can 
>> use no to have git status return more quickly without showing untracked 
>> files.
>
>
> Other techniques which can improve the performance of "git status" include:
>
>- Use a local file system on an SSD disc drive for the working 
>directory rather than a rotating disc 
>- Use a local file system for the working directory rather than a 
>network mounted file system
>- Reduce the number of files in the working directory by using a 
>sparse checkout to only checkout the files which are relevant to your work
>
> Mark Waite
>
> On Tuesday, November 22, 2016 at 10:36:29 PM UTC-7, Ravalika wrote: 
>>
>> Thank you 
>>
>> I have tried setting server on the same machine still the git status 
>> takes more time 
>>
>> real0m25.969s
>> user0m10.293s
>> sys 0m5.270s
>>
>> And tried the below flags also
>> ignoreStat = true
>> fscache = true
>> preloadindex = true
>> untrackedCache = true
>>
>> Still the git status takes more time ?
>>
>> On Thursday, November 17, 2016 at 1:27:43 PM UTC+5:30, Philip Oakley 
>> wrote: 
>>>
>>> Git is a DVCS - the first D is the key feature of it's performance. 
>>>  
>>> Do have a rethink of the use of a centralised network file server for 
>>> the user repository. The network delays are killing you.
>>>  
>>> Step one: Decentralise.
>>> Step two: Distribute control.
>>> Step three: fun and profit (and speed!)
>>>  
>>> Old habits die hard.. Try and get away from them
>>>  
>>> Philip
>>>
>>&g

Re: [git-users] How to improve the Git status performance

[Philip Oakley]

Hi Mark, Ravalika,

We need to distinguish between URL style network connections (push, pull, 
fetch, clone) and mapped drive local area network LAN style network connections.

Mark is right that git status does not do any URL network style checks. 
However, depending on Ravalika's local setup (which may be corporate), it is 
quite possible for the local drive that the repository is held on to actually 
be across the LAN - I often have that at my work where there are lots of 
project shares, and one is expected to hold project data (even personal 
workings) on those project drives. 

Obviously(?) the network is then setup to respond quickly most of the time, 
though occassional I see slow downs if others are saturating a (my) network 
switch with some large file transfer, such that my file transfer goes slowly. 
Some corporate setups are not as reponsive...

The other aspect is the number of files checked out that need to be stat'd and 
the total size of the worktree. There are philiosophical differences between 
Linux and Windows regarding what parts of the stat process should trusted 
meaning that Windows is generally slower, so for a large number of files, all 
stored via a network share, a process delay is quite possible.

In addition to Mark's advice, it is worth determining how the repository 
storage is organised with regard to internal vs network share storage. (and OS 
/ Git versions)

Philip
  - Original Message - 
  From: Mark Waite 
  To: Git for human beings 
  Sent: Saturday, November 26, 2016 1:48 PM
  Subject: Re: [git-users] How to improve the Git status performance


  Based on the description in the git-status manual page, I don't think that it 
is performing any network operations.  Changing the location of the server 
should not affect the performance of the "git status" command.


  The manual page notes that if you have a repository which includes many 
files, you can improve the performance of "git status" by not reporting 
untracked files ("git status -uno" or "git status --untracked-files=no").  The 
specific quote is


Because it takes extra work to find untracked files in the filesystem, this 
mode may take some time in a large working tree. Consider enabling untracked 
cache and split index if supported (see git update-index --untracked-cache and 
git update-index --split-index), Otherwise you can use no to have git status 
return more quickly without showing untracked files.


  Other techniques which can improve the performance of "git status" include:
a.. Use a local file system on an SSD disc drive for the working directory 
rather than a rotating disc
b.. Use a local file system for the working directory rather than a network 
mounted file system

c.. Reduce the number of files in the working directory by using a sparse 
checkout to only checkout the files which are relevant to your work
  Mark Waite

  On Tuesday, November 22, 2016 at 10:36:29 PM UTC-7, Ravalika wrote:
Thank you 


I have tried setting server on the same machine still the git status takes 
more time


real0m25.969s
user0m10.293s
sys 0m5.270s


And tried the below flags also
ignoreStat = true
fscache = true
preloadindex = true
untrackedCache = true


Still the git status takes more time ?

On Thursday, November 17, 2016 at 1:27:43 PM UTC+5:30, Philip Oakley wrote:
  Git is a DVCS - the first D is the key feature of it's performance. 

  Do have a rethink of the use of a centralised network file server for the 
user repository. The network delays are killing you.

  Step one: Decentralise.
  Step two: Distribute control.
  Step three: fun and profit (and speed!)

  Old habits die hard.. Try and get away from them

  Philip
- Original Message - 
From: Ravalika 
To: Git for human beings 
Sent: Thursday, November 17, 2016 5:56 AM
Subject: [git-users] How to improve the Git status performance


Hi All, 


We are using git-2.10.2 version for version control.

It is an centralized server and git status takes too long


How to improve the performance of git status



Git repo details:



Size of the .git folder is 8.9MB
Number of commits approx 53838  (git rev-list HEAD --count)
Number of branches -  330  
Number of files - 63883
Working tree clone size is 4.3GB


time git status shows
real 0m23.673s

user 0m9.432s
sys 0m3.793s


then after 5 mins
real0m4.864s
user0m1.417s
sys 0m4.710s


And I have experimented following ways but no significant change 

-  -  Setting core.ignorestat to true

-  -  Git gc &git clean

-  -  Shallow clone – Reducing number of commits

  

Re: [git-users] How to improve the Git status performance

[Mark Waite]

Based on the description in the git-status manual page 
, I don't think that it is performing 
any network operations.  Changing the location of the server should not 
affect the performance of the "git status" command.

The manual page notes that if you have a repository which includes many 
files, you can improve the performance of "git status" by not reporting 
untracked files ("git status -uno" or "git status --untracked-files=no"). 
 The specific quote is

Because it takes extra work to find untracked files in the filesystem, this 
> mode may take some time in a large working tree. Consider enabling 
> untracked cache and split index if supported (see git update-index 
> --untracked-cache and git update-index --split-index), Otherwise you can 
> use no to have git status return more quickly without showing untracked 
> files.


Other techniques which can improve the performance of "git status" include:

   - Use a local file system on an SSD disc drive for the working directory 
   rather than a rotating disc
   - Use a local file system for the working directory rather than a 
   network mounted file system
   - Reduce the number of files in the working directory by using a sparse 
   checkout to only checkout the files which are relevant to your work

Mark Waite

On Tuesday, November 22, 2016 at 10:36:29 PM UTC-7, Ravalika wrote:
>
> Thank you 
>
> I have tried setting server on the same machine still the git status takes 
> more time
>
> real0m25.969s
> user0m10.293s
> sys 0m5.270s
>
> And tried the below flags also
> ignoreStat = true
> fscache = true
> preloadindex = true
> untrackedCache = true
>
> Still the git status takes more time ?
>
> On Thursday, November 17, 2016 at 1:27:43 PM UTC+5:30, Philip Oakley wrote:
>>
>> Git is a DVCS - the first D is the key feature of it's performance. 
>>  
>> Do have a rethink of the use of a centralised network file server for the 
>> user repository. The network delays are killing you.
>>  
>> Step one: Decentralise.
>> Step two: Distribute control.
>> Step three: fun and profit (and speed!)
>>  
>> Old habits die hard.. Try and get away from them
>>  
>> Philip
>>
>> - Original Message - 
>> *From:* Ravalika 
>> *To:* Git for human beings 
>> *Sent:* Thursday, November 17, 2016 5:56 AM
>> *Subject:* [git-users] How to improve the Git status performance
>>
>> Hi All, 
>>
>> We are using git-2.10.2 version for version control.
>> It is an centralized server and git status takes too long
>>
>> How to improve the performance of git status
>>
>> Git repo details:
>>
>> Size of the .git folder is 8.9MB
>> Number of commits approx 53838  (git rev-list HEAD --count)
>> Number of branches -  330  
>> Number of files - 63883
>> Working tree clone size is 4.3GB
>>
>> time git status shows
>> real 0m23.673s
>> user 0m9.432s
>> sys 0m3.793s
>>
>> then after 5 mins
>> real0m4.864s
>> user0m1.417s
>> sys 0m4.710s
>>
>> And I have experimented following ways but no significant change 
>>
>> -  -  Setting core.ignorestat to true
>>
>> -  -  Git gc &git clean
>>
>> -  -  Shallow clone – Reducing number of commits
>>
>> -  -  Clone only one branch 
>>
>>   - Git repacking - git repack -ad && git prune
>>
>> - - Cold/warm cache 
>>
>> Could you please let me know, what are the other ways to improve the git 
>> performance ?
>>
>> Thank you,
>> Renuka
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Git for human beings" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to git-users+...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups "Git 
for human beings" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to git-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [git-users] How to improve the Git status performance

[Ravalika]

Thank you 

I have tried setting server on the same machine still the git status takes 
more time

real0m25.969s
user0m10.293s
sys 0m5.270s

And tried the below flags also
ignoreStat = true
fscache = true
preloadindex = true
untrackedCache = true

Still the git status takes more time ?

On Thursday, November 17, 2016 at 1:27:43 PM UTC+5:30, Philip Oakley wrote:
>
> Git is a DVCS - the first D is the key feature of it's performance. 
>  
> Do have a rethink of the use of a centralised network file server for the 
> user repository. The network delays are killing you.
>  
> Step one: Decentralise.
> Step two: Distribute control.
> Step three: fun and profit (and speed!)
>  
> Old habits die hard.. Try and get away from them
>  
> Philip
>
> - Original Message - 
> *From:* Ravalika  
> *To:* Git for human beings  
> *Sent:* Thursday, November 17, 2016 5:56 AM
> *Subject:* [git-users] How to improve the Git status performance
>
> Hi All, 
>
> We are using git-2.10.2 version for version control.
> It is an centralized server and git status takes too long
>
> How to improve the performance of git status
>
> Git repo details:
>
> Size of the .git folder is 8.9MB
> Number of commits approx 53838  (git rev-list HEAD --count)
> Number of branches -  330  
> Number of files - 63883
> Working tree clone size is 4.3GB
>
> time git status shows
> real 0m23.673s
> user 0m9.432s
> sys 0m3.793s
>
> then after 5 mins
> real0m4.864s
> user0m1.417s
> sys 0m4.710s
>
> And I have experimented following ways but no significant change 
>
> -  -  Setting core.ignorestat to true
>
> -  -  Git gc &git clean
>
> -  -  Shallow clone – Reducing number of commits
>
> -  -  Clone only one branch 
>
>   - Git repacking - git repack -ad && git prune
>
> - - Cold/warm cache 
>
> Could you please let me know, what are the other ways to improve the git 
> performance ?
>
> Thank you,
> Renuka
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Git for human beings" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to git-users+...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups "Git 
for human beings" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to git-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [git-users] How to improve the Git status performance

[Philip Oakley]

Git is a DVCS - the first D is the key feature of it's performance. 

Do have a rethink of the use of a centralised network file server for the user 
repository. The network delays are killing you.

Step one: Decentralise.
Step two: Distribute control.
Step three: fun and profit (and speed!)

Old habits die hard.. Try and get away from them

Philip
  - Original Message - 
  From: Ravalika 
  To: Git for human beings 
  Sent: Thursday, November 17, 2016 5:56 AM
  Subject: [git-users] How to improve the Git status performance


  Hi All,


  We are using git-2.10.2 version for version control.

  It is an centralized server and git status takes too long


  How to improve the performance of git status



  Git repo details:



  Size of the .git folder is 8.9MB
  Number of commits approx 53838  (git rev-list HEAD --count)
  Number of branches -  330  
  Number of files - 63883
  Working tree clone size is 4.3GB


  time git status shows
  real 0m23.673s

  user 0m9.432s
  sys 0m3.793s


  then after 5 mins
  real0m4.864s
  user0m1.417s
  sys 0m4.710s


  And I have experimented following ways but no significant change 

  -  -  Setting core.ignorestat to true

  -  -  Git gc &git clean

  -  -  Shallow clone – Reducing number of commits

  -  -  Clone only one branch 

- Git repacking - git repack -ad && git prune

  - - Cold/warm cache 



  Could you please let me know, what are the other ways to improve the git 
performance ?



  Thank you,
  Renuka



  -- 
  You received this message because you are subscribed to the Google Groups 
"Git for human beings" group.
  To unsubscribe from this group and stop receiving emails from it, send an 
email to git-users+unsubscr...@googlegroups.com.
  For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "Git 
for human beings" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to git-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.