Re: [fossil-users] Git just got shallow and sparse clones

[Steve Stefanovich]

Just‎ to contribute my 2 cents to this discussion, for our workflow I'd find 
sparse clones much more useful than shallow ones, although I can see how deep 
history can slow initial clone, and that past certain point is rarely useful.

It's not our problem - but we have thousands of separate source files in 
modules that are directory subtrees. Yes, we split those in different Fossil 
repos, but this approach introduces dependencies and repo management overheads. 
It's easier to have a single encompassing repo than 30 separate ones, 
especially if documentation and tickets are involved. 
‎
There is also IP protection aspect. Being able to clone a certain directory 
only from the source tree would allow us to restrict the third party developers 
to see just what they are supposed to see and are allowed to work on. From that 
perspective sparse clones would have to be 'special' in a sense that you can't 
sync fully on demand afterwards. This probably means special privilege with 
list of directory globs an user is allowed to clone sparsely. 
‎
I am not quite sure how would it all work in Fossil though. The sparse clone 
from a dir would have to fetch all the check-ins that contain changes to 
affected files, but drop the other artifacts outside the dir. I suppose we 
could have missing artifact warnings on update/status/diff (similar to current 
missing common ancestor warnings in certain merge scenarios), but it will be 
nice if Fossil is aware that repo is a special sparsely cloned one, and 
interpret those missing artifacts by showing (or hiding, if preferred) 
appropriate warnings to avoid confusion. 

Finally, there is a question of wikis and tickets. Following this model sparse 
clone would not fetch them, but perhaps it is more useful if this is governed 
by existing user wiki/ticket privileges. 
‎
S. ‎


  Original Message  
From: Warren Young
Sent: Tuesday, 7 February 2017 10:58
To: Fossil SCM user's discussion
Reply To: Fossil SCM user's discussion
Subject: Re: [fossil-users] Git just got shallow and sparse clones

On Feb 6, 2017, at 4:49 PM, Warren Young <war...@etr-usa.com> wrote:
> 
>> allow cloning a repository “shallowly"
> 
> That’s not what I mean by “shallow,” since according to the docs,[1]

Oh, I see the --no-single-branch bit now. Sigh, another confusing Git default.

Even so, it still doesn’t quite do what I want:

- I want to give --depth in days, not checkins.

- I want to allow the cloned-from repository to set a sensible default, so that 
very large repos can require that cloners specify something like --deep to get 
everything.

Personally, I don’t need this for my own repositories. I’m just putting out 
ideas to handle extremely large repos.

As for my second point above, I’d have been happy if the Linux kernel source 
repo I checked out onto a Raspberry Pi had only given me the past 30 days or 
so, without me having to know about --depth. I only wanted --depth=1 anyway.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Git just got shallow and sparse clones

[Warren Young]

On Feb 6, 2017, at 4:49 PM, Warren Young  wrote:
> 
>>   allow cloning a repository “shallowly"
> 
> That’s not what I mean by “shallow,” since according to the docs,[1]

Oh, I see the --no-single-branch bit now.  Sigh, another confusing Git default.

Even so, it still doesn’t quite do what I want:

- I want to give --depth in days, not checkins.

- I want to allow the cloned-from repository to set a sensible default, so that 
very large repos can require that cloners specify something like --deep to get 
everything.

Personally, I don’t need this for my own repositories.  I’m just putting out 
ideas to handle extremely large repos.

As for my second point above, I’d have been happy if the Linux kernel source 
repo I checked out onto a Raspberry Pi had only given me the past 30 days or 
so, without me having to know about --depth.  I only wanted --depth=1 anyway.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Git just got shallow and sparse clones

[Warren Young]

On Feb 4, 2017, at 6:07 AM, Konstantin Khomoutov 
 wrote:
> 
> commit 016e6ccbe03438454777e43dd73d67844296a3fd
> Author: Johannes Schindelin 
> Date:   Mon Oct 30 20:09:29 2006 +0100
> 
>allow cloning a repository “shallowly"

That’s not what I mean by “shallow,” since according to the docs,[1] it only 
works for a single branch.  It’s also not clear to me that there’s a way to 
pull more history at need with this mechanism.  If I’m online, and I step back 
one checkin on a branch past the files I have locally, will it go pull what it 
needs transparently from the repo I originally cloned from?

To me, a shallow clone says “give me the most recent history at the tip of all 
open branches, and fetch more at need.”  This Git VFS sounds like an extreme 
form of that, where the initial clone depth is 0.  I’d prefer something that 
copies more like 30 days to my local machine on the initial checkout, so that I 
could immediately go off-line and have an excellent chance of still having 
everything I needed.

All of this is is an effort to avoid sending the deep history of a repository — 
at least on initial clone — which most of the time you never need.

A secondary use of DVCSes is widespread total-repository replication, but for 
that you can trickle the history in over the course of days, if need be.  For 
day-to-day use, you rarely need deep history.


[1]: https://www.git-scm.com/docs/git-clone#git-clone---depthltdepthgt

> commit ed5336a7541e19b267de53afc8d15cffdbde8286
> Author: Nguyễn Thái Ngọc Duy 
> Date:   Thu Aug 20 20:47:05 2009 +0700
> 
>Introduce "sparse checkout”

According to Stack Overflow,[2] you still get all of the remote artifacts in 
your local repo clone when you do this, subject of course to --depth and such.

What I mean by “narrow” is that it only clones a particular subdirectory, or 
some other pattern match, as Subversion allowed.

For example, if I have three top-level directories in my repository, A, B, and 
C, I’d like to be able to say:

$ fossil clone --narrow B https://my-project.org B.fossil

which Fossil would realize means I want only B and its children.

(We can argue separately over the syntax.)

A nice side effect is that this would let you “shard” a large repository, since 
the local B.fossil clone could be re-shared separately.  I have multiple cases 
here where I put multiple projects in a single repository, because Subversion 
encouraged that, and now I want them broken up into separate repositories, 
because Fossil likes that better.

If the A, B, and C sub-trees are the same size, I expect this narrow clone to 
take about 1/3 the time as a full clone, and take about 1/3 the disk space.


[2]: http://stackoverflow.com/q/2336580
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Git just got shallow and sparse clones

[Martin S. Weber]

On 2017-02-04 15:01:43, Jan Danielsson wrote:
> (...)
>So when you do your initial clone, it only downloads some metadata
> about what files exist in the repository, but not the contents of them.
> When you open a file, the vfs will fetch the file for you.
> 
>Say a project has a doc/ directory which is unrelated to the build
> process, and there are 200MB worth of documents in there.  A person who
> just wants to build the project probably doesn't want to download all
> those docs, so they just kick off the build without touching the files
> in doc/.
> 
>Like I said, I only skimmed a few seconds so I could be way off, but
> that's what it looked like to me.
> 
>If it is what I think it is:  With all these indexing tools out
> there, scanning every file it finds, I wonder how useful this feature
> really is?

It's clearcase all over again. I don't remember the specifics (it's
nearly 20 years ago I was working on a clearcase VFS checkout under
solaris), but we (clearcase users) preferred the explicit nature
of CVS back then. IIRC, shared access to the VFS was leading to all
sorts of kinky side-effects.

Personally, I'd neither use a git, nor a fossil virtual file system,
much less when it requires an up to date Windows 10 installation
and a MS git binary.

Regards,
-Martin
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Git just got shallow and sparse clones

[Jan Danielsson]

On 2017-02-04 14:07, Konstantin Khomoutov wrote:
> On Fri, 3 Feb 2017 09:00:54 -0700
> Warren Young  wrote:
> 
>> https://developers.slashdot.org/story/17/02/03/1427213/microsoft-introduces-gvfs-git-virtual-file-system
> 
> Care to elaborate a bit?
[---]

   Granted, I just skimmed a few seconds, but I got the feeling that the
new feature makes it look like it has a bunch of stuff checked out when
it really hasn't.  And once you try to access files which haven't been
fetched before, it fetches them for you.

   So when you do your initial clone, it only downloads some metadata
about what files exist in the repository, but not the contents of them.
When you open a file, the vfs will fetch the file for you.

   Say a project has a doc/ directory which is unrelated to the build
process, and there are 200MB worth of documents in there.  A person who
just wants to build the project probably doesn't want to download all
those docs, so they just kick off the build without touching the files
in doc/.

   Like I said, I only skimmed a few seconds so I could be way off, but
that's what it looked like to me.

   If it is what I think it is:  With all these indexing tools out
there, scanning every file it finds, I wonder how useful this feature
really is?

-- 
Kind Regards,
Jan
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Git just got shallow and sparse clones

[Konstantin Khomoutov]

On Fri, 3 Feb 2017 09:00:54 -0700
Warren Young  wrote:

> https://developers.slashdot.org/story/17/02/03/1427213/microsoft-introduces-gvfs-git-virtual-file-system

Care to elaborate a bit?

commit 016e6ccbe03438454777e43dd73d67844296a3fd
Author: Johannes Schindelin 
Date:   Mon Oct 30 20:09:29 2006 +0100

allow cloning a repository "shallowly"

Available since v1.5.

commit ed5336a7541e19b267de53afc8d15cffdbde8286
Author: Nguyễn Thái Ngọc Duy 
Date:   Thu Aug 20 20:47:05 2009 +0700

Introduce "sparse checkout"

Available since v1.7.

The current stable Git version is 2.11.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

[fossil-users] Git just got shallow and sparse clones

[Warren Young]

https://developers.slashdot.org/story/17/02/03/1427213/microsoft-introduces-gvfs-git-virtual-file-system
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users