Re: Checkpointing

[Thomas Singer]

Imagine working on one larger feature (or even multiple features). You 
already have created a couple of local commits, but are not yet 
finished. Following use cases come to my mind:


1) some other developer commits new revisions
- it should be possible to continue working on your feature-queue 
keeping it based on the old revision, but sooner or later it comes the 
time to integrate the new revision(s)
- this means to create a new queue on the new revision and cherry-pick 
all local commits of the old queue onto this new revision (could this be 
done with purely local information or would each cherry-pick require a 
possibly slow communication with the server?)
- with each cherry-pick, a conflict might occur - aborting should abort 
the creation of the new queue and application of the old queue, 
resulting in the deletion of the partly finished new queue keeping the 
old one

- each queue should be rebased independently onto the new revisions

2) you have to make a quick-fix revision
- you need to switch from your local feature-queue to the latest 
revision and fix the bug a coworker requests to get fixed
- you may now switch back to your queue to proceed with the work, but 
there is a new revision now, so you should base your work sooner or 
later on that revision


3) you want to rework your local commits, e.g. change order, squash some 
commits, change the message

- create a new queue based on the same revision
- create new local commits by cherry-picking commits from the other 
(old) queue, maybe amending some local commits
- after the new queue is ready (verify to diff it with the old queue) 
the old queue can be deleted
- often enough I find it useful to be able to make one of my first 
feature commits public, so the x first local commits should be possible 
to be become revisions => the queue becomes shorter and based on the new 
revision(s)


4) you need to fork an existing queue at any local commit
- switching to one of the local commits of your current queue you detect 
that it contains a flaw
- creating now a local commit to fix would mean to first create a new 
queue based on the same revision, apply all previous local commits (no 
conflict risk) - the old queue would be kept



It might be useful to be able to "store" somehow one or another queue on 
the server, e.g. for backup reasons, so no change is lost if my hard 
disk crashes or my code fails and cleans the disk. (Creating a new, real 
feature branch with revisions in the repository I don't like because 
then they would be cut in stone and this would force me to create nice 
and clean commits. But because we are not without error, such code will 
contain back and forth changes and hence hard to read.)


As long as I'm working on a non-trivial feature/refactoring, I prefer to 
have complete control over my commits, I even like to commit completely 
unstable and incomplete code with Git - because I have the possibility 
to clean up later.


--
Best regards,
Thomas Singer



On 02/07/2019 18:32, Nathan Hartman wrote:

On Sun, Jun 30, 2019 at 11:22 AM Thomas Singer 
wrote:


With "rebasing" I mean, that such list of "local commits" needs to be
re-applied (on demand, not automatically) onto a different revision.
Something like a continues series of cherry-picking (with the
possibility to get a conflict in each step; and a possibility to
continue after conflict resolution or abort). This means to me, that at
least cherry-picking needs to be possible from a revision or a "local
commit".



Could you describe how you would like to use this capability? E.g.,
give an example use case?

Re: Checkpointing

[Nathan Hartman]

On Sun, Jun 30, 2019 at 11:22 AM Thomas Singer 
wrote:

> With "rebasing" I mean, that such list of "local commits" needs to be
> re-applied (on demand, not automatically) onto a different revision.
> Something like a continues series of cherry-picking (with the
> possibility to get a conflict in each step; and a possibility to
> continue after conflict resolution or abort). This means to me, that at
> least cherry-picking needs to be possible from a revision or a "local
> commit".
>

Could you describe how you would like to use this capability? E.g.,
give an example use case?

Re: Checkpointing

[Branko Čibej]

On 30.06.2019 16:56, Nathan Hartman wrote:
> On Sat, Jun 29, 2019 at 7:21 AM Branko Čibej  > wrote:
>
> As Mark explained, it will do none of the above unless someone
> steps up
> and writes the code.
>
> For reference, what Nathan described was discussed here on the
> list and
> in person during hackathons years ago, yet nothing happened until
> Julian
> started writing code (and even then, what Julian is doing is a limited
> subset of the "ideal").
>
> If there's no interest amongst people to take the time to write
> the code
> ... well, we can all tell tall stories about the future, but that
> won't
> change it one bit.
>
>
> I know.
>
> We all know.
>
> I understand the frustration I see here.
>
> I understand that you've seen these wonderful discussions time and
> again and then nothing happened. And you've seen it so many times that
> you've become inoculated to the idea that it could change.
>
> But it will change, because:
>
> There was a wise man named Albert Einstein, and I have no idea if he
> actually said this or not but he's widely credited with saying that
> the definition of insanity is to do the same thing over and over and
> expect different results.
>
> Telling the closed dev@ community that we need new developers didn't
> work until now and I don't expect it to start working miraculously.
> I'm sorry to use an expletive -- marketing -- but that's what we need
> with the outside world. No business can sustain itself without telling
> the world what it's all about and there's no reason to believe that an
> open source project is any different. There's still a profit motive.
> In a business, it's profit in money; in an open source project, it's
> profit in mindshare and participation. So we need to get out there and
> drum up some new business, BUT:
>
> There's a bit of a chicken and egg problem here. If we entice new
> potential devs to join dev@ and they come here and see discussions of
> decline, defeat, and despair,


Who's despairing? Stating facts is neither decline, defeat, nor despair. :)

I would actually love to see all of you who are so enthusiastic in this
thread to come aboard. Especially as one of the things the developers
here were traditionally lacking is real-world user experience on
non-trivial projects (I count Subversion itself as "trivial" in terms of
the complexity of our version control workflow).


> they'll get turned off and go somewhere
> else. People want to be part of something successful! We need those
> who join to see discussions of all the cool things Subversion WILL do.
>
> Of course it won't do any of it until after the code gets written. For
> the code to get written we need devs. To get devs we need to change
> our thinking from despair to planning for a great future. So let's have
> some positive discussions over here!
>
> I'm going to search for those old discussions -- and the ones about
> what the command line syntax should be like -- and I'll be back later
> with some concrete thoughts.


Perfect.

-- Brane

Re: Checkpointing

[Thomas Singer]

Hi Nathan,

Thanks for your detailed answers.


- will it support "rebasing" such a local history onto the latest
updated commit?



It will have to support "rebasing" which is what "svn update" already
does today. Otherwise you couldn't commit your work!


With "rebasing" I mean, that such list of "local commits" needs to be 
re-applied (on demand, not automatically) onto a different revision. 
Something like a continues series of cherry-picking (with the 
possibility to get a conflict in each step; and a possibility to 
continue after conflict resolution or abort). This means to me, that at 
least cherry-picking needs to be possible from a revision or a "local 
commit".


Tom

Re: Checkpointing

[Nathan Hartman]

On Sat, Jun 29, 2019 at 7:21 AM Branko Čibej  wrote:

> As Mark explained, it will do none of the above unless someone steps up
> and writes the code.
>
> For reference, what Nathan described was discussed here on the list and
> in person during hackathons years ago, yet nothing happened until Julian
> started writing code (and even then, what Julian is doing is a limited
> subset of the "ideal").
>
> If there's no interest amongst people to take the time to write the code
> ... well, we can all tell tall stories about the future, but that won't
> change it one bit.
>

I know.

We all know.

I understand the frustration I see here.

I understand that you've seen these wonderful discussions time and
again and then nothing happened. And you've seen it so many times that
you've become inoculated to the idea that it could change.

But it will change, because:

There was a wise man named Albert Einstein, and I have no idea if he
actually said this or not but he's widely credited with saying that
the definition of insanity is to do the same thing over and over and
expect different results.

Telling the closed dev@ community that we need new developers didn't
work until now and I don't expect it to start working miraculously.
I'm sorry to use an expletive -- marketing -- but that's what we need
with the outside world. No business can sustain itself without telling
the world what it's all about and there's no reason to believe that an
open source project is any different. There's still a profit motive.
In a business, it's profit in money; in an open source project, it's
profit in mindshare and participation. So we need to get out there and
drum up some new business, BUT:

There's a bit of a chicken and egg problem here. If we entice new
potential devs to join dev@ and they come here and see discussions of
decline, defeat, and despair, they'll get turned off and go somewhere
else. People want to be part of something successful! We need those
who join to see discussions of all the cool things Subversion WILL do.

Of course it won't do any of it until after the code gets written. For
the code to get written we need devs. To get devs we need to change
our thinking from despair to planning for a great future. So let's have
some positive discussions over here!

I'm going to search for those old discussions -- and the ones about
what the command line syntax should be like -- and I'll be back later
with some concrete thoughts.

Re: Checkpointing

[Nathan Hartman]

On Fri, Jun 28, 2019 at 1:12 PM Thomas Singer 
wrote:

> - so the working copy can have checked out multiple commits or one
> checkpoint?
>

The working copy has always provided one "view." I say "view" for lack
of a better word but I mean the checked out directory of your data
that is under version control. Currently, that "view" can be changed
through updating to a different revision, switching to a different
branch, changing depth on a sparse/shallow checkout, etc; but it's
still one view. As I imagine it, checkpoints would add the ability to
update to a checkpoint, or revert to the last checkpoint.

This begs a few important questions, such as:

When someone does "svn revert" with no additional parameters, what are
we reverting to? Do we revert to BASE as we always have? Or to the
last checkpoint? That requires further study...


> - will it support multiple histories ("branches") planned, e.g. for
> different features?
>

Subversion already provides server-side branches. For local work,
earlier I described multiple "arrays" of checkpoints. We need a better
name than "array" but the idea is that a checkpoint is like a local
commit, and the array is a local linear history of such commits. And
you can have multiple arrays. You could say that each array is like a
local branch. But I would rather think of each array as being for a
certain subject. So, for example if I'm editing code to write a new
feature, I might have a checkpoint array called "feature" and a second
checkpoint array called "unrelated" just as an example. I'd work on my
feature, making checkpoints to the "feature" array as I go. Suppose
that while I'm working I come across typos in comments. I could fix
those typos and make a checkpoint of that un the "unrelated" array.
When I'm ready to commit, I'll commit the typo fixes separately from
the feature work. That would make it much easier to have one subject
per commit.

Anyway that's how I imagine it. If you have other thoughts, I'd love
to hear them!

- will it support "rebasing" such a local history onto the latest
> updated commit?


It will have to support "rebasing" which is what "svn update" already
does today. Otherwise you couldn't commit your work!

Re: Checkpointing

[Branko Čibej]

On 28.06.2019 19:12, Thomas Singer wrote:
> Hi Nathan,
>
> Thanks for your explanation. This sounds quite interesting. I have
> some questions though:
>
> - so the working copy can have checked out multiple commits or one
> checkpoint?
>
> - will it support multiple histories ("branches") planned, e.g. for
> different features?
>
> - will it support "rebasing" such a local history onto the latest
> updated commit?
>

As Mark explained, it will do none of the above unless someone steps up
and writes the code.

For reference, what Nathan described was discussed here on the list and
in person during hackathons years ago, yet nothing happened until Julian
started writing code (and even then, what Julian is doing is a limited
subset of the "ideal").

If there's no interest amongst people to take the time to write the code
... well, we can all tell tall stories about the future, but that won't
change it one bit.

-- Brane

Re: Checkpointing

[Thomas Singer]

Hi Nathan,

Thanks for your explanation. This sounds quite interesting. I have some 
questions though:


- so the working copy can have checked out multiple commits or one 
checkpoint?


- will it support multiple histories ("branches") planned, e.g. for 
different features?


- will it support "rebasing" such a local history onto the latest 
updated commit?


--
Best regards,
Thomas Singer

Re: Shelving / Checkpointing thoughts

[Branko Čibej]

On 06.06.2018 16:53, Paul Hammant wrote:
> [going back in time to August 29th]
>
>
>
> Apache products aren't allowed to depend on GPL'd code for mandatory /
> core features.
>
>
> This is academic now as Julian is making great progress with his
> shelve tech that's built in to Subversion and idiomatic, but there are
> MIT licensed versions of Git:
>
>     https://github.com/go-gitea/git/blob/master/LICENSE (MIT)
>
> You could link that in instead of LibGit2 (GPL) for storage and
> dissemination of patch sets.
>
> Sure, a Go run-time library is not as bytespace/ram/CPU efficient as C
> or Rust, but it's closer by a mile than Java or Python (for example).


The problem with any kind of git-based shelving solution is that it
doesn't admit the existence of directories. We'll want to support those
one day.

Also, no, we can't have core functionality written in Go or Rust; it has
to be C because that's what brings us cross-platform portability.

-- Brane

Re: Shelving / Checkpointing thoughts

[Paul Hammant]

[going back in time to August 29th]



> Apache products aren't allowed to depend on GPL'd code for mandatory /
> core features.
>
>
This is academic now as Julian is making great progress with his shelve
tech that's built in to Subversion and idiomatic, but there are MIT
licensed versions of Git:

https://github.com/go-gitea/git/blob/master/LICENSE (MIT)

You could link that in instead of LibGit2 (GPL) for storage and
dissemination of patch sets.

Sure, a Go run-time library is not as bytespace/ram/CPU efficient as C or
Rust, but it's closer by a mile than Java or Python (for example).

- Paul

Re: FYI - Shelving & Checkpointing featured in Assembla's Cornerstone 4

[Branko Čibej]

On 05.04.2018 21:25, Julian Foad wrote:
> Branko Čibej wrote:
>> On 05.04.2018 21:06, Julian Foad wrote:
>>> The Cornerstone 4 svn client for Mac ...
>>>
>>> https://cornerstone.assembla.com/
>>>
>>> ... with Shelving and Checkpointing called out as new features.
>>>
>>> It's exciting for me to see this new work going so quickly into
>>> production!
>>
>> What really "excites" me about that page is that:
>>
>>    * there's no indication that shelving is an experimental feature
>> of an
>>  as yet unreleased version;
>>    * it advertises checkpointing which, as far as I'm aware, we don't
>>  even have on trunk — so if this client does have it, the
>>  implementation is incompatible with any other client;
>>    * it does not even remotely attempt to use the Apache and Subversion
>>  trademarks correctly.
>>
>> I'd appreciate it, Julian, if you brought these issues to the attention
>> of whoever is responsible for this.
>
> Thanks for your quick review and response, Brane. I will do.


Thanks! And sorry if I came over a bit terse and grumpy. Good job on
getting shelving out into the wild!

-- Brane

Re: FYI - Shelving & Checkpointing featured in Assembla's Cornerstone 4

[Julian Foad]

Branko Čibej wrote:

On 05.04.2018 21:06, Julian Foad wrote:

The Cornerstone 4 svn client for Mac ...

https://cornerstone.assembla.com/

... with Shelving and Checkpointing called out as new features.

It's exciting for me to see this new work going so quickly into
production!


What really "excites" me about that page is that:

   * there's no indication that shelving is an experimental feature of an
 as yet unreleased version;
   * it advertises checkpointing which, as far as I'm aware, we don't
 even have on trunk — so if this client does have it, the
 implementation is incompatible with any other client;
   * it does not even remotely attempt to use the Apache and Subversion
 trademarks correctly.

I'd appreciate it, Julian, if you brought these issues to the attention
of whoever is responsible for this.


Thanks for your quick review and response, Brane. I will do.

- Julian

Re: FYI - Shelving & Checkpointing featured in Assembla's Cornerstone 4

[Branko Čibej]

On 05.04.2018 21:06, Julian Foad wrote:
> The Cornerstone 4 svn client for Mac ...
>
> https://cornerstone.assembla.com/
>
> ... with Shelving and Checkpointing called out as new features.
>
> It's exciting for me to see this new work going so quickly into
> production!


What really "excites" me about that page is that:

  * there's no indication that shelving is an experimental feature of an
as yet unreleased version;
  * it advertises checkpointing which, as far as I'm aware, we don't
even have on trunk — so if this client does have it, the
implementation is incompatible with any other client;
  * it does not even remotely attempt to use the Apache and Subversion
trademarks correctly.


I'd appreciate it, Julian, if you brought these issues to the attention
of whoever is responsible for this.

-- Brane

FYI - Shelving & Checkpointing featured in Assembla's Cornerstone 4

[Julian Foad]

The Cornerstone 4 svn client for Mac ...

https://cornerstone.assembla.com/

... with Shelving and Checkpointing called out as new features.

It's exciting for me to see this new work going so quickly into production!

- Julian

Re: Checkpointing v1 design

[Julian Foad]

Julian Foad wrote:

[...]
This aims to streamline the manual method, which I use myself, of 
using 'svn diff' and 'svn patch' to create and apply patches named 
'feature-v1.patch', 'feature-v2.patch', etc.


Now on trunk:

$ svn help | grep "shel[fv]"
   shelf-diff
   shelf-drop
   shelf-list (shelves)
   shelf-log
   shelf-save
   shelve
   unshelve

- Julian

Checkpointing v1 - TortoiseSVN dialog sketches

[Julian Foad]

I have attached sketches to 
https://issues.apache.org/jira/browse/SVN-3626 for Shelve/Checkpoint, 
Unshelve/Restore, and Shelf Chooser dialogs.


The idea is these should replace the current Shelve and Unshelve 
dialogs. They will support shelving and unshelving, and also 
checkpointing (saving a new version of a shelf) and restoring 
(retrieving an old version of a shelf).


I plan to get these implemented on a TSVN branch next, that will be 
built against Subversion's 'shelve-checkpoint' branch where the API and 
command-line UI currently exists.


- Julian

Re: Checkpointing v1 in diagrams

[Julian Foad]

And here are the said diagrams...

Julian Foad wrote:

Here are a couple of diagrams to show how Checkpointing v1 behaves so far.

1. Shelve, Unshelve

Shows a difference between how Shelving-v1 worked and how it works in 
the new sheving-and-checkpointing design. Note that Unshelve no longer 
deletes the shelf. Shelving again would create a second version "foo 2".


2. Checkpoint, Revert, Restore

Shows how checkpointing and restoring works. Note that a 'revert' is 
needed before restoring.


(SVG format, drawn in Inkscape.)

- Julian

Checkpointing v1 in diagrams

[Julian Foad]

Here are a couple of diagrams to show how Checkpointing v1 behaves so far.

1. Shelve, Unshelve

Shows a difference between how Shelving-v1 worked and how it works in 
the new sheving-and-checkpointing design. Note that Unshelve no longer 
deletes the shelf. Shelving again would create a second version "foo 2".


2. Checkpoint, Revert, Restore

Shows how checkpointing and restoring works. Note that a 'revert' is 
needed before restoring.


(SVG format, drawn in Inkscape.)

- Julian

Re: Checkpointing v1 design

[Julian Foad]

Julian Foad wrote:

[...]
This aims to streamline the manual method, which I use myself, of using 
'svn diff' and 'svn patch' to create and apply patches named 
'feature-v1.patch', 'feature-v2.patch', etc.


An initial version is now committed to 'shelve-checkpoint' branch.

- Julian

Re: Checkpointing v1 design -- terminology

[Paul Hammant]

>
> $ svn save --revert -v2 "add hair color to person page"
>>
>> .. does the save and THEN drops the CL and it's changes in working copy -
>> back to no changes but not necessarily up to date.
>>
>
> That sounds like you're proposing an alternative syntax for what the
> 'shelve' command does -- except I'm not sure what you want the 'v2' option
> to do here; was that accidental?
>
>
Maybe alternate, sure.

$ svn savelist

add hair color to person page; v1
add hair color to person page; v1.2
add hair color to person page; whatever_passes_fora_versn_string

$ svn save -v v2 -m "add hair color to person page"
# Overwrites v2, perhaps would need a --force.

$ svn save -v three
# Created a new 'three' version of (implicit) "add hair color to person
page"

$ svn save -v three --revert
# As above but also deletes the changelist and any changes to working copy
versus last 'svn up'

$ svn restore -v three
# Creates a CL with all those changes on them.

Oh and shelve/save/checkpoint would all reuse impl as they are pretty
close. As does "Pull-request" (PR).

$ svn pull-request -m "add hair color to person page"
# Goes into codereview for Rietveld, Assembla, RhodeCode (etc),
# Possibly needing a plugin, or CURL configuration in .subversion/

- Paul

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 10.11.2017 14:20, Julian Foad wrote:
> Julian Foad wrote:
>> [...] To achieve such rollbacks, the user would have to first revert,
>> which involves deciding what to revert.
>>
>> And that seems OK to me.
>
> The larger point is that this kind of work flow, and the difficulty of
> working with changes in overlapping subtrees, is analogous to some
> existing techniques. For example, if one tries to use two different
> branches to manage two different change-sets, and work with them both
> in one WC by switching part of the WC to one branch and another part
> to another branch, and the changes involve overlapping subdirectories,
> maybe like this...
>
>  svn switch "^/branches/#1" A/B
>  svn switch "^/branches/#2" A/B/F A/D
>  svn switch "^/branches/#1"   A/D/G
>
> ... it seems to me that more or less the same issues arise.

To be fair, I've yet to hear of a VCS that can handle this sort of
complexity and make it easy for users. Git cops out by having
all-or-nothing branches, commits, etc. But add submodules and you're in
the same leaky boat again.

-- Brane

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Julian Foad wrote:
[...] To achieve such rollbacks, the user would have to first revert, which 
involves deciding what to revert.


And that seems OK to me.


The larger point is that this kind of work flow, and the difficulty of 
working with changes in overlapping subtrees, is analogous to some 
existing techniques. For example, if one tries to use two different 
branches to manage two different change-sets, and work with them both in 
one WC by switching part of the WC to one branch and another part to 
another branch, and the changes involve overlapping subdirectories, 
maybe like this...


 svn switch "^/branches/#1" A/B
 svn switch "^/branches/#2" A/B/F A/D
 svn switch "^/branches/#1"   A/D/G

... it seems to me that more or less the same issues arise.

Looking more specifically at one of the details, we note that 'svn 
switch URL PATH' recursively overrides the current URL on PATH and all 
its descendents (hence having to write the above example as three 
steps), whereas 'svn update' doesn't, and we have yet to define whether 
the proposed 'restore' command would do one or the other or something 
different. But if it's broadly analogous it seems reasonable.


- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Paul Hammant wrote:
I'd be happier with save-cl rather than ci-save.  Or just 'save', as I'm 
not sure what you're saving if not a change list.  Incidentally CL looks 
pretty close to CI in lower case and that's already used.


Spaces allowed in FOO name?

$ svn save -v2 "add hair color to person page"


Yes. The name obeys existing rules for a changelist name, for forward 
compatibility, and that does allow spaces.



Also..

$ svn save --revert -v2 "add hair color to person page"

.. does the save and THEN drops the CL and it's changes in working copy 
- back to no changes but not necessarily up to date.


That sounds like you're proposing an alternative syntax for what the 
'shelve' command does -- except I'm not sure what you want the 'v2' 
option to do here; was that accidental?



Revert files in (active) changelist FOO:
   svn revert --cl=FOO  # existing syntax


I don't know what this one does Julian?


I abbreviated a bit; the actual existing syntax for reverting the files 
tagged as changelist 'FOO' is:


  svn revert --cl=FOO -R .

(assuming all the files you need to revert are under '.').

- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Julian Foad wrote:
You can work out the 'rollbacks' from there (I haven't time right now); 


(That sounded like a crass cop-out, but I seriously didn't have a moment 
more to spare to think about it right then.)


The 'rollback' commands as written in steps 6 and 7 would both error. To 
achieve such rollbacks, the user would have to first revert, which 
involves deciding what to revert.


And that seems OK to me.

- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

I shudder to think of the interactions and side effects of partial
checkpoints. Consider our standard Greek tree and the following sequence:


First think about saving checkpoints the manual way, to set our initial 
expectations fairly low:


  svn diff foo-paths > foo-4.patch
  svn diff bar-paths > bar-17.patch

Then let's look at how we can improve on that.


  1. 
  2. svn checkpoint save "#1" A/B A/D/G
  3. 
  4. svn checkpoint save "#2" A/B/F A/D


I assume (1) these named changelists '#1' and '#2' were not defined (so 
not checkpointed) before step 1; and (2) you intend these 'save' 
commands to act with depth 'infinity' on the specified subtrees.


So you are concerned about how to handle overlapping subtrees -- in 
particular the case where a directory is associated with one changelist, 
and a subdirectory within it is (already or later) associated with another.



  5. 


I'll interpret that as 'do normal Subversion operations' rather than 
'execute arbitrary SQL statements on wc.db' :-)



  6. svn checkpoint rollback "#1"
  7. svn checkpoint rollback "#2"


OK, now is a good time to think about possible semantics. Here is a 
starting point.


  * A subtree is associated with a changelist either explicitly by 
attaching the changelist name to its root directory, or if not 
explicitly then implicitly by inheritance from its parent. A subtree 
association applies only to parts of the subtree that are not associated 
with a different changelist.


(Therefore a directory can be associated with only one changelist, and 
an explicit assignment overrides any association that would otherwise be 
inherited from its parents. After step 4 of your example everything in 
A/B is associated with '#1' *except* everything in A/B/F which is 
associated with '#2'.)


  * 'Saving' a changelist 'NAME' records the explicit attachment of 
NAME to particular PATHS.


  * Restoring a changelist 'NAME' restores the explicit attachment of 
NAME to particular PATHS. If some paths are already associated with 
changelist NAME, behaviour is client-defined, e.g. error, append to 
exiting assignments, replace existing assignments, with or without a 
warning. Let's say the initial CLI behaviour is error, for simplicity. 
(In other words the user has to revert before restoring, as I suggested 
a little earlier in this thread.)


  * The syntax '... save NAME PATHS...' first assigns changelist NAME 
to PATH..., replacing any previous assignment for PATH..., and then 
saves all paths belonging to NAME.


With this, your step 2 would save the whole subtree at A/B (and the one 
at A/D/G) in a first checkpoint of '#1', while your step 4 would 
unassign A/B/F from changelist '#1' and assign it to changelist '#2' 
instead.


You can work out the 'rollbacks' from there (I haven't time right now); 
I acknowledge it won't do magic for you -- but I'm sure you wouldn't 
want it to.


How does that strike you from a pragmatic point of view?

- Julian

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 07.11.2017 16:15, Julian Foad wrote:
> Branko Čibej wrote:
>> On 06.11.2017 17:30, Julian Foad wrote:
>>> The key point I am making here is the "topic" we're talking about in
>>> these commands is the "changelist" rather than the "savepoint" which
>>> is merely a stored version of a changelist.
>>
>> All our current commands that use changelists use them as an alias for a
>> set of targets. [...]
>>
>> Your proposal elevates changelists to a status that they do not
>> currently have in the client, and makes their usage inconsistent with
>> the rest of the command line. You might as well propose
>>
>>  svn changelist commit
>>  svn changelist diff
>>  svn changelist switch
>>
>> and so on. Whether that would be a good idea is not the issue here: the
>> issue is consistency, and (possibly) backwards-compatibility.
>
> I absolutely am considering elevating the status of changelists to
> that level, yes! That makes a very good model in my opinion.
>
>> And the
>> rather important detail that changelists cannot be associated with
>> directories.
>
> Yes, that's noted as a needed enhancement.
>
>> Changelists were always a sort of afterthought ...
>
> Indeed, that made me sad from their beginning. And what better
> opportunity to at last make a plan to improve them?

Noted. +1. Enjoy the corner cases. :)


>> On another note: changelists with checkpoints make no sense IMO; a
>> checkpoint should save and restore the state of the whole working copy.
>> Shelved changes are different. Their internal representation may be the
>> same, but the semantics are not.
>
> Why do you think 'checkpointing' your work should be across the whole
> WC whereas 'shelving' it should not? I can see how 'checkpoint your
> whole WC' has a utility, sure, but no more so than 'shelve your whole
> WC'. I think it makes a great deal of sense to checkpoint a unit of
> work rather than the whole WC, and to work towards elevating
> 'changelist' to fulfil that role.

I shudder to think of the interactions and side effects of partial
checkpoints. Consider our standard Greek tree and the following sequence:

 1. 
 2. svn checkpoint save "#1" A/B A/D/G
 3. 
 4. svn checkpoint save "#2" A/B/F A/D
 5. 
 6. svn checkpoint rollback "#1"
 7. svn checkpoint rollback "#2"


Um ... is that a rebase, a reorder, or a pretzel? Or does rolling back
to the state before #1 imply rolling back #2 as well? If the latter,
then checkpoints form a stack and represent the whole WC state.

-- Brane

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

On 06.11.2017 17:30, Julian Foad wrote:

The key point I am making here is the "topic" we're talking about in
these commands is the "changelist" rather than the "savepoint" which
is merely a stored version of a changelist.


All our current commands that use changelists use them as an alias for a
set of targets. [...]

Your proposal elevates changelists to a status that they do not
currently have in the client, and makes their usage inconsistent with
the rest of the command line. You might as well propose

     svn changelist commit
     svn changelist diff
     svn changelist switch

and so on. Whether that would be a good idea is not the issue here: the
issue is consistency, and (possibly) backwards-compatibility.


I absolutely am considering elevating the status of changelists to that 
level, yes! That makes a very good model in my opinion.



And the
rather important detail that changelists cannot be associated with
directories.


Yes, that's noted as a needed enhancement.


Changelists were always a sort of afterthought ...


Indeed, that made me sad from their beginning. And what better 
opportunity to at last make a plan to improve them?



unless you plan to
elevate them to first-class concepts across the whole command-line, I
recommend that the shelve/save/whatnot commands simply accept a
--changelist option and not complicate further. That this doesn't
preclude populating a changelist when a shelved change is restored.

On another note: changelists with checkpoints make no sense IMO; a
checkpoint should save and restore the state of the whole working copy.
Shelved changes are different. Their internal representation may be the
same, but the semantics are not.


Why do you think 'checkpointing' your work should be across the whole WC 
whereas 'shelving' it should not? I can see how 'checkpoint your whole 
WC' has a utility, sure, but no more so than 'shelve your whole WC'. I 
think it makes a great deal of sense to checkpoint a unit of work rather 
than the whole WC, and to work towards elevating 'changelist' to fulfil 
that role.


- Julian

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 06.11.2017 17:30, Julian Foad wrote:
> The key point I am making here is the "topic" we're talking about in
> these commands is the "changelist" rather than the "savepoint" which
> is merely a stored version of a changelist.

All our current commands that use changelists use them as an alias for a
set of targets. Instead of 'svn commit foo bar baz' you'd say 'svn
commit --changelist fbb'.

Your proposal elevates changelists to a status that they do not
currently have in the client, and makes their usage inconsistent with
the rest of the command line. You might as well propose

    svn changelist commit
    svn changelist diff
    svn changelist switch

and so on. Whether that would be a good idea is not the issue here: the
issue is consistency, and (possibly) backwards-compatibility. And the
rather important detail that changelists cannot be associated with
directories.

Changelists were always a sort of afterthought ... unless you plan to
elevate them to first-class concepts across the whole command-line, I
recommend that the shelve/save/whatnot commands simply accept a
--changelist option and not complicate further. That this doesn't
preclude populating a changelist when a shelved change is restored.

On another note: changelists with checkpoints make no sense IMO; a
checkpoint should save and restore the state of the whole working copy.
Shelved changes are different. Their internal representation may be the
same, but the semantics are not.

-- Brane

Re: Checkpointing v1 design -- terminology

[Paul Hammant]

I'd be happier with save-cl rather than ci-save.  Or just 'save', as I'm
not sure what you're saving if not a change list.  Incidentally CL looks
pretty close to CI in lower case and that's already used.

Spaces allowed in FOO name?

$ svn save -v2 "add hair color to person page"

Also..

$ svn save --revert -v2 "add hair color to person page"

.. does the save and THEN drops the CL and it's changes in working copy -
back to no changes but not necessarily up to date.

> Revert files in (active) changelist FOO:
>   svn revert --cl=FOO  # existing syntax

I don't know what this one does Julian?

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Julian Foad wrote:
'Roll back' means retrieving an old version of a shelved change and 
could be named like one or a permutation of these:


   svn unshelve --checkpoint=3 foo
   svn shelf restore foo@3
   svn revert --savepoint=3 --cl=foo
   svn changelist restore -v3 foo

(I am deliberately bringing 'changelist' into the mix, as I believe 
integration with 'changelist' is where we need to take this.)


The integration with changelists that I am thinking of is basically that 
changelist FOO corresponds to the working (unshelved) version of a 
shelved change named FOO. Unshelving or restoring FOO will apply the 
saved change FOO and assign the affected files to changelist FOO; and 
saving or shelving FOO will use the changelist FOO as the set of files 
to save or shelve.


With this changelist integration we could have... (I know we can't 
compatibly have exactly this syntax; see below for alternatives)...


Shelve = save and revert:
  svn cl shelve FOO

Checkpoint = save and don't revert:
  svn cl save FOO

Unshelve / restore / roll back (all possible names for same thing):
  svn cl unshelve [-v VERSION_NUM] FOO
  svn cl restore [-v VERSION_NUM] FOO
  (etc.)

Revert files in (active) changelist FOO:
  svn revert --cl=FOO  # existing syntax
  svn cl revert FOO

Commit the (possibly shelved) changelist FOO:
  svn commit --cl=FOO  # existing syntax
  svn cl commit FOO


Possible different ways to write the topic noun and the verb:

  svn clsave  FOO [PATH...]
  svn cl save FOO [PATH...]
  svn cl --save   FOO [PATH...]
  svn changelist save FOO [PATH...]
  svn changelist-save FOO [PATH...]

  svn save   --cl=FOO [PATH...]


The key point I am making here is the "topic" we're talking about in 
these commands is the "changelist" rather than the "savepoint" which is 
merely a stored version of a changelist.


- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

[...] I've often wished we'd
made our argument parser slightly more flexible in that respect, so that
[...] we could write

     svn [topic] command [options args]



FWIW, the reason I'm slightly in favour of "savepoint" vs. "checkpoint"
has been discussed before: the abbreviation "sp" is not taken yet, [...]


Ack.



I am thinking for example that we might want to 'send' this change to
a review system or 'commit' it to the svn server (ok we'd use 'svn
commit ' for that) or export it to a patch or email (git
format-patch). The main actions on a Perforce changelist are done by
first-class commands 'p4 submit', 'p4 shelve', etc. while the 'p4
change[list]' command has (option-letter) verbs for 'input' and
'output' actions.

Any suggestions for the 'shelve' terminology problem? Change to
'stash' and incur false expectations based on git stash? Is there
something else that, like 'stash', can be used as a noun and a verb
and sounds good?


IIUC the shelving prototype is a lot closer to Perforce's than Git's,
right? Then thinking about shelves is better than about stashes. Why not
do the same as with (save|check)points: the [topic] should be a noun,
the [command] a verb, so:

     svn shelf save ...
     svn shelf restore ...


Next observation: 'shelves' and 'checkpoints' are not two alternative 
kinds of thing; rather savepoints/checkpoints are versions of a 
(shelved) change.


So 'save this point' and 'roll back to a saved point' need not be in 
grouped in their own 'savepoint' topic namespace but would perhaps be 
better described as actions on a (potentially shelved) change.


'Roll back' means retrieving an old version of a shelved change and 
could be named like one or a permutation of these:


  svn unshelve --checkpoint=3 foo
  svn shelf restore foo@3
  svn revert --savepoint=3 --cl=foo
  svn changelist restore -v3 foo

(I am deliberately bringing 'changelist' into the mix, as I believe 
integration with 'changelist' is where we need to take this.)


I was particularly looking at the case where we have some local 
modifications already and we want to 'roll back' to an earlier version 
of the files. I had been thinking about how Subversion should decide 
which modifications in the WC it should revert before restoring the old 
version. E.g. revert(union(files in requested old version, files in 
most-recently-restored version)). That seems prone to unintended 
consequences as the 'most recently restored version isn't necessarily 
known at all, or correctly known in cases where the user restored 
something and then manually reverted and applied some other change. Also 
the set of files that was previously recorded may not be the same as the 
set of files currently undergoing editing, as it is common to start 
affecting other files. (To handle this sort of case, we need the 
'changelist' mechanism augmented by the notion of assigning new changes 
to a 'default changelist'.)


Instead of all that uncertainty, let the user be responsible for 
reverting the current modifications currently in progress, and then roll 
back using a simple form of 'unshelve' that does not attempt to revert 
anything first. For added safety, 'unshelve' should warn (e.g. require 
'force' to override) if it attempts to modify any already-modified 
files. (This was already a suggested enhancement.)


- Julian

Re: Checkpointing v1 design -- terminology

[Daniel Shahaf]

Branko Čibej wrote on Sat, 04 Nov 2017 13:51 +0100:
> Traditionally we've striven to appear marginally literate in the text we
> produce, that includes command structure. There's no harm in trying to
> adhere to that tradition, especially as it does not cost us more than
> some chattering on the developers' list.
> 
> "Checkpoint" is not a verb. Written English has deteriorated to the
> toddler level in the last several decades, but I see no reason for us to
> follow suit.

I agree that we should use correct English but I don't see how using
"checkpoint" as a verb would be a problem in any way.

Let's just agree to disagree.

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 04.11.2017 00:21, Daniel Shahaf wrote:
> But I don't understand what's the problem that we're trying to solve
> here.  Why is it a problem that "shelf" is a noun and "shelve" a verb?
> And for that matter, why couldn't we use "checkpoint" as a verb?

Traditionally we've striven to appear marginally literate in the text we
produce, that includes command structure. There's no harm in trying to
adhere to that tradition, especially as it does not cost us more than
some chattering on the developers' list.

"Checkpoint" is not a verb. Written English has deteriorated to the
toddler level in the last several decades, but I see no reason for us to
follow suit.

-- Brane

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 03.11.2017 22:51, Julian Foad wrote:
> Branko Čibej wrote:
>>>    * main commands are 'checkpoint' to save a new version and
>>> 'rollback' to revert to version N
>>
>> The main issue I have with this is that "checkpoint" is not a verb. (I
>> know, neither are 'changelist' or 'auth').
>>
>> How about 'save' (to create a savepoint) and 'restore' instead?
>>
>> As to why I'm nitpicking at this point: whatever we choose will be
>> pretty much set in stone, in the UI and the API, for the conceivable
>> future.
>
> Thanks for helping with the terminology. I agree it's important.
>
> The prototype CLI I'm working on (in the 'shelve-checkpoint' branch)
> actually has 'svn checkpoint save' and 'svn checkpoint rollback'. Do
> you think mean to suggest plain 'svn save' and 'svn restore'?

No, the two-keyword approach is actually better. I've often wished we'd
made our argument parser slightly more flexible in that respect, so that
instead of

    svn command [options args]

we could write

    svn [topic] command [options args]

which would be useful for topics such as "(save|check)point",
"changelist", "auth", "branch" and so on. I'm absolutely in favour of
the second form for complex cases.

[...]

> A reason for choosing a noun (phrase) such as 'checkpoint',
> 'savepoint', 'changelist' or 'changeset' followed by a verb is to
> allow us to add verbs without cluttering the main command name space.

Precisely. +1

FWIW, the reason I'm slightly in favour of "savepoint" vs. "checkpoint"
has been discussed before: the abbreviation "sp" is not taken yet, but
"cp" is; I suspect it would save "power users" of the command line a ton
of grief if they could write "svn sp s[quash] 0:-3" instead of "svn cp
squash --from 0 --to -3" and then get a syntax error. :)


> I am thinking for example that we might want to 'send' this change to
> a review system or 'commit' it to the svn server (ok we'd use 'svn
> commit ' for that) or export it to a patch or email (git
> format-patch). The main actions on a Perforce changelist are done by
> first-class commands 'p4 submit', 'p4 shelve', etc. while the 'p4
> change[list]' command has (option-letter) verbs for 'input' and
> 'output' actions.
>
> Any suggestions for the 'shelve' terminology problem? Change to
> 'stash' and incur false expectations based on git stash? Is there
> something else that, like 'stash', can be used as a noun and a verb
> and sounds good?

IIUC the shelving prototype is a lot closer to Perforce's than Git's,
right? Then thinking about shelves is better than about stashes. Why not
do the same as with (save|check)points: the [topic] should be a noun,
the [command] a verb, so:

    svn shelf save ...
    svn shelf restore ...

I'm not sure about the abbreviation here ... "sh" sounds tempting but is
also a well-known abbreviation for "shell".

-- Brane

Re: Checkpointing v1 design

[Vincent Lefevre]

On 2017-11-03 19:39:51 +0100, Branko Čibej wrote:
> On 03.11.2017 15:54, Julian Foad wrote:
> > After playing with two pre-prototypes and discussing a wide variety of
> > ideas on this list, I have given detailed thought to a v1
> > checkpointing design with the following properties:
> >
> >   * local (not server)
> >   * built on shelving v1 (patches)
> >   * main commands are 'checkpoint' to save a new version and
> > 'rollback' to revert to version N
> 
> The main issue I have with this is that "checkpoint" is not a verb. (I
> know, neither are 'changelist' or 'auth').
> 
> How about 'save' (to create a savepoint) and 'restore' instead?

Wouldn't it be better if these related commands have a common prefix?
A bit like property-related commands start with "prop".

-- 
Vincent Lefèvre  - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Checkpointing v1 design -- terminology

[Daniel Shahaf]

Julian Foad wrote on Fri, Nov 03, 2017 at 21:51:37 +:
> Branko Čibej wrote:
> > >    * main commands are 'checkpoint' to save a new version and
> > > 'rollback' to revert to version N
> > 
> > The main issue I have with this is that "checkpoint" is not a verb. (I
> > know, neither are 'changelist' or 'auth').
> > 
> > How about 'save' (to create a savepoint) and 'restore' instead?
> > 
> > As to why I'm nitpicking at this point: whatever we choose will be
> > pretty much set in stone, in the UI and the API, for the conceivable future.
> 
> Thanks for helping with the terminology. I agree it's important.
> 
> The prototype CLI I'm working on (in the 'shelve-checkpoint' branch)
> actually has 'svn checkpoint save' and 'svn checkpoint rollback'. Do you
> think mean to suggest plain 'svn save' and 'svn restore'? I'm not sure about
> that. They could deserve to be first-class 'svn' commands if we consider
> them first-class companions to 'commit', 'revert' and 'diff'.
> 
> The analogy to 'save' a file or the work in almost any application is apt.
> (For the converse we should then consider 'open' or 'load', but neither of
> these sound good.)

"edit"?  Inspired by Vim's :edit command; goes well with the "prototype
commit" mental model.

> Any suggestions for the 'shelve' terminology problem? Change to 'stash' and
> incur false expectations based on git stash? Is there something else that,
> like 'stash', can be used as a noun and a verb and sounds good?

pile, pocket, fold, store, house, dock, box, rack, hold, stow.

But I don't understand what's the problem that we're trying to solve
here.  Why is it a problem that "shelf" is a noun and "shelve" a verb?
And for that matter, why couldn't we use "checkpoint" as a verb?

Cheers,

Daniel

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

   * main commands are 'checkpoint' to save a new version and
'rollback' to revert to version N


The main issue I have with this is that "checkpoint" is not a verb. (I
know, neither are 'changelist' or 'auth').

How about 'save' (to create a savepoint) and 'restore' instead?

As to why I'm nitpicking at this point: whatever we choose will be
pretty much set in stone, in the UI and the API, for the conceivable future.


Thanks for helping with the terminology. I agree it's important.

The prototype CLI I'm working on (in the 'shelve-checkpoint' branch) 
actually has 'svn checkpoint save' and 'svn checkpoint rollback'. Do you 
think mean to suggest plain 'svn save' and 'svn restore'? I'm not sure 
about that. They could deserve to be first-class 'svn' commands if we 
consider them first-class companions to 'commit', 'revert' and 'diff'.


The analogy to 'save' a file or the work in almost any application is 
apt. (For the converse we should then consider 'open' or 'load', but 
neither of these sound good.)


Saving the work-in-progress is also like a deferred not-quite commit, so 
some variant of the 'commit' command such as 'svn commit [--]save' might 
be appropriate.


A reason for choosing a noun (phrase) such as 'checkpoint', 'savepoint', 
'changelist' or 'changeset' followed by a verb is to allow us to add 
verbs without cluttering the main command name space. I am thinking for 
example that we might want to 'send' this change to a review system or 
'commit' it to the svn server (ok we'd use 'svn commit ' for 
that) or export it to a patch or email (git format-patch). The main 
actions on a Perforce changelist are done by first-class commands 'p4 
submit', 'p4 shelve', etc. while the 'p4 change[list]' command has 
(option-letter) verbs for 'input' and 'output' actions.


Any suggestions for the 'shelve' terminology problem? Change to 'stash' 
and incur false expectations based on git stash? Is there something else 
that, like 'stash', can be used as a noun and a verb and sounds good?


- Julian

Re: Checkpointing v1 design

[Branko Čibej]

On 03.11.2017 15:54, Julian Foad wrote:
> After playing with two pre-prototypes and discussing a wide variety of
> ideas on this list, I have given detailed thought to a v1
> checkpointing design with the following properties:
>
>   * local (not server)
>   * built on shelving v1 (patches)
>   * main commands are 'checkpoint' to save a new version and
> 'rollback' to revert to version N


The main issue I have with this is that "checkpoint" is not a verb. (I
know, neither are 'changelist' or 'auth').

How about 'save' (to create a savepoint) and 'restore' instead?

As to why I'm nitpicking at this point: whatever we choose will be
pretty much set in stone, in the UI and the API, for the conceivable future.

-- Brane

Re: Checkpointing v1 design

[Julian Foad]

Listing shelved and applied change-sets could look something like this:

[[[
$ svn shelves
applied:
  X (at version 1)
  thing (at version 10)
shelved:
  foo (1 version)
  bar (4 versions)
  old-thing (1 version)
  new-changes (18 versions)
]]]

or integrated with changelists:

[[[
$ svn changelist --list
--- Applied Changelists:
  X (checkpoint 1, 1 file)
  thing (checkpoint 10, 2 files)
--- Shelved Changelists:
  foo (checkpoint 1, 14 files)
  bar (checkpoint 4, 1 file)
  old-thing (checkpoint 1, 2 files)
  new-changes (checkpoint 18, 3 files)
]]]

Listing checkpoints:

[[[
$ svn checkpoint list [--cl=]thing
thing-1 1230 mins old874 bytes
 (no log message)
 hello.txt |1 +
 1 file changed, 1 insertion(+)
...
thing-10  10 mins old  12345 bytes
 log message: Blah blah blah...
 hello.txt |1 +
 bar.txt   |   12 
 2 files changed, 13 insertions(+)
]]]

- Julian

Checkpointing v1 design

[Julian Foad]

After playing with two pre-prototypes and discussing a wide variety of 
ideas on this list, I have given detailed thought to a v1 checkpointing 
design with the following properties:


  * local (not server)
  * built on shelving v1 (patches)
  * main commands are 'checkpoint' to save a new version and 'rollback' 
to revert to version N

  * not intended for making a series of commits
  * designed to integrate with 'changelists' later

This aims to streamline the manual method, which I use myself, of using 
'svn diff' and 'svn patch' to create and apply patches named 
'feature-v1.patch', 'feature-v2.patch', etc.


I plan to go ahead and implement this, in order to have something 
useful. At the same time I will keep an eye on further developments and 
the wider picture.



== Shelving and Checkpointing Commands ==

unshelve X [N]
  assert(not X.applied)
  if (any patch-get-paths(X) is modified):
warn?
  patch-apply X [N]
  X.applied := true

shelve X [PATH...]
  # X may or may not be previously 'applied'
  assert(any PATH... is modified)
  patch-save(X, PATH...)
  patch-unapply X
  X.applied := false

checkpoint X [PATH...]
  # X may or may not be previously 'applied'
  assert(any PATH... is modified)
  patch-save(X, PATH...)
  X.applied := true

rollback X [N]
  assert(X.applied)
  revert(union( patch-get-files(X, X.current),
patch-get-files(X, N) ))
  patch-apply(X, N)
  patch-set-current(X, N)

drop/remove/delete X
  # current syntax "svn shelve --delete X"
  assert(not X.applied)
  patch-delete-all X


== Interaction with existing commands ==

Existing ways to remove a change from the WC are 'revert' and 'commit'.

commit X
  # possible syntax "svn commit --changelist X"
  assert(X.applied)
  commit(patch-get-files(X))
  X.applied := false

revert X
  # possible syntax "svn revert --changelist X"
  assert(X.applied)
  revert(patch-get-files(X))
  X.applied := false

revert PATH...
  revert(PATH...)
  if PATH... includes patch-get-files(X):
X.active := false


== Low-level Interface ==

patch-save(X, PATH...)
  save diff(PATH...) as 'X-.patch'

patch-apply(X, N=latest)
  apply 'X-N.patch' to the WC

patch-unapply(X, N=latest)
  reverse-apply 'X-N.patch' to the WC

patch-delete-all(X)
  delete all versions of X

patch-get-current(X)
  return N = the maximum '*' in 'X-*.patch'

path-set-current(X, N)
  mark the current applied version of X as being N
  (options: prune all versions greater than N, or set X.current=N,
  or even perhaps 'symlink' 'X-.patch' to 'X-N.patch')

patch-get-files(X, N=latest)
  return the list of (file) paths in X version N


=== Remembering the Applied/Shelved state ===

For each change-set name X, the WC will remember whether X is considered 
to be currently 'applied' to the WC, and if so, at which version N. This 
is needed so that 'rollback' knows (roughly) what to revert, and to a 
lesser degree so that commands like 'shelve' and 'unshelve' can say 
'hold on, you already have [or do not have] that change-set applied'.


The Shelving v1 implementation by default deleted a patch (or rather 
renamed it to '.bak') when unshelving, and when shelving it objected if 
the given name was already shelved. Now in this design we are going to 
keep all versions of X (and 'shelve' will always add a new version) so 
we will store a separate 'applied' flag instead.


Multiple independent change-sets can be active in the WC at the same 
time, like with 'changelists'. Clashes (overlaps) in the list of paths 
affected must be managed manually. (Potential enhancement: add some 
warnings, e.g. if 'rollback' is going to revert a file that is part of 
both the specified change-set and another one.)



=== Rollback and Revert ===

Perhaps one of the trickiest parts is which files will 'rollback' 
revert. Suggestion is union(files in current applied version, files in 
requested version).


May want to check and warn if those files overlap with any other 
currently applied change-set.


Integration with changelists should clarify this.


=== Path spec [PATH...] ===

* is applicable to 'save' commands (shelve, checkpoint)
* is not applicable to other commands
* default is "." like in most svn commands
* is restrictive (restricts operation to PATH...)
* is not tracked or managed
* clashes (same path in more than one applied patch) are not managed

Future considerations:

* when checkpointing, warn if PATH... excludes any paths that were in 
the previous version?

* consider allowing restricting paths on 'apply' commands.


=== Changelists and Terminology ===

Already in this design we see a need to specify a 'change set&#

Re: Shelving / Checkpointing thoughts

[Paul Hammant]

>
> Apache products aren't allowed to depend on GPL'd code for mandatory /
> core features.
>

Even GPL2 with the linking exception?


> What is the advantage of linking to libgit2 over simply telling users to
> use git-svn with 'git stash' and local branches?
>

None, assuming you've learned the Git command line. I guess git-svn
semantics, too.

- Paul

Re: Shelving / Checkpointing thoughts

[Paul Hammant]

Fully expecting this to have a popularity of -5 on a 0 - 10 scale, I
thought I'd share that I'm progressing with a "subversion shelve" tech on
GitHub, that uses Git for the Shelve/unshelve (making a single 'bundle'
blob) :- https://github.com/paul-hammant/svn-shelve

Pull Requests welcome :)

Id thought Python would be easy to work with here, but I keep getting
caught out when Py.Test chokes on me changing directories in order to do
operations, so I'm wanting some form of try/finally fu for that and maybe
also pushd/popd .

It would be great if Subversion could do the command naming trick of Git.
svn-shelve could be handled by the svn command itself, just the same as Git
does - I previously created git-slim and git-size

.

Regards,

- Paul

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Julian Foad wrote:
> Stefan Kueng wrote:
>> Another reason to store whole files: patches don't work with non-text
>> files. Which is a major problem especially on Windows where files often
>> are encoded in utf16.
> 
> Making 'svn diff' and 'svn patch' support non-text files is already a
> dependency of shelving-by-patch-files.

See the "Existing Issues to Overcome" section,
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.4sj99cgksge9

> At present, 'svn diff --git' supports non-text files by writing into the
> patch file (and encoded into ASCII), the whole of the old file and the
> whole of the new file. So that already does what you are asking: it
> 'stores whole files'. And 'svn patch' handles this as input. Just like
> 'svn merge', it chooses either one side change or the other, and if both
> sides changed that is a conflict. (I haven't tested it, I assume it
> works like that.)

Which implies that binary file support might "just work" if I simply add
the "--git" option in to the "diff" call in the prototype. I plan to try
that.

And before anyone raises their hand to say this design will be too slow
for use cases involving very large binary files, yes, I know, it almost
certainly will be. This is called a prototype for good reason.

- Julian


> Storing whole files will not magically make merging UTF-16 text changes
> work. For that, the merge will need to be taught how to process UTF-16
> text. And if we do that, then 'svn diff' and 'svn patch' could process
> UTF-16 as text too. That would be a great enhancement to Subversion, but
> is an orthogonal issue.

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Stefan Kueng wrote:
> Another reason to store whole files: patches don't work with non-text
> files. Which is a major problem especially on Windows where files often
> are encoded in utf16.

Making 'svn diff' and 'svn patch' support non-text files is already a
dependency of shelving-by-patch-files.

At present, 'svn diff --git' supports non-text files by writing into the
patch file (and encoded into ASCII), the whole of the old file and the
whole of the new file. So that already does what you are asking: it
'stores whole files'. And 'svn patch' handles this as input. Just like
'svn merge', it chooses either one side change or the other, and if both
sides changed that is a conflict. (I haven't tested it, I assume it
works like that.)

Storing whole files will not magically make merging UTF-16 text changes
work. For that, the merge will need to be taught how to process UTF-16
text. And if we do that, then 'svn diff' and 'svn patch' could process
UTF-16 as text too. That would be a great enhancement to Subversion, but
is an orthogonal issue.

- Julian

Re: Shelving / Checkpointing thoughts

[Stefan Kueng]

On 27.08.2017 02:09, Branko Čibej wrote:

On 27.08.2017 02:04, Daniel Shahaf wrote:

Julian Foad wrote on Fri, 25 Aug 2017 21:42 +0100:

Let's clarify. We can mean two possible things when we say 'a series of
patches':

   1. "patch versions": a series of successively better patches, all
attempting the same logical thing, all from the same base, and only one
of which is applied at any time;

   2. a series of patches, each providing a different logical change,
where each patch is based on the result of applying the previous one.
("quilt" is a tool for managing path series of this kind. My 'option 3'
(local repository) design for checkpointing could also be used in this
way, in a primitive way, but would not support revising earlier patches
in the series which is a key strength of what "quilt" can do.)

I am talking about definition 1 ("patch versions").

I propose patches in a series of patch versions be named "featureA-1",
"featureA-2", ... (This is what I do already, manually, in my own work.)

I propose that we should not attempt to provide any special support for
definition 2 within this "shelving" feature; users can manage that
themselves by simply remembering which feature names depend on which
other ones, or by including some other numbering system within the names.

Could you explain how you see this working?  In use-case #2, later patches
in the series typically depend on earlier patches.  I don't see how that 
use-case
can be addressed if the patches are all implemented against deltas against
the same base (for example, because if patch #2 in a 5-patch series is edited,
all later patches would have to be regenerated).

If that's not clear enough I'd be happy to elaborate.

Cheers,

Daniel

P.S. I think brane's suggestion would make the "edit patch #2 in a 5-patch 
series"
use-case a lot easier: diff3 is exactly the right tool for rebasing patch #3 on 
top
of the edited #2.


Yes, that's one of the reasons I proposed storing whole files instead of
patches. Rebasing and reordering would become much simpler (although not
exactly "simple" if we take tree restructuring into account).


Another reason to store whole files: patches don't work with non-text 
files. Which is a major problem especially on Windows where files often 
are encoded in utf16.


Stefan

Re: Shelving / Checkpointing thoughts

[Daniel Shahaf]

Paul Hammant wrote on Sun, Aug 27, 2017 at 07:04:03 -0400:
> Delegating to libgit2 to invisibly handle: shelves, local-branches and
> pull-requests could yield a workable and flexible implementation.

Apache products aren't allowed to depend on GPL'd code for mandatory /
core features.

What is the advantage of linking to libgit2 over simply telling users to
use git-svn with 'git stash' and local branches?

Re: Shelving / Checkpointing thoughts

[Paul Hammant]

Here's a prototype in Python2 that makes a git repo in a 'shelve' folder.
Two commits - one with the starting position, and one with the ending
position. The 'svn info' for the resource is copied in too (same file name
with a '.info' suffix).

import sh
import os
from stat import S_IWUSR, S_IREAD


import fileinput

sh.rm("-rf", "maven-gpg-plugin")
sh.rm("-rf", "shelve")

sh.svn("co", 
"https://svn.apache.org/repos/asf/maven/plugins/trunk/maven-gpg-plugin/";)

# Change two files.
for line in fileinput.input("maven-gpg-plugin/pom.xml", inplace=True):
 print "%d: %s" % (fileinput.filelineno(), line),
for line in 
fileinput.input("maven-gpg-plugin/src/main/java/org/apache/maven/plugin/gpg/SigningBundle.java",
inplace=True):
 print "%d: %s" % (fileinput.filelineno(), line),

# While files changed?
files = sh.svn("st", "maven-gpg-plugin")

sh.mkdir("shelve")
sh.git("init", "shelve")

# Copy originals in.
for line in files.splitlines():
 file_name = line[1:].strip()
 info = sh.svn("info", file_name)
 for iLine in info:
  if iLine.startswith("Checksum:"):
   checksum = iLine.split(" ")[1].strip()
   dir = checksum[0:2]
   sh.mkdir("-p", "shelve/" + "/".join(file_name.split('/')[:-1]))
   sh.cp("maven-gpg-plugin/.svn/pristine/" + dir + "/" +
checksum + ".svn-base", "shelve/" + file_name)
   os.chmod("shelve/" + file_name, S_IWUSR | S_IREAD) #
make writable
 f = open("shelve/" + file_name + ".info", 'w')
 f.writelines(info)
 f.close()

# Do a commit
sh.cd("shelve")
sh.git("add", ".")
sh.git("commit", "-m", "start")
sh.cd("..")

# Copy changed versions in.
for line in files.splitlines():
 file_name = line[1:].strip()
 sh.cp(file_name, "shelve/" + file_name)

# Do a commit
sh.cd("shelve")
sh.git("add", ".")
sh.git("commit", "-m", "finish")
sh.cd("..")

Re: Shelving / Checkpointing thoughts

[Paul Hammant]

Perhaps easy to prototype in Python, too.

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Paul Hammant wrote:
> Delegating to libgit2 to invisibly handle: shelves, local-branches and
> pull-requests could yield a workable and flexible implementation.
> [...]

It will definitely be worth me looking into the possibilities. Thanks
for the suggestion.

- Julian

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Daniel Shahaf wrote:
> Julian Foad wrote on Fri, 25 Aug 2017 21:42 +0100:
>> Let's clarify. We can mean two possible things when we say 'a series of
>> patches':
>>
>>   1. "patch versions": a series of successively better patches, all
>> attempting the same logical thing, all from the same base, and only one
>> of which is applied at any time;
>>
>>   2. a series of patches, each providing a different logical change,
>> where each patch is based on the result of applying the previous one.
>> ("quilt" is a tool for managing path series of this kind. My 'option 3'
>> (local repository) design for checkpointing could also be used in this
>> way, in a primitive way, but would not support revising earlier patches
>> in the series which is a key strength of what "quilt" can do.)
>>
>> I am talking about definition 1 ("patch versions").
>> [...]
>>
>> I propose that we should not attempt to provide any special support for
>> definition 2 within this "shelving" feature; users can manage that
>> themselves by simply remembering which feature names depend on which
>> other ones, or by including some other numbering system within the names.
> 
> Could you explain how you see this working?  In use-case #2, later patches
> in the series typically depend on earlier patches.  I don't see how that 
> use-case
> can be addressed if the patches are all implemented against deltas against
> the same base (for example, because if patch #2 in a 5-patch series is edited,
> all later patches would have to be regenerated).

This kind of shelving (with simple checkpointing add-ons) would *not*
help with use case #2: in particular it would not provide any support
for editing an earlier patch and rebasing the later patches. All I meant
is that you would still be free to use (manually) any naming/numbering
system you wish to help you remember how the patches that you shelve are
related.

- Julian

Re: Shelving / Checkpointing thoughts

[Paul Hammant]

Delegating to libgit2 to invisibly handle: shelves, local-branches and
pull-requests could yield a workable and flexible implementation.

Pull-requests flow back to the central repo, of course) and there
interactive and automatic workflows include: code review, classic CI,
linting and other badge-style data-points.  Those workflows of course are
the purview of other pieces of software that are triggered by Subversion.

- Paul

Re: Shelving / Checkpointing thoughts

[Branko Čibej]

On 27.08.2017 06:00, Daniel Shahaf wrote:
> Branko Čibej wrote on Sun, 27 Aug 2017 02:09 +0200:
>> On 27.08.2017 02:04, Daniel Shahaf wrote:
>>> P.S. I think brane's suggestion would make the "edit patch #2 in a 5-patch 
>>> series"
>>> use-case a lot easier: diff3 is exactly the right tool for rebasing patch 
>>> #3 on top
>>> of the edited #2.
>> Yes, that's one of the reasons I proposed storing whole files instead of
>> patches. Rebasing and reordering would become much simpler (although not
>> exactly "simple" if we take tree restructuring into account).
> The part with tree restructuring would "simply" require a working diff3
> implementation for tree changes.  Isn't that exactly what the tree
> conflicts work is supposed to produce?

I was wondering about that, yes. Not sure how one would combine
(un)shelving and conflict resoiltion, but clearly it must be possible
somehow.

-- Brane

Re: Shelving / Checkpointing thoughts

[Daniel Shahaf]

Branko Čibej wrote on Sun, 27 Aug 2017 02:09 +0200:
> On 27.08.2017 02:04, Daniel Shahaf wrote:
> > P.S. I think brane's suggestion would make the "edit patch #2 in a 5-patch 
> > series"
> > use-case a lot easier: diff3 is exactly the right tool for rebasing patch 
> > #3 on top
> > of the edited #2.
> 
> Yes, that's one of the reasons I proposed storing whole files instead of
> patches. Rebasing and reordering would become much simpler (although not
> exactly "simple" if we take tree restructuring into account).

The part with tree restructuring would "simply" require a working diff3
implementation for tree changes.  Isn't that exactly what the tree
conflicts work is supposed to produce?

Re: Shelving / Checkpointing thoughts

[Branko Čibej]

On 27.08.2017 02:04, Daniel Shahaf wrote:
> Julian Foad wrote on Fri, 25 Aug 2017 21:42 +0100:
>> Let's clarify. We can mean two possible things when we say 'a series of
>> patches':
>>
>>   1. "patch versions": a series of successively better patches, all
>> attempting the same logical thing, all from the same base, and only one
>> of which is applied at any time;
>>
>>   2. a series of patches, each providing a different logical change,
>> where each patch is based on the result of applying the previous one.
>> ("quilt" is a tool for managing path series of this kind. My 'option 3'
>> (local repository) design for checkpointing could also be used in this
>> way, in a primitive way, but would not support revising earlier patches
>> in the series which is a key strength of what "quilt" can do.)
>>
>> I am talking about definition 1 ("patch versions").
>>
>> I propose patches in a series of patch versions be named "featureA-1",
>> "featureA-2", ... (This is what I do already, manually, in my own work.)
>>
>> I propose that we should not attempt to provide any special support for
>> definition 2 within this "shelving" feature; users can manage that
>> themselves by simply remembering which feature names depend on which
>> other ones, or by including some other numbering system within the names.
> Could you explain how you see this working?  In use-case #2, later patches
> in the series typically depend on earlier patches.  I don't see how that 
> use-case
> can be addressed if the patches are all implemented against deltas against
> the same base (for example, because if patch #2 in a 5-patch series is edited,
> all later patches would have to be regenerated).
>
> If that's not clear enough I'd be happy to elaborate.
>
> Cheers,
>
> Daniel
>
> P.S. I think brane's suggestion would make the "edit patch #2 in a 5-patch 
> series"
> use-case a lot easier: diff3 is exactly the right tool for rebasing patch #3 
> on top
> of the edited #2.

Yes, that's one of the reasons I proposed storing whole files instead of
patches. Rebasing and reordering would become much simpler (although not
exactly "simple" if we take tree restructuring into account).

-- Brane

Re: Shelving / Checkpointing thoughts

[Daniel Shahaf]

Julian Foad wrote on Fri, 25 Aug 2017 21:42 +0100:
> Let's clarify. We can mean two possible things when we say 'a series of
> patches':
> 
>   1. "patch versions": a series of successively better patches, all
> attempting the same logical thing, all from the same base, and only one
> of which is applied at any time;
> 
>   2. a series of patches, each providing a different logical change,
> where each patch is based on the result of applying the previous one.
> ("quilt" is a tool for managing path series of this kind. My 'option 3'
> (local repository) design for checkpointing could also be used in this
> way, in a primitive way, but would not support revising earlier patches
> in the series which is a key strength of what "quilt" can do.)
> 
> I am talking about definition 1 ("patch versions").
> 
> I propose patches in a series of patch versions be named "featureA-1",
> "featureA-2", ... (This is what I do already, manually, in my own work.)
> 
> I propose that we should not attempt to provide any special support for
> definition 2 within this "shelving" feature; users can manage that
> themselves by simply remembering which feature names depend on which
> other ones, or by including some other numbering system within the names.

Could you explain how you see this working?  In use-case #2, later patches
in the series typically depend on earlier patches.  I don't see how that 
use-case
can be addressed if the patches are all implemented against deltas against
the same base (for example, because if patch #2 in a 5-patch series is edited,
all later patches would have to be regenerated).

If that's not clear enough I'd be happy to elaborate.

Cheers,

Daniel

P.S. I think brane's suggestion would make the "edit patch #2 in a 5-patch 
series"
use-case a lot easier: diff3 is exactly the right tool for rebasing patch #3 on 
top
of the edited #2.

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Forgot the conclusion...

Julian Foad wrote:
> Johan Corveleyn wrote:
>> Still only one shelf per WC (*the* shelf)? Grouping them through
>> naming ("savepoint-1", "savepoint-2" are two shelved patches belonging
>> to the same series, but "featureA" (which was reverted) is separate
>> because it doesn't have the same prefix)?

Absolutely yes! I thought I had previously indicated so.

>> Or do we need multiple shelves with some name too?

Absolutely not. Not for any reason that I can see.

- Julian

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Johan Corveleyn wrote:
>>   * "Shelving" or "to shelve" means putting something on a shelf. There
>> is one "shelf" per WC.
>>
>>   * The thing we put on the shelf is called a "patch" or a "shelved
>> change", and is analogous to a book or a paper placed on the shelf. A
>> numbered version of a patch can be called a "checkpoint".
>>
>>   * A series of checkpoint patches is a series of "patch versions" or a
>> "checkpoint series". I think this is simpler than introducing a new term.
> 
> Ah yes, of course. Sorry, no need to invent a new term.
> 
> Just wondering then, when we create a "series of patches" that belong
> together, that have some ordering, how do we organize that?

Let's clarify. We can mean two possible things when we say 'a series of
patches':

  1. "patch versions": a series of successively better patches, all
attempting the same logical thing, all from the same base, and only one
of which is applied at any time;

  2. a series of patches, each providing a different logical change,
where each patch is based on the result of applying the previous one.
("quilt" is a tool for managing path series of this kind. My 'option 3'
(local repository) design for checkpointing could also be used in this
way, in a primitive way, but would not support revising earlier patches
in the series which is a key strength of what "quilt" can do.)

I am talking about definition 1 ("patch versions").

I propose patches in a series of patch versions be named "featureA-1",
"featureA-2", ... (This is what I do already, manually, in my own work.)

I propose that we should not attempt to provide any special support for
definition 2 within this "shelving" feature; users can manage that
themselves by simply remembering which feature names depend on which
other ones, or by including some other numbering system within the names.

> Still only one shelf per WC (*the* shelf)? Grouping them through
> naming ("savepoint-1", "savepoint-2" are two shelved patches belonging
> to the same series, but "featureA" (which was reverted) is separate
> because it doesn't have the same prefix)? Or do we need multiple
> shelves with some name too?
> 
> Just one more thought: in the namespace of shelved changes, we might
> want to reserve "svn:" or some such prefix, for internal use, to give
> us possibilities for features built upon the shelving infrastructure.

Good thought.

- Julian

Re: Shelving / Checkpointing thoughts

[Johan Corveleyn]

On Fri, Aug 25, 2017 at 4:59 PM, Julian Foad  wrote:
> Johan Corveleyn wrote:
>> On Fri, Aug 25, 2017 at 3:33 PM, Julian Foad  wrote:
> [...]>> The Checkpoint feature could add the copy-and-modify facility
> for the
>>> log message.
>>
>> Yes, maybe we'll need to have some grouping structure / namespacing in
>> the shelves for this. A "rack" or something :-). The rack carries a
>> name ("savepoints", "feature A"); a single shelf in a rack is just
>> 'svn shelve --rack "feature A"'; If I add more shelves to a rack, they
>> get numbered. [...]
> I think the terminology works best, and most in line with other tools
> (p4, hg, bzr) like this:
>
>   * "Shelving" or "to shelve" means putting something on a shelf. There
> is one "shelf" per WC.
>
>   * The thing we put on the shelf is called a "patch" or a "shelved
> change", and is analogous to a book or a paper placed on the shelf. A
> numbered version of a patch can be called a "checkpoint".
>
>   * A series of checkpoint patches is a series of "patch versions" or a
> "checkpoint series". I think this is simpler than introducing a new term.

Ah yes, of course. Sorry, no need to invent a new term.

Just wondering then, when we create a "series of patches" that belong
together, that have some ordering, how do we organize that?

Still only one shelf per WC (*the* shelf)? Grouping them through
naming ("savepoint-1", "savepoint-2" are two shelved patches belonging
to the same series, but "featureA" (which was reverted) is separate
because it doesn't have the same prefix)? Or do we need multiple
shelves with some name too?

Just one more thought: in the namespace of shelved changes, we might
want to reserve "svn:" or some such prefix, for internal use, to give
us possibilities for features built upon the shelving infrastructure.

-- 
Johan

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Johan Corveleyn wrote:
> On Fri, Aug 25, 2017 at 3:33 PM, Julian Foad  wrote:
[...]>> The Checkpoint feature could add the copy-and-modify facility
for the
>> log message.
> 
> Yes, maybe we'll need to have some grouping structure / namespacing in
> the shelves for this. A "rack" or something :-). The rack carries a
> name ("savepoints", "feature A"); a single shelf in a rack is just
> 'svn shelve --rack "feature A"'; If I add more shelves to a rack, they
> get numbered. [...]
I think the terminology works best, and most in line with other tools
(p4, hg, bzr) like this:

  * "Shelving" or "to shelve" means putting something on a shelf. There
is one "shelf" per WC.

  * The thing we put on the shelf is called a "patch" or a "shelved
change", and is analogous to a book or a paper placed on the shelf. A
numbered version of a patch can be called a "checkpoint".

  * A series of checkpoint patches is a series of "patch versions" or a
"checkpoint series". I think this is simpler than introducing a new term.

- Julian

Re: Shelving / Checkpointing thoughts

[Johan Corveleyn]

On Fri, Aug 25, 2017 at 3:33 PM, Julian Foad  wrote:
...
>> == Shelving ==
>>
>> Looks great so far. Of course a lot of challenges remain for all the
>> cases which are not yet (correctly) covered by 'svn diff' and 'svn
>> patch' (property changes, tree operations, binary files, unresolved
>> conflicts, etc.). Attaching a log message to a shelf is key, and the
>> association with changelists looks like a good approach.
>>
>> - Shelves should (eventually) support directories as "versioned
>> items". Changelists currently don't support directories.
>
> I agree with all this too. A log message is supported in the prototype
> (but not yet integrated with changelists, as discussed in another thread).

We'll need to have the discussion about what changelists should do
with directories though. Not sure what prior discussions were already
held about this, when the changelist feature was designed ...

>> - Suggestion: 'svn shelve --keep', to create a shelf (patch) in the
>> "shelf storage" but not revert it. That would enable some crude way of
>> checkpointing your work, through simple patches (which can be applied
>> later by fuzzy patching, or ...):
>>
>> work on feature A
>> svn shelve --keep --name "feature A"
>> continue work on feature A
>> turns out badly, lets go back
>> svn revert; svn unshelve "feature A"
>
> Yes; implemented now in r1806168.

Great!

> In fact this usage aligns very well with how I use patch files myself. I
> often save a series of patches to checkpoint my development of a
> feature, with names like 'foo-1.patch' ... 'foo-N.patch'.
>
> One thing I like to do in that case is to copy the log message from the
> top of the previous patch into the top of the new patch and update it.
> If Subversion could make that step easier, that would help.
>
> At present the prototype 'svn shelve' accepts a log message with '-m'
> or'-F' but doesn't provide an easy way to invoke an editor, assuming you
> don't want a log message if you don't specify one. One way to solve that
> would be by adopting the same convention as 'commit': pop up an editor
> by default and allow quitting with an empty message if desired. Another
> way could be a command-line option if we think no-log-message use cases
> should be the default. But that's a UI detail that we can leave till later.
>
> The Checkpoint feature could add the copy-and-modify facility for the
> log message.

Yes, maybe we'll need to have some grouping structure / namespacing in
the shelves for this. A "rack" or something :-). The rack carries a
name ("savepoints", "feature A"); a single shelf in a rack is just
'svn shelve --rack "feature A"'; If I add more shelves to a rack, they
get numbered.

svn shelve --keep-local --rack savepoints -> shelf "savepoints-0" (or
just "savepoints"?)
svn shelve --keep-local --rack savepoints -> shelf "savepoints-1"
svn revert
svn unshelve savepoints-0

Just thinking out loud here ...

>
>> == Checkpointing ==
>>
>> I think only "option 3" looks viable / interesting in the long run
>> (option 1, storing patches, looks a lot like simple shelving, so not
>> much more value imho). Or even a completely different approach which
>> is implemented in working copy storage (option 4? I haven't thought
>> this through, but I'm afraid of the disk space requirements, and init
>> I/O cost, of option 3 for large multi-GB working copies).
>>
>> It's very hard for me to not think of checkpoints as local branches of
>> some sort. And my users will immediately want to use them in that way.
>> In all honesty, I think we should aim for powerful local branches (and
>> think of an architecture / design / ui for that), and then think about
>> how we can perhaps start with something simpler and more limited as a
>> first step, but which goes in that direction. I.e. a more holistic
>> design around "local branches", "local commits", "checkpointing".
>> What's the big picture?
>
> Good question. While I am continuing to think about ways in which some
> larger scope towards "local branching" could play out, I don't currently
> see that ever working well for Subversion, at least not the current
> generation 1.x.
>
> Given your comments about shelving being almost enough, I now wonder if
> we should design simple checkpointing as a simple extension to shelving,
> whereby we:
>
>   * automati

Re: Shelving / Checkpointing thoughts

[Julian Foad]

Johan, thank you very much for these considered thoughts!

Johan Corveleyn wrote:
> First of all: thanks for working on shelving and checkpointing,
> Julian. These could become very important and big features. Daunting
> to take them on, but it's good to see someone having a go at it.
> 
> I've tried to read through all the docs and recent mail threads. Below
> are a couple of thoughts.
> 
> TL;DR: I'd suggest first going for a very good Shelving
> implementation, and not rushing for Checkpointing. Shelving can
> already solve some checkpointing use-cases (see below), and
> Checkpointing as a full-blown feature needs very careful thought (even
> if limited, it's risky to paint ourselves into a corner), and will
> quickly raise expectations of "local commits" or "local branching"
> (which still seem far away).

Agreed.

> == Shelving ==
> 
> Looks great so far. Of course a lot of challenges remain for all the
> cases which are not yet (correctly) covered by 'svn diff' and 'svn
> patch' (property changes, tree operations, binary files, unresolved
> conflicts, etc.). Attaching a log message to a shelf is key, and the
> association with changelists looks like a good approach.
> 
> - Shelves should (eventually) support directories as "versioned
> items". Changelists currently don't support directories.

I agree with all this too. A log message is supported in the prototype
(but not yet integrated with changelists, as discussed in another thread).

> - Suggestion: 'svn shelve --keep', to create a shelf (patch) in the
> "shelf storage" but not revert it. That would enable some crude way of
> checkpointing your work, through simple patches (which can be applied
> later by fuzzy patching, or ...):
> 
> work on feature A
> svn shelve --keep --name "feature A"
> continue work on feature A
> turns out badly, lets go back
> svn revert; svn unshelve "feature A"

Yes; implemented now in r1806168.

In fact this usage aligns very well with how I use patch files myself. I
often save a series of patches to checkpoint my development of a
feature, with names like 'foo-1.patch' ... 'foo-N.patch'.

One thing I like to do in that case is to copy the log message from the
top of the previous patch into the top of the new patch and update it.
If Subversion could make that step easier, that would help.

At present the prototype 'svn shelve' accepts a log message with '-m'
or'-F' but doesn't provide an easy way to invoke an editor, assuming you
don't want a log message if you don't specify one. One way to solve that
would be by adopting the same convention as 'commit': pop up an editor
by default and allow quitting with an empty message if desired. Another
way could be a command-line option if we think no-log-message use cases
should be the default. But that's a UI detail that we can leave till later.

The Checkpoint feature could add the copy-and-modify facility for the
log message.


> == Checkpointing ==
> 
> I think only "option 3" looks viable / interesting in the long run
> (option 1, storing patches, looks a lot like simple shelving, so not
> much more value imho). Or even a completely different approach which
> is implemented in working copy storage (option 4? I haven't thought
> this through, but I'm afraid of the disk space requirements, and init
> I/O cost, of option 3 for large multi-GB working copies).
> 
> It's very hard for me to not think of checkpoints as local branches of
> some sort. And my users will immediately want to use them in that way.
> In all honesty, I think we should aim for powerful local branches (and
> think of an architecture / design / ui for that), and then think about
> how we can perhaps start with something simpler and more limited as a
> first step, but which goes in that direction. I.e. a more holistic
> design around "local branches", "local commits", "checkpointing".
> What's the big picture?

Good question. While I am continuing to think about ways in which some
larger scope towards "local branching" could play out, I don't currently
see that ever working well for Subversion, at least not the current
generation 1.x.

Given your comments about shelving being almost enough, I now wonder if
we should design simple checkpointing as a simple extension to shelving,
whereby we:

  * automatically increment the patch suffix number,
if the latest shelved patch was made with '--keep-local';

  * also transfer the log message transfer from the previous numbered
patch and offer it for editing;

  * offer a UI shortcut to revert to the previou

Shelving / Checkpointing thoughts

[Johan Corveleyn]

First of all: thanks for working on shelving and checkpointing,
Julian. These could become very important and big features. Daunting
to take them on, but it's good to see someone having a go at it.

I've tried to read through all the docs and recent mail threads. Below
are a couple of thoughts.

TL;DR: I'd suggest first going for a very good Shelving
implementation, and not rushing for Checkpointing. Shelving can
already solve some checkpointing use-cases (see below), and
Checkpointing as a full-blown feature needs very careful thought (even
if limited, it's risky to paint ourselves into a corner), and will
quickly raise expectations of "local commits" or "local branching"
(which still seem far away).


== Shelving ==

Looks great so far. Of course a lot of challenges remain for all the
cases which are not yet (correctly) covered by 'svn diff' and 'svn
patch' (property changes, tree operations, binary files, unresolved
conflicts, etc.). Attaching a log message to a shelf is key, and the
association with changelists looks like a good approach.

- Shelves should (eventually) support directories as "versioned
items". Changelists currently don't support directories.

- Suggestion: 'svn shelve --keep', to create a shelf (patch) in the
"shelf storage" but not revert it. That would enable some crude way of
checkpointing your work, through simple patches (which can be applied
later by fuzzy patching, or ...):

work on feature A
svn shelve --keep --name "feature A"
continue work on feature A
turns out badly, lets go back
svn revert; svn unshelve "feature A"


== Checkpointing ==

I think only "option 3" looks viable / interesting in the long run
(option 1, storing patches, looks a lot like simple shelving, so not
much more value imho). Or even a completely different approach which
is implemented in working copy storage (option 4? I haven't thought
this through, but I'm afraid of the disk space requirements, and init
I/O cost, of option 3 for large multi-GB working copies).

It's very hard for me to not think of checkpoints as local branches of
some sort. And my users will immediately want to use them in that way.
In all honesty, I think we should aim for powerful local branches (and
think of an architecture / design / ui for that), and then think about
how we can perhaps start with something simpler and more limited as a
first step, but which goes in that direction. I.e. a more holistic
design around "local branches", "local commits", "checkpointing".
What's the big picture?

- After reading the "Terminology" section of
https://wiki.apache.org/subversion/SavePoints, I agree with that
document that "Savepoints" might be a better name. But don't want to
bikeshed over it ...

- In a prior mail-thread between you and Nathan Hartman the "rebasing"
problem was mentioned. In [1] you said:

> Performing an 'update' with a checkpoint series is a bigger ask than it
> might at first seem. In effect, it requires rebasing the series of
> checkpoints on the new base, which gets ugly because of the need to
> handle conflicts (which is ugly enough already in the existing
> single-depth WC).

However, that seems to only sane way to go for me. Rebasing the
checkpoints one by one, and resolving conflicts along the way. Don't
know how you'd do that though, if the checkpoints are revisions 1, 2,
3 in a local repository (with immutable history etc). This is really
something where the "local repository" technique breaks down IMHO. In
contract with DVCS's, in SVN history is immutable. But mutability is
quite important for local branches / commits.

In that sense, a series of patches is more flexible: you can still
apply them with fuzz even if applying them to a different BASE state,
and often that will "just work" (and conflicts "just" need to be
resolved).

In that light, Nathan's latest post in that thread ([2]) was also
interesting, where he suggested to store the BASE together with the
WORK for every checkpoint. I'm not sure if that's the way to go (maybe
you only need a "description of BASE", not the pristines etc), but it
made me realize: don't expect checkpoints to work for "undo 'svn
update'" if you can't restore the original BASE tree (down to the
exact mixed-revisionness).

[1] https://svn.haxx.se/dev/archive-2017-07/0302.shtml
[2] https://svn.haxx.se/dev/archive-2017-08/0064.shtml

-- 
Johan

Re: Shelving and Checkpointing support log messages

[Stefan Sperling]

On Fri, Aug 25, 2017 at 10:08:04AM +0100, Julian Foad wrote:
> Example workflow:
> 
> 1. There exists a shelved patch 'foo' with a verbose log message
> 2. 'svn unshelve foo'

There can only be one changelist per file.
What if at this point there is another changelist 'bar' which includes a
file modified by the shelved patch? Would 'unshelve' error out or change
that file's changelist to 'foo'?

I suppose it should error out and ask the user to commit or remove
the conflicting changelist first.

> 3. 'make check'
> 4. 'svn commit --changelist foo'
> 
> In this example, the unshelve creates & populates changelist 'foo', and
> the commit takes its log msg from changelist foo.
> 
> To me this seems very appealing.

Yes, indeed!

Re: Shelving and Checkpointing support log messages

[Johan Corveleyn]

On Fri, Aug 25, 2017 at 11:08 AM, Julian Foad  wrote:
> Daniel Shahaf wrote:
>> Julian Foad wrote on Thu, 24 Aug 2017 19:46 +0100:
>>> One rough edge is that "unshelving" a patch or finishing/squashing a
>>> checkpoint series simply discards the log message(s). [...]
>>
>> [...] to enable the "long log message" use-case, there should be an easy
>> "upgrade path" for the shelf's log message to become a commit log message;
>> optionally with editing first.
>>
>> Example workflow:
>>
>> 1. There exists a shelf with a verbose log message
>> 2. Unshelve that patch
>> 3. 'make check'
>> 4. 'svn commit'
>>
>> In this workflow, it would be nice for $EDITOR to be pre-filled with the
>> shelf's log message.
>>
>> The catch is, what happens if the patch is reverted [...] step #4
>> doesn't know whether it's committing the patch that had been
>> unshelved, or something else [...]
>
> A neat solution could be to integrate Shelving with Changelists in such
> a way that Unshelving means converting a patch to a changelist, and a
> changelist grows support for storing a log message.
>
> This would be a two-way integration: shelving a changelist would then
> mean converting the changelist to a patch and reverting the changelist
> from the WC.
>
> This would be more in line with how, for example, Perforce defines a
> changelist: there it is a change, whether committed or shelved or in one
> or two other states; and includes a log message and some other metadata.
>
> (In fact, right from when we introduced "changelists", I have wanted
> them to support storing a log message, without even thinking about the
> possibility of shelving.)
>
> Example workflow:
>
> 1. There exists a shelved patch 'foo' with a verbose log message
> 2. 'svn unshelve foo'
> 3. 'make check'
> 4. 'svn commit --changelist foo'
>
> In this example, the unshelve creates & populates changelist 'foo', and
> the commit takes its log msg from changelist foo.
>
> To me this seems very appealing.
>
> I outlined this idea in the section "Integrate Shelving with
> Changelists" in the "Shelving-Checkpointing Dev" doc:
> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.ay5h5pfumpv8
>
> What do you think?

+1, that looks like a good approach.

In the google doc you mentioned some issues with this:

* a changelist can include unmodified paths
  should a shelved patch also include unmodified paths?

-> I guess that would be the best solution for this. The shelve should
keep (in metadata) a list of paths that were part of the changelist.

* a path can appear in multiple shelves, but only in one changelist
  what to do when unshelving if a path clashes?

-> Hmm, that's a hard one. No idea for now, except perhaps extending
changelist to make it possible for paths to be part of multiple
changelists too.

FWIW: IntelliJ IDEA gives you a warning when you start editing a file
that's part of some changelist, while your default changelist is set
to another changelist ("File from non-active changelist is modified").
The user can "Move changes", "Switch changelist" or "Ignore" (though
it's not clear to me what "Ignore" does ... the file doesn't become
part of the "active changelist" suddenly, or of both, so this seems
equal to "Move changes").

* any extensions should apply uniformly to both shelving and changelists
  if a shelved change supports a log msg, a changelist should support
a log msg too; this would be a good enhancement for changelists anyway
even without shelving

-> Yup, sounds good to me :)


I see one more important caveat: changelists currently don't support
directories as members.

I have some more thoughts on shelves and checkpoints, but I'll put
them in a separate thread.

-- 
Johan

Re: Shelving and Checkpointing support log messages

[Julian Foad]

Daniel Shahaf wrote:
> Julian Foad wrote on Thu, 24 Aug 2017 19:46 +0100:
>> One rough edge is that "unshelving" a patch or finishing/squashing a
>> checkpoint series simply discards the log message(s). [...]
> 
> [...] to enable the "long log message" use-case, there should be an easy
> "upgrade path" for the shelf's log message to become a commit log message;
> optionally with editing first.
> 
> Example workflow:
> 
> 1. There exists a shelf with a verbose log message
> 2. Unshelve that patch
> 3. 'make check'
> 4. 'svn commit'
> 
> In this workflow, it would be nice for $EDITOR to be pre-filled with the
> shelf's log message.
> 
> The catch is, what happens if the patch is reverted [...] step #4
> doesn't know whether it's committing the patch that had been
> unshelved, or something else [...]

A neat solution could be to integrate Shelving with Changelists in such
a way that Unshelving means converting a patch to a changelist, and a
changelist grows support for storing a log message.

This would be a two-way integration: shelving a changelist would then
mean converting the changelist to a patch and reverting the changelist
from the WC.

This would be more in line with how, for example, Perforce defines a
changelist: there it is a change, whether committed or shelved or in one
or two other states; and includes a log message and some other metadata.

(In fact, right from when we introduced "changelists", I have wanted
them to support storing a log message, without even thinking about the
possibility of shelving.)

Example workflow:

1. There exists a shelved patch 'foo' with a verbose log message
2. 'svn unshelve foo'
3. 'make check'
4. 'svn commit --changelist foo'

In this example, the unshelve creates & populates changelist 'foo', and
the commit takes its log msg from changelist foo.

To me this seems very appealing.

I outlined this idea in the section "Integrate Shelving with
Changelists" in the "Shelving-Checkpointing Dev" doc:
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.ay5h5pfumpv8

What do you think?

- Julian

Re: Shelving and Checkpointing support log messages

[Daniel Shahaf]

Julian Foad wrote on Thu, 24 Aug 2017 19:46 +0100:
> I think we will encounter two distinct styles of usage:
> 
>   * "just put whatever I have aside, I haven't time to think about it
> now, I'll come back to it later" -- no description or just a very brief
> hint, and perhaps not even a name;
> 
>   * "this is a patch I have prepared" with a carefully written log message.
> 

Agreed.

> One rough edge is that "unshelving" a patch or finishing/squashing a
> checkpoint series simply discards the log message(s). I welcome your
> thoughts.

I doubt anyone would store a verbose long message on a shelved
patch if applying the patch would discard the log message.  People would
grow the habit of storing the log message out of band.

Conversely, to enable the "long log message" use-case, there should be an easy
"upgrade path" for the shelf's log message to become a commit log message;
optionally with editing first.

Example workflow:

1. There exists a shelf with a verbose log message
2. Unshelve that patch
3. 'make check'
4. 'svn commit'

In this workflow, it would be nice for $EDITOR to be pre-filled with the
shelf's log message.

The catch is, what happens if the patch is reverted (with 'svn revert'
or with an equivalent '/usr/bin/patch -R') before step 4.  In that case,
the user wouldn't want the shelf's message to be presupplied.

The problem is that between #2 and #4 we have hidden state: step #4
doesn't know whether it's committing the patch that had been
unshelved, or something else --- and if we have to guess, we'll
sometimes guess wrongly.

(That's all I have for now...)

Cheers,

Daniel

Shelving and Checkpointing support log messages

[Julian Foad]

I had some feedback that the ability to give a short description or log
message seemed important, so I have added that feature on the prototype
branches.

$ svn shelve --help
...
  1. Save the local changes in the given PATHs to a patch file, and
 revert those changes from the WC. If a log message is given with
 '-m' or '-F', include it at the beginning of the patch file.
...

and similarly for the two designs of 'svn checkpoint'.

The respective 'list' functions then display the message (currently the
first line of it) of each patch or checkpoint.

I think we will encounter two distinct styles of usage:

  * "just put whatever I have aside, I haven't time to think about it
now, I'll come back to it later" -- no description or just a very brief
hint, and perhaps not even a name;

  * "this is a patch I have prepared" with a carefully written log message.

One rough edge is that "unshelving" a patch or finishing/squashing a
checkpoint series simply discards the log message(s). I welcome your
thoughts.

- Julian

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Tue, Aug 1, 2017 at 9:19 AM, Julian Foad  wrote:
> Keep in mind that a revision -- or any versioned state -- can be
> thought of in two ways: as a complete snapshot of the tree, or as
> representing a change relative to the previous snapshot.
>
> First let us be sure we understand that updating *is* rebasing. To
> explain this, think about a plain WC with no checkpointing. The WC
> has a base tree and a working tree. The work you do in a working copy
> is to create a change based on the base. An 'update' is a request to
> update the base and also to adjust the working version so that it
> represents the 'same' change that it previously did, except the
> change is now against the new base. Hence update is 'rebasing' your
> local change. The adjustment of the old change (against the old base)
> to a new change (against the new base) is accomplished by a merge.
>
> Illustration: a plain old WC based on r20 is like a one-commit local
> branch based on r20.
>
> [ repo -(r10)--(r20)--(r30)--(r40)---
> [||
> [||
> [ WC   (BASE)-(WORK)
>
> [(X) represents a tree snapshot. WORK means the working/on-disk state
> in the WC, including what libsvn_wc calls 'actual'.]
>
> Updating to r30 involves merging (r20:30) with (rBASE:WORK) to create
> (WORK'), and setting the new base to r30.
>
> [ repo -(r10)--(r20)--(r30)--(r40)---
> [   ||
> [   ||
> [ WC  (BASE')-(WORK')

I appreciate that you've taken the time and effort to explain this,
complete with graphs. I suppose I should have known that an update
rebases the working copy onto the new revision, but for whatever reason
I was under the impression that it was the other way around, that the
incoming changes get applied on top of the working copy.

The only drawback of the rebase is that update is a one-way operation,
with no 'undo' mechanism. It's the only place in Subversion where you
could potentially lose a very particular configuration of your files,
which may have taken some tricky manipulations to arrive at, and the
nature of update (being the action that immediately precedes a commit)
is such that your intention was probably to save those bits, not change
them. Currently in such a situation, the user should mitigate that by a
svn copy to a new branch, or adopt a policy of branching for everything
and avoid this issue altogether. Personally I've been known to copy the
contents of my working copy to a temp directory before performing an
update, just in case. :-) But the premise of version control is that
you should never fear losing anything. That makes update the perfect
use case and showcase for a checkpoint feature, even if the user never
takes advantage of checkpoints otherwise.

Referring to this last graph:

> (b) Keep the existing checkpoints based on the older base, and bring
> in the new base to be used only for new checkpoints. For this
> illustration, assume we had no outstanding working changes before
> running the update. We merge the incoming changes with the *complete*
> checkpoint series and record the result as (Pn').
>
> [ repo -(r10)--(r20)--(r30)--(r40)---
> [|| ||
> [|| ||
> [|  (P0)  (P0')
> [|\   \
> [ WC | \   \
> [|  \   \
> [| (P1)-...-(Pn)--(Pn'=BASE)
> [ \
> [  \_(WORK')
>
> The WC now contains two "base" snapshots (P0 and P0') copied from the
> repository, and (n) checkpoint snapshots, and a merged checkpoint.

If checkpoints are implemented using a hidden repo, is it feasible to
place two directories, BASE and WORK, at its root, and save the BASE
with each checkpoint? If so, Subversion's atomic repo-wide commits
would automatically solve the issue of keeping each checkpoint
synchronized with its base, since both would be saved together. The
repo's temporal compression would mean that very little additional data
would be saved, in comparison to the size of BASE.

If this is feasible, then I would treat checkpoints as a sort of time
line, rather than a private branch. That is, each action against the
checkpoints, whether to create a new checkpoint, perform an update, or
rollback to an earlier checkpoint, would create a new checkpoint with
the contents of the current BASE and WORK. So, rolling *back* (to an
earlier, in time, checkpoint) could actually rol

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

Nathan Hartman wrote:
> Julian Foad wrote:
> Performing an 'update' with a checkpoint series is a bigger ask than it
> might at first seem. In effect, it requires rebasing the series of
> checkpoints on the new base, which gets ugly because of the need to
> handle conflicts (which is ugly enough already in the existing
> single-depth WC).
> 
> Why does update with a checkpoint require rebasing? [...]

[tl;dr: Update fundamentally involves rebasing the working change.
Rebasing all the checkpoints as well is one approach but has drawbacks.
We examine other approaches.]


==

Keep in mind that a revision -- or any versioned state -- can be thought
of in two ways: as a complete snapshot of the tree, or as representing a
change relative to the previous snapshot.

First let us be sure we understand that updating *is* rebasing. To
explain this, think about a plain WC with no checkpointing. The WC has a
base tree and a working tree. The work you do in a working copy is to
create a change based on the base. An 'update' is a request to update
the base and also to adjust the working version so that it represents
the 'same' change that it previously did, except the change is now
against the new base. Hence update is 'rebasing' your local change. The
adjustment of the old change (against the old base) to a new change
(against the new base) is accomplished by a merge.

Illustration: a plain old WC based on r20 is like a one-commit local
branch based on r20.

[ repo -(r10)--(r20)--(r30)--(r40)---
[||
[||
[ WC   (BASE)-(WORK)

[(X) represents a tree snapshot. WORK means the working/on-disk state in
the WC, including what libsvn_wc calls 'actual'.]

Updating to r30 involves merging (r20:30) with (rBASE:WORK) to create
(WORK'), and setting the new base to r30.

[ repo -(r10)--(r20)--(r30)--(r40)---
[   ||
[   ||
[ WC  (BASE')-(WORK')


==

Now, what is a checkpoint series? We are exploring a couple of different
definitions.

1. The saved-patches definition: a series of patches, each representing
the state of the WC at the time it was saved, enabling an earlier state
of the WC to be recovered. (The 'option 1' design implements the patches
part of this, and may later be extended to record the WC base.)

[ repo -(r10)--(r20)--(r30)--(r40)---
[||
[||
[ WC   (BASE)
[ _(P1)
[  \\\_...
[   \\_(Pn)
[\_(WORK)

2. The local-branch definition: a series of changes, rather like
revisions in a repository (and implemented as such in 'option 3'
design). The zero'th checkpoint "P0" is (a copy of) what was the
original WC base tree. Then there are successive snapshots P1, P2... Pn
and then a working change which is currently being edited. This behaves
like you would expect from a private, local branch in Subversion based
on the original base point from the original repository.

[ repo -(r10)--(r20)--(r30)--(r40)---
[||
[||
[ WC(P0)-(P1)-...-(Pn=BASE)
[   \
[\_(WORK)


Each of these two definitions of checkpointing suggests its own
conceptual approaches to updating.

1. For saved-patches checkpointing, we might update the WC base and
working version, and start saving any new checkpoints relative to the
new base.

[ repo -(r10)--(r20)--(r30)--(r40)---
[|| ||
[|| ||
[ WC   (BASE)  (BASE')
[_(P1) \\_... <= (Pn+1) will go here
[ \\\_...   \
[  \\_(Pn)   \
[   \ \
[\_(WORK)  \_(WORK')

A consequence of different checkpoints having different bases is that
rolling back (or forward) to a checkpoint based on a different base
would require one of:

  * apply the patch using fuzzy matching
  * (if online) rebase the desired checkpoint at this time
  * (if online) update the WC base to the checkpoint's recorded base

An alternative would be that updating attempts to rebase every recorded
checkpoint patch, but see 2(a) below.

2. For local-branch checkpointing, there are at least two ways we could
update:

(a) Update P0 and successively rebase all the changes represented by the
checkpoints: rewrite P1 (by merging) to be based on P0, rebase P2 on P1,
and so on.

[ repo -(r10)--(r20)--(r30)

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Fri, Jul 28, 2017 at 8:19 AM, Julian Foad  wrote:

> Nathan Hartman wrote:
> All considered, it feels to me like we have neither the collective will
> nor the resources to take this in the direction of full local branching.
> I can't commit myself to that. The present work should strive to keep
> that option open, largely because a successful open-source software
> system is one that others can adapt to their needs later on.
>
> Rather, for me, this line of thought helps to clarify the differences
> between what we are and are not creating.
>

Agreed. Just to emphasize, I don't suggest going the route of full
local branching today. That involves challenges that require some
serious thought, not only implementation but also user perspective:
making it easy to learn and easy to use, like the rest of Subversion.


> > (2) [...] "svn update" should perform an automatic implicit
> > "checkpoint save" [...] If an update turns ugly, you will be able to do
> > that 3-way diff I was just talking about.
>
> Or you could revert the update. Yes, I would love to have this feature too.
>
> Performing an 'update' with a checkpoint series is a bigger ask than it
> might at first seem. In effect, it requires rebasing the series of
> checkpoints on the new base, which gets ugly because of the need to
> handle conflicts (which is ugly enough already in the existing
> single-depth WC).
>

Why does update with a checkpoint require rebasing? There must be a
large gap in my understanding here. Forgive me if I misunderstood, but
I thought a checkpoint is a snapshot of what the working copy looked
like at the time it was taken, to provide a sort of high-level
undo/redo capability. Suppose I did a checkout at 8:00 AM and then did
some work, taking snapshots of what my working copy looked like at
9:00 AM, 11:00 AM, and 2:00 PM. At 2:30 PM I invoke svn update. I
would expect to end up with four snapshots: 9:00 AM, 11:00 AM, 2:00
PM, 2:30 PM; and I would expect the working copy, after the update, to
be the same as though I checked out at 8:00 AM, worked until 2:30 PM,
and did the update, exactly as it occurs today. In other words, I
would expect my snapshots to remain untouched and I would expect the
update to be the "next step" in the working copy contents' linear
evolution. I could potentially continue working and taking snapshots,
and make subsequent updates as well. The implementation would probably
have to record the BASE at the time of each checkpoint, so that if you
rollback to it, passing through an "update" point, it could restore
that as the working copy's BASE along with restoring the working
copy's contents to those of the snapshot.


> Initially, I suggest 'update' and 'switch' will be incompatible with
> checkpoints. The user must first squash the checkpoints.
>

Agreed. This is reasonable for a first implementation.

As an enhancement, on running 'update', Subversion could 'squash' the
> checkpoints and shelve the resulting patch (but without reverting the
> WC). That would be enough to allow better manual 'undo' if the update
> went badly and if the user can remember the previous base state
> (revision/revisions). If the shelving function would also save a
> description of the base state, so much the better.
>

I don't like the idea of mixing checkpoints and updates with shelving.
It seems too kludgy to me because shelving is a way to temporarily
interrupt work and come back to it later, whereas updating is a linear
step forward, not an interruption. I propose this alternative: "svn
checkpoint squash" will in effect discard all checkpoints and take a
new checkpoint of the working copy. Following my earlier example of
four checkpoints, if I squash at 4:00 PM, I'll end up with just one
checkpoint with the 4:00 PM timestamp, identical to my working copy.
If "update" performs this squash operation (or simply takes a
checkpoint if none exists), then I believe it's a more logical
solution.


> > [...] I suggest ability to specify a log message [...]
>
> A log message option is intentionally omitted for now. My thinking is
> that specifying a log message reinforces the idea that each checkpoint
> commit will eventually become a central-repo commit, and with that in
> mind, the user's expectations would be a little different.


Fair enough. I'm in 100% agreement that this implementation should
focus on simplicity and reliability.

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

Nathan Hartman wrote:
> Some thoughts (probably too many)...

Not at all, this is fantastic, right where I want us to be, and very
helpful.

> I agree that 'init' and 'finish' should happen automatically.
> [...] As for when the 'finish' should occur, [...] probably not
> until the results have been squashed and committed.
> 
> As far as what to name the commands, if the "checkpoint" command is
> changed to "local" and the "save" subcommand is changed to "commit" then
> you get:
> 
> 1. svn local commit
> 2. svn local revert
> 3. svn local rollback
> 4. svn local list|log
> 5. svn local squash
> 6. svn commit   <--- to the central repository
> 
> I think users will immediately start getting ideas about what this
> means. Is that a good thing? It depends on what the goals are for
> checkpoints, how checkpoints are intended to fit into the larger
> workflow, and strategy for the future in general.

Excellent: this shows really clearly one way of thinking about
checkpointing.

> If checkpoints are intended to be merely a convenience with a limited
> purpose, then I would call the command "checkpoint" and "checkpoint
> save" to avoid confusion and make documentation, teaching, and learning
> easier.
> 
> If checkpoints are intended to have a much wider application, eventually
> providing an answer to git's/DVCSs' ability to work offline and/or to do
> personal work on projects to which one lacks commit privileges, then I
> would use the shorter "local" [...] and try to keep the sub-sub-commands
> as close in semantics and function as possible to Subversion's original
> command set.

Agreed.

> [...] Working on the
> local machine essentially means working on a private branch. With that
> in mind, to answer questions such as: whether a status or diff should be
> against the original base or against the checkpoint, such questions
> should always be answered consistently with how server side operations
> are carried out today.

Agreed.

> [...] if this feature is supposed to have such broad scope [...]
> users will have immediate sky-high expectations [...]; if that
> kind of future appeals to enough people, then the present work should
> [maintain] such a parallel between this new "local branching" and the
> original server-side branching.

All considered, it feels to me like we have neither the collective will
nor the resources to take this in the direction of full local branching.
I can't commit myself to that. The present work should strive to keep
that option open, largely because a successful open-source software
system is one that others can adapt to their needs later on.

Rather, for me, this line of thought helps to clarify the differences
between what we are and are not creating.

> Some more thoughts...
> 
> (1) It will eventually be necessary to create a new "URL specifier" or
> some sort of syntax to direct an operation into the checkpoint repo.
> This will make it possible to diff between any permutation of working
> copy, central repository, and local checkpoints, which will be crucial
> once users begin to use checkpoints as local branches. I can see a use
> case where a 3-way diff between all three is useful.

I'm not yet sure how important is diffing (2-way and 3-way) between
central repo and checkpoints. If we treat checkpointing as intermediate
WC states rather than as local branching then providing a syntax for
these kinds of operation is arguably less important. It still would be
of some use, just as 'rBASE' is useful, but not that 'rWORKING' has
always been missing from the UI despite a filed request for it, so there
is no syntax to request "diff -rWORKING:BASE" for example.


> (2) [...] "svn update" should perform an automatic implicit
> "checkpoint save" [...] If an update turns ugly, you will be able to do
> that 3-way diff I was just talking about.

Or you could revert the update. Yes, I would love to have this feature too.

Performing an 'update' with a checkpoint series is a bigger ask than it
might at first seem. In effect, it requires rebasing the series of
checkpoints on the new base, which gets ugly because of the need to
handle conflicts (which is ugly enough already in the existing
single-depth WC).

Initially, I suggest 'update' and 'switch' will be incompatible with
checkpoints. The user must first squash the checkpoints.

As an enhancement, on running 'update', Subversion could 'squash' the
checkpoints and shelve the resulting patch (but without reverting the
WC). That would be enough to allow better manual 'undo' if the update
went badly and if the user can rememb

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Thu, Jul 27, 2017 at 7:56 AM, Julian Foad  wrote:

> What can we learn from this mock-up? Here are some of my thoughts

(snip)

> If I checkpoint my changes, do I then want to see 'status' and 'diff'
>
against the original base or against the checkpoint? Both are useful.
>
(snip)

> Probably 'checkpoint init' should not be needed, and instead running
> 'checkpoint save' should initialize the checkpointing repo if not
> already done.
>
> Probably 'checkpoint finish|uninit' should not be needed, and instead
> should happen automatically when the user either commits or reverts the
> entire set of changes.
>

Some thoughts (probably too many)...

I agree that 'init' and 'finish' should happen automatically. Creation and
destruction of a hidden repository (or any data structure) is an
implementation detail that could change in the future and should not be
leaked. As you've already said, the 'init' should occur the first time a
checkpoint operation is done. As for when the 'finish' should occur, that
is a question I've been grappling with... I would say, probably not until
the results have been squashed and committed.

As far as what to name the commands, if the "checkpoint" command is changed
to "local" and the "save" subcommand is changed to "commit" then you get:

1. svn local commit
2. svn local revert
3. svn local rollback
4. svn local list|log
5. svn local squash
6. svn commit   <--- to the central repository

I think users will immediately start getting ideas about what this means.
Is that a good thing? It depends on what the goals are for checkpoints, how
checkpoints are intended to fit into the larger workflow, and strategy for
the future in general.

If checkpoints are intended to be merely a convenience with a limited
purpose, then I would call the command "checkpoint" and "checkpoint save"
to avoid confusion and make documentation, teaching, and learning easier.

If checkpoints are intended to have a much wider application, eventually
providing an answer to git's/DVCSs' ability to work offline and/or to do
personal work on projects to which one lacks commit privileges, then I
would use the shorter "local" (unless someone can think of a better,
shorter, and more descriptive name) and try to keep the sub-sub-commands as
close in semantics and function as possible to Subversion's original
command set. Even though the word "local" is technically a command, we
pretend it's more of a mode specifier that redirects what would normally be
server side operations to the local machine instead. Working on the local
machine essentially means working on a private branch. With that in mind,
to answer questions such as: whether a status or diff should be against the
original base or against the checkpoint, such questions should always be
answered consistently with how server side operations are carried out
today. Of course, if this feature is supposed to have such broad scope,
there is the (not insignificant) issue that users will have immediate
sky-high expectations that will take significant additional time and effort
to fulfill, so clearly this is not by any means an immediate goal; it's a
*possible* future direction, and if that kind of future appeals to enough
people, then the present work should try to be compatible with that future
by maintaining (to the extent possible) such a parallel between this new
"local branching" and the original server-side branching.

Some more thoughts...

(1) It will eventually be necessary to create a new "URL specifier" or some
sort of syntax to direct an operation into the checkpoint repo. This will
make it possible to diff between any permutation of working copy, central
repository, and local checkpoints, which will be crucial once users begin
to use checkpoints as local branches. I can see a use case where a 3-way
diff between all three is useful.

(2) Once checkpoints are working reliably, I suggest that "svn update"
should perform an automatic implicit "checkpoint save" (or whatever the
command is called) before attempting to fetch from the repository and merge
with the working copy. This will add a big safety net to what can be a
dreaded process. If an update turns ugly, you will be able to do that 3-way
diff I was just talking about.

(3) Eventually, squashing should be optional; that said, it's not feasible
to checkout a WC from a trunk that has lots of activity on it, work locally
for weeks, and then expect to replay the local commits onto the trunk
successfully. So I suggest this alternative: You can checkout a working
copy and make as many checkpoints as you want. When ready to commit, one of
the following happens:

(a) you squash, update, and commit (which will turn into an i

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

What can we learn from this mock-up? Here are some of my thoughts.

* Status and diff against last incremental change vs. against the
'commit base'.

* The 'init' and 'finish' work flow should be simpler.

* s/w architecture: exporting from WC, changing the WC base, ...


== Status and diff against last incremental change vs. against the
'commit base'

If I checkpoint my changes, do I then want to see 'status' and 'diff'
against the original base or against the checkpoint? Both are useful.

This is completely analogous to working on a branch. Sometimes I want to
review just my latest change on the branch, and sometimes I want to
review what I have changed in total on the branch, in other words what I
will be merging back to the parent branch. In this regard, checkpointing
is like a local branch.

For backward compatibility with Subversion's user interface, we have to
choose one base or the other as the 'default' diff/status base. Ideally
we will make both bases support exactly the same enquiry features, and
the selection of the default base will be merely a user-interface
convenience.

(Long ago I drafted a patch to make it easy to run a diff against the
merge base, when working on a branch, because doing this useful task
manually was fiddly. The UI looked like 'svn diff --against-merge-base'
or some such option, nothing more needed to be specified, and it invoked
the merge base-finding code and then ran a diff against the found base.)

Diff and status are the obvious commands which compare against a base,
where choosing the base will be appropriate. I need to check what other
commands it might also apply to.

Commands that run back through history, especially 'log' and 'blame',
also need some adjustment. In the mock-up right now, these just report
what is in the local repo. We might instead prefer 'log' and 'blame' to
show revisions from the original repository (assuming it's online), so
they keep their original meaning and ignore checkpoints. Or we might
prefer them to show both the checkpoints and the original repository
revisions.


== The 'init' and 'finish' work flow should be simpler.

Probably 'checkpoint init' should not be needed, and instead running
'checkpoint save' should initialize the checkpointing repo if not
already done.

Probably 'checkpoint finish|uninit' should not be needed, and instead
should happen automatically when the user either commits or reverts the
entire set of changes.

The 'checkpoint squash' command is provided only for experimentation.
Squashing back to a plain WC with local mods is something that happens
as part of 'finish', and would be needed within a future 'checkpoint
commit' that commits everything in one revision. I don't think it is
needed separately.


== s/w architecture: exporting from WC, changing the WC base, ...

Some APIs that would be useful do not currently exist.

First, I wanted to copy the WC base into a new repository. Ideally the
WC will expose standard 'editor' API to export the base, and we would
pipe this into a commit editor. At the moment, the 'export' command
instead uses the WC's 'walk status' API with a set of callbacks that
write the exported data onto disk, where as 'export' from a repository
uses a standard 'editor' with a different set of callbacks that write
the exported data onto disk.

The 'squash' code is implemented by updating the WC back to r1 in the
checkpoints repo, and then merging all the checkpoints repo content
(r1:HEAD) into the WC working state, without mergeinfo. Instead, it
would be nice to have an API that directly updates the WC base back to
r1 while adjusting the 'scheduling' of each path (add/del/modify) so as
to keep the WC working state constant. (That is, 'rebasing' the WC.)


Your thoughts, anyone?

In particular, how well is the work flow shaping up, if we assume we
will eliminate the 'init' and teach 'revert' and 'commit' to do the
'finish' actions automatically?

- Julian


On 26/07/17 16:13, Julian Foad wrote:
> I committed an initial prototype for checkpointing backed by a local
> repo embedded in the WC (the "option 3" design), on the
> "shelve-checkpoint3" branch.
> 
> http://svn.apache.org/r1803046 and follow-ups.
> 
> Here is the help output:
> [[[
> $ svn checkpoint --help
> checkpoint: Checkpoint the local changes.
> usage: 0. checkpoint init
>1. checkpoint save
>2. checkpoint revert
>3. checkpoint rollback NUMBER
>4. checkpoint list|log
>5. checkpoint squash
>6. checkpoint finish|uninit
[...]
> 
> If you read this far and get a chance to try it out, I'd love to hear
> your thoughts on the overall shape and the big-picture possibilities. (I
> am not so concerned about details at this stage, but mention those too
> and hopefully they will spark further thoughts.)

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

Nathan Hartman wrote:
> Julian Foad wrote:
> I committed an initial prototype for checkpointing backed by a local
> repo embedded in the WC (the "option 3" design), on the
> "shelve-checkpoint3" branch.
> 
> http://svn.apache.org/r1803046 and follow-ups.
> 
> Woohoo! Very exciting! 
> 
> I didn't quite understand the difference between uninit and finish,

They're synonyms. I was just trying out different names to see how they
sound.

> but
> that's okay. I'll re-read more carefully soon. More important to try it.
> I'll be back with thoughts...

Looking forward to hearing them. Thanks!

- Julian

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Wed, Jul 26, 2017 at 11:13 AM, Julian Foad  wrote:

> I committed an initial prototype for checkpointing backed by a local
> repo embedded in the WC (the "option 3" design), on the
> "shelve-checkpoint3" branch.
>
> http://svn.apache.org/r1803046 and follow-ups.
>

Woohoo! Very exciting!

I didn't quite understand the difference between uninit and finish, but
that's okay. I'll re-read more carefully soon. More important to try it.
I'll be back with thoughts...

Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

I committed an initial prototype for checkpointing backed by a local
repo embedded in the WC (the "option 3" design), on the
"shelve-checkpoint3" branch.

http://svn.apache.org/r1803046 and follow-ups.

Here is the help output:
[[[
$ svn checkpoint --help
checkpoint: Checkpoint the local changes.
usage: 0. checkpoint init
   1. checkpoint save
   2. checkpoint revert
   3. checkpoint rollback NUMBER
   4. checkpoint list|log
   5. checkpoint squash
   6. checkpoint finish|uninit

  0. Initialize checkpointing.
 ### WC must be a checkout of r1 of repo root
 ### required before using other checkpointing commands
  1. Save the working state as a new checkpoint.
  2. Revert the working state to the current checkpoint.
  3. Roll back the working state to checkpoint NUMBER.
  4. List all checkpoints.
  5. Squash all checkpoints to just a base and working state.
  6. Return to a plain WC based on the original repository.
 ### doesn't yet delete the checkpoint repo
]]]

There are more subsubcommands there than we would probably want in a
final user interface.

Here is some demo output.

[[[
$ . demo-setup.sh  # creates a repo and wc with three text files
Adding config.txt
Adding hello.txt
Adding old.txt
Transmitting file data ...done
Committing transaction...
Committed revision 1.

$ svn checkpoint init
creating repo
copying base
Skipped '.svn/.svn'
Adding config.txt
Adding hello.txt
Adding old.txt
Committing transaction...
switching base

$ echo foo >> hello.txt

$ svn checkpoint save
Sendinghello.txt
Transmitting file data .done
Committing transaction...
saved checkpoint 1

$ svn checkpoint list
* 1 2017-07-26T13:36

  0 2017-07-26T13:36


$ svn st

$ svn checkpoint finish
checkpointing any outstanding changes
squashing checkpoints to a working state
  (squash: updating to r1)
Updating '.':
Uhello.txt
Updated to revision 1.
  (squash: merging checkpointed changes into WC)
--- Merging r2 into '.':
Uhello.txt
pruning checkpoint 1
switching back to original base
destroying the checkpoints repo
  (destroying the repo is not implemented yet)

$ svn st
M   hello.txt

]]]

To commit the set of changes to the original repo, you use 'finish'
(which does a 'squash' followed by 'relocate'), giving a normal modified
WC state, and use a normal commit from there.

If you read this far and get a chance to try it out, I'd love to hear
your thoughts on the overall shape and the big-picture possibilities. (I
am not so concerned about details at this stage, but mention those too
and hopefully they will spark further thoughts.)

- Julian


On 2017-07-14, Julian Foad wrote:
> I committed an initial prototype for checkpointing on the
> "shelve-checkpoint" branch.  http://svn.apache.org/r1801942
> 
> This prototype is of the "store snapshot patches" ("option 1") design.
[...]

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

Hi Nathan. Thanks for your thoughts.

Nathan Hartman wrote:
> Julian Foad wrote:
> Diffing checkpoints is quite hard (well, needs a patch-arithmetic
> library) in this design. Even just determining whether two patches
> are "identical". That might be a good reason for me to try the
> alternative design "store checkpoints in a local repository"
> ("option 3") next.
> 
> My thinking is that options 1 or 2 will eventually grow into a redundant
> implementation of a VCS within svn to handle various revisions of the
> working copy. That will most likely be faster to get off the ground, but
> results in redundant code, or refactoring existing code to do the patch
> arithmetic, etc. Option 3 requires reinventing the client outright but
> gives more flexibility in the long run.

Yes, this is a serious consideration. Re-using the repository design
gives not only flexibility but the reliability and maintainability that
comes from sharing a single implementation of the functionality (DRY). I
am going to look further into this approach next.

> I was thinking that svn switch needs to be handled, or chaos would ensue
> if someone does some work, makes some shelves and checkpoints, and then
> does svn switch to a different path. A solution could be to remember the
> checkout path and revision with each shelve/checkpoint. If the user
> performs svn switch and later wants to reinstate a shelved working copy,
> the client will notice that it can't be unshelved until the workspace is
> switched back to the correct path and error out with a descriptive
> message. The user will then svn switch to the correct branch and reissue
> the unshelve command.

It's not extremely unusual to want to apply some changes onto a
different branch, but I agree it's probably best to record the base path
used, and for the 'svn' CLI to assume this might be a mistake and so
warn or error and require 'force' to unshelve to a different path. (A
more interactive client could offer a better UX.)

I have now listed this suggestion under 'extensions', and promoted
'recording the base state' into the 'good usability' column of the table
rather than 'no'.

> The normal rules apply, which mean that if there
> are uncommitted changes in the working copy, the user will have to
> either commit them or shelve them before performing the svn switch.
> [...]

Actually, 'switch' normally allows uncommitted changes in just the same
way that 'update' does.

- Julian

Re: [RFC] Shelving and Checkpointing

[Nathan Hartman]

On Fri, Jul 14, 2017 at 7:56 AM, Julian Foad  wrote:
> Diffing checkpoints is quite hard (well, needs a patch-arithmetic library) in 
> this design. Even just determining whether two patches are "identical". That 
> might be a good reason for me to try the alternative design "store 
> checkpoints in a local repository" ("option 3") next.

My thinking is that options 1 or 2 will eventually grow into a redundant 
implementation of a VCS within svn to handle various revisions of the working 
copy. That will most likely be faster to get off the ground, but results in 
redundant code, or refactoring existing code to do the patch arithmetic, etc. 
Option 3 requires reinventing the client outright but gives more flexibility in 
the long run.

I was thinking that svn switch needs to be handled, or chaos would ensue if 
someone does some work, makes some shelves and checkpoints, and then does svn 
switch to a different path. A solution could be to remember the checkout path 
and revision with each shelve/checkpoint. If the user performs svn switch and 
later wants to reinstate a shelved working copy, the client will notice that it 
can't be unshelved until the workspace is switched back to the correct path and 
error out with a descriptive message. The user will then svn switch to the 
correct branch and reissue the unshelve command. The normal rules apply, which 
mean that if there are uncommitted changes in the working copy, the user will 
have to either commit them or shelve them before performing the svn switch. A 
--force option could allow the user to unshelve the work into the wrong path, 
possibly with conflicts (but maybe this is intentional).

>> 
>>> References:
>>>[1] Shelving-Checkpointing Dev doc. (J Foad)
>>> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#
>>>  

Re: Checkpointing - is not Quilt

[Julian Foad]

Branko Čibej wrote:

In order to use checkpoints as a sort of local commit, you do have to be
able to push a series of commits to the server. You'd distinguish
explicit checkpoints (with log messages) and implicit/automatic
checkpoints; only the explicit ones would constitute separate commits.


That would be lovely. It is beyond what I see as basic checkpointing 
functionality (which is more like a local 'undo' mechanism while 
composing one commit).



In order to support that, you'd have to have a way to rebase the whole
checkpoint stack onto current HEAD (or at least to a non-conflicting
state). Ideally you'd also support operations such as checkpoint
reordering, (local) squashing, removal, etc.


Yes. I'll aim to design in such a way as to make those kinds of things 
possible in future extensions.


I like the idea of being able to have both explicit and 
implicit/automatic/lightweight checkpoints and distinguish them. The 
more I think about automatic checkpoints, the more I see this as 
bringing to Subversion's WC the 'auto-save' and 'undo' features that 
nearly all user-friendly editor applications offer. I can see a GUI 
wanting to make frequent automatic checkpoints so that the user doesn't 
have to remember to do so.


- Julian

Re: Checkpointing - is not Quilt

[Branko Čibej]

On 17.07.2017 13:42, Julian Foad wrote:
> Initially I wrote that 'checkpointing' should allow committing the
> result either all at once or as a corresponding series of commits.
> Having thought more, I believe the use case for a series of commits
> falls out of scope.
>
> Definitions:
>
>   Quilt:
>
> Preparing a series of patches that will be applied (perhaps
> committed) sequentially in a given order.
> Each patch represents a self-contained logical change, likely with
> its own log message.
> Later patches may depend on earlier ones.
> Ability to jump to any patch in the series and modify it in the
> context of the earlier patches having been applied and the later ones
> not.
> After such a modification, updating (as in rebasing) the later
> patches accordingly.
> Ability to commit the series as a corresponding series of commits,
> or all in one commit.
>
>   Checkpointing:
>
> Saving intermediate, unfinished, working states during the
> preparation of a single logical change.
> The change will be committed as one revision when finished.
> Changing a state that has been checkpointed may be accomplished
> either by rolling back to an earlier state and then re-doing all
> subsequent changes in the desired way, or by leaving that checkpoint
> in place and making a further checkpoint after changing the working
> state in the desired way.
> Like an "undo stack", with or without "redo".


In order to use checkpoints as a sort of local commit, you do have to be
able to push a series of commits to the server. You'd distinguish
explicit checkpoints (with log messages) and implicit/automatic
checkpoints; only the explicit ones would constitute separate commits.

In order to support that, you'd have to have a way to rebase the whole
checkpoint stack onto current HEAD (or at least to a non-conflicting
state). Ideally you'd also support operations such as checkpoint
reordering, (local) squashing, removal, etc.

-- Brane

Checkpointing - is not Quilt

[Julian Foad]

Initially I wrote that 'checkpointing' should allow committing the 
result either all at once or as a corresponding series of commits. 
Having thought more, I believe the use case for a series of commits 
falls out of scope.


Definitions:

  Quilt:

Preparing a series of patches that will be applied (perhaps 
committed) sequentially in a given order.
Each patch represents a self-contained logical change, likely with 
its own log message.

Later patches may depend on earlier ones.
Ability to jump to any patch in the series and modify it in the 
context of the earlier patches having been applied and the later ones not.
After such a modification, updating (as in rebasing) the later 
patches accordingly.
Ability to commit the series as a corresponding series of commits, 
or all in one commit.


  Checkpointing:

Saving intermediate, unfinished, working states during the 
preparation of a single logical change.

The change will be committed as one revision when finished.
Changing a state that has been checkpointed may be accomplished 
either by rolling back to an earlier state and then re-doing all 
subsequent changes in the desired way, or by leaving that checkpoint in 
place and making a further checkpoint after changing the working state 
in the desired way.

Like an "undo stack", with or without "redo".


- Julian

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

I committed an initial prototype for checkpointing on the 
"shelve-checkpoint" branch.  http://svn.apache.org/r1801942


This prototype is of the "store snapshot patches" ("option 1") design.

Here is the help output:
[[[
$ svn checkpoint --help
checkpoint: Checkpoint the local changes.
usage: 1. checkpoint save
   2. checkpoint revert
   3. checkpoint rollback NUMBER
   4. checkpoint list|--list

  1. Save the working state as a new checkpoint.
  2. Revert the working state to the current checkpoint.
  3. Roll back the working state to checkpoint NUMBER.
  4. List all checkpoints. A synonym for 'svn checkpoints'.
[...]
]]]

Please try it and let me know your thoughts on the directions we should 
take it in, if you have a chance.


I think the next most important things it needs are:

  * recognize when there are no further changes relative to the last 
checkpoint, and do nothing in that case;

  * see the status and diff of the WC against the current checkpoint;
  * see a summary and diff of each checkpoint against the previous one.

Diffing checkpoints is quite hard (well, needs a patch-arithmetic 
library) in this design. Even just determining whether two patches are 
"identical". That might be a good reason for me to try the alternative 
design "store checkpoints in a local repository" ("option 3") next.


- Julian


On 12/07/17, Julian Foad wrote:
I committed an initial prototype for shelving on the "shelve-checkpoint" 
branch.


Here is the help output:
[[[
$ svn shelve --help
shelve: Shelve changes.
usage: 1. shelve NAME PATH...
2. shelve --delete NAME
3. shelve --list

   1. Shelve as NAME the local changes in the given PATHs.
   2. Delete the shelved patch NAME.
   3. List shelved patches.
[...]

$ svn unshelve --help
unshelve: Unshelve changes.
usage: unshelve [--keep] NAME

]]]

Here is an example session:
[[[
$ # create and commit some files

$ mkdir doc; echo hello > doc/hello.txt; echo config > config

$ svn add --force .; svn ci -q -m ""
A config
A doc
A doc/hello.txt

$ # start making some changes to docs

$ echo new > doc/new.txt; svn add doc/new.txt
A doc/new.txt

$ echo more >> config

$ svn st
M   config
A   doc/new.txt

$ # shelve these documentation changes

$ svn shelve docs .
U config
D doc/new.txt
shelved 'docs'

$ svn st

$ svn shelve --list
docs.patch

$ # work on a quick fix and commit it

$ echo Hello > doc/hello.txt

$ svn ci -q -m "Fix capitalization."

$ # unshelve to continue work on docs

$ svn unshelve docs
U config
A doc/new.txt
unshelved 'docs'

$ svn st
M   config
A   doc/new.txt

$ svn shelve --list

]]]

If you have a chance to try it yourself, I'd love to hear your first 
impressions.


- Julian



References:
[1] Shelving-Checkpointing Dev doc. (J Foad)
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit# 

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

Bert Huijben wrote:

Julian Foad wrote:

Mark Phippard wrote:

[...]  does patch just create conflicts that you
resolve like anything else?


It is 'svn patch' -- so it raises conflicts.


Svn patch creates reject files. It doesn't create conflicts (yet).


Gosh, so it does! How primitive! :-)

I will expand on that aspect.

- Julian

RE: [RFC] Shelving and Checkpointing

[Bert Huijben]

> -Original Message-
> From: Julian Foad [mailto:julianf...@apache.org]
> Sent: woensdag 12 juli 2017 16:38
> To: Mark Phippard 
> Cc: Subversion Developers 
> Subject: Re: [RFC] Shelving and Checkpointing
> 
> Mark Phippard wrote:
> > Nice to see you have gotten this far.
> >
> > I am not even sure how this behaves with git stash but what happens if
> > the patch cannot be applied cleanly?  Is the path to "fixing things"
> > relatively clear?  Like does patch just create conflicts that you
> > resolve like anything else?
> 
> It is 'svn patch' -- so it raises conflicts.

Svn patch creates reject files. It doesn't create conflicts (yet).

Bert

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

Mark Phippard wrote:

Nice to see you have gotten this far.

I am not even sure how this behaves with git stash but what happens if 
the patch cannot be applied cleanly?  Is the path to "fixing things" 
relatively clear?  Like does patch just create conflicts that you 
resolve like anything else?


It is 'svn patch' -- so it raises conflicts.

- Julian

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

Johan Corveleyn wrote:

I've quickly scanned your google docs, but have to go through them in
some more detail. I'll try to dig into them and give some feedback if
I can.


Thanks you Johan! I look forward to hearing your comments when you have 
a chance.



One thing that crossed my mind: a nice additional feature that
probably could be built on top of shelving would be "partial commit"
(committing only some hunks out of a modified file). I've recently
looked up past discussions about this, in response to a post on
users@. See my response with a lot of references here:

 https://svn.haxx.se/users/archive-2017-06/0004.shtml


Yes, I would find that useful myself. A good future extension.

That's what I mean by the 'patch hunks' feature listed in the 'Role 
Models' table in the 'UI' doc which is linked from the 'Dev' doc. Bazaar 
'bzr shelve' does that interactively by default, and both Mercurial and 
Git do it with 'hg shelve --interactive' and 'git stash --patch'.


- Julian

Re: [RFC] Shelving and Checkpointing

[Mark Phippard]

On Wed, Jul 12, 2017 at 4:18 AM, Julian Foad  wrote:

> I committed an initial prototype for shelving on the "shelve-checkpoint"
> branch.
>
> Here is the help output:
> [[[
> $ svn shelve --help
> shelve: Shelve changes.
> usage: 1. shelve NAME PATH...
>2. shelve --delete NAME
>3. shelve --list
>
>   1. Shelve as NAME the local changes in the given PATHs.
>   2. Delete the shelved patch NAME.
>   3. List shelved patches.
> [...]
>
> $ svn unshelve --help
> unshelve: Unshelve changes.
> usage: unshelve [--keep] NAME
>
> ]]]
>
> Here is an example session:
> [[[
> $ # create and commit some files
>
> $ mkdir doc; echo hello > doc/hello.txt; echo config > config
>
> $ svn add --force .; svn ci -q -m ""
> A config
> A doc
> A doc/hello.txt
>
> $ # start making some changes to docs
>
> $ echo new > doc/new.txt; svn add doc/new.txt
> A doc/new.txt
>
> $ echo more >> config
>
> $ svn st
> M   config
> A   doc/new.txt
>
> $ # shelve these documentation changes
>
> $ svn shelve docs .
> U config
> D doc/new.txt
> shelved 'docs'
>
> $ svn st
>
> $ svn shelve --list
> docs.patch
>
> $ # work on a quick fix and commit it
>
> $ echo Hello > doc/hello.txt
>
> $ svn ci -q -m "Fix capitalization."
>
> $ # unshelve to continue work on docs
>
> $ svn unshelve docs
> U config
> A doc/new.txt
> unshelved 'docs'
>
> $ svn st
> M   config
> A   doc/new.txt
>
> $ svn shelve --list
>
> ]]]
>
> If you have a chance to try it yourself, I'd love to hear your first
> impressions.
>
>
>

Nice to see you have gotten this far.

I am not even sure how this behaves with git stash but what happens if the
patch cannot be applied cleanly?  Is the path to "fixing things" relatively
clear?  Like does patch just create conflicts that you resolve like
anything else?

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: [RFC] Shelving and Checkpointing

[Johan Corveleyn]

On Mon, Jul 10, 2017 at 2:59 PM, Julian Foad  wrote:
> Dear Subversion Developers,
>
> I am delighted to announce that I am working with Assembla to develop
> shelving and checkpointing functionality in Subversion. These have
> been on the wish list for many years, and are becoming ever more in
> demand since the popularity of git is making more users appreciate the
> benefits of local operations.
>
> Based on our many discussions in the past, along with a fresh look at
> how other VCS's implement these features, I have written down the
> requirements and started working on a design in the document
> Shelving-Checkpointing Dev [1]. I will follow up with an HTML email
> containing a copy of today's version. I invite you to leave comments
> directly in Google Docs, or send them by reply here, as you prefer.
>
> I will be the project lead on this within Assembla. I will regularly
> demonstrate small increments of functionality, which will also act as
> calls for feedback. We want to keep the scope quite small to get it
> done as quickly as possible.
>
> Please read through and if you are able to contribute with any
> suggestions or practical help, that would be wonderful. Thanks!
>
> - Julian
>
>
> References:
>[1] Shelving-Checkpointing Dev doc. (J Foad)
> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#

Hi Julian,

That's great news! I hope this will give Subversion some much needed
extra momentum (together with the tree conflict resolution
improvements of 1.10). Indeed, these "local / offline" features have
been on many people's wish list for a long time.

I've quickly scanned your google docs, but have to go through them in
some more detail. I'll try to dig into them and give some feedback if
I can.

One thing that crossed my mind: a nice additional feature that
probably could be built on top of shelving would be "partial commit"
(committing only some hunks out of a modified file). I've recently
looked up past discussions about this, in response to a post on
users@. See my response with a lot of references here:

https://svn.haxx.se/users/archive-2017-06/0004.shtml
(For the "partial commit" feature, start reading below "Is there an
option to inspect each file further line-by-line for lines that have
changed to either be selected or excluded from the commit?")

As I explained in that post, TortoiseSVN has implemented this in some
way with what they call "restore after commit" [1].

So I imagine we could do the same by (1) making the user select some
lines he wants to commit; (2) shelve the modifications that were not
selected; (3) commit; (4) unshelve. We even have a command line UI
(sort of) for selecting lines out of a diff, namely the UI stsp
created for handling text conflicts interactively.

Just a thought ... Of course fleshing out the fundamentals of the
Shelving and Checkpointing features in the first place will be most
important.

[1] 
https://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-dug-commit.html#tsvn-dug-commit-restore

-- 
Johan

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

I committed an initial prototype for shelving on the "shelve-checkpoint" 
branch.


Here is the help output:
[[[
$ svn shelve --help
shelve: Shelve changes.
usage: 1. shelve NAME PATH...
   2. shelve --delete NAME
   3. shelve --list

  1. Shelve as NAME the local changes in the given PATHs.
  2. Delete the shelved patch NAME.
  3. List shelved patches.
[...]

$ svn unshelve --help
unshelve: Unshelve changes.
usage: unshelve [--keep] NAME

]]]

Here is an example session:
[[[
$ # create and commit some files

$ mkdir doc; echo hello > doc/hello.txt; echo config > config

$ svn add --force .; svn ci -q -m ""
A config
A doc
A doc/hello.txt

$ # start making some changes to docs

$ echo new > doc/new.txt; svn add doc/new.txt
A doc/new.txt

$ echo more >> config

$ svn st
M   config
A   doc/new.txt

$ # shelve these documentation changes

$ svn shelve docs .
U config
D doc/new.txt
shelved 'docs'

$ svn st

$ svn shelve --list
docs.patch

$ # work on a quick fix and commit it

$ echo Hello > doc/hello.txt

$ svn ci -q -m "Fix capitalization."

$ # unshelve to continue work on docs

$ svn unshelve docs
U config
A doc/new.txt
unshelved 'docs'

$ svn st
M   config
A   doc/new.txt

$ svn shelve --list

]]]

If you have a chance to try it yourself, I'd love to hear your first 
impressions.


- Julian



References:
[1] Shelving-Checkpointing Dev doc. (J Foad)
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

Daniel Shahaf wrote:

Julian Foad wrote on Tue, 11 Jul 2017 21:53 +0100:

2. What I was thinking there is to rewrite as much of our libs as needed
to implement deeply integrated local branching in Svn client. The full
extent of what that might entail or look like is unknown.


I don't understand the distinction apparent here between 'checkpointing'
and 'local branching'.  Are these two terms not synonymous?


Sorry, I didn't mean to make a distinction. "... implement deeply 
integrated ". Such as extra trees in the WC DB, as 
envisioned in Reference 5. Svn Wiki page "Design: SavePoints" 
<https://wiki.apache.org/subversion/SavePoints>, for example.


- Julian

Re: [RFC] Shelving and Checkpointing

[Branko Čibej]

On 11.07.2017 10:53, Julian Foad wrote:
> Thanks for your suggestion, Nathan.
>
> Nathan Hartman wrote:
>> [...] What if, instead of just a pristine copy, it actually created a
>> private local repository. Revision 1 of this repository would be the
>> pristine copy. [...] if you type some other command instead of
>> commit, or maybe prepend the word "local" or something, the commit
>> would go into the local repository [...] Shelving, stashing, etc.,
>> all become local operations against this local souped up pristine
>> copy replacement.
>
> That is exactly what I was thinking about when I wrote "Option 3...
> Checkpoints are commits in a local repository embedded in the WC"
> towards the end of that document. My feeling is that both the amount
> of effort required just to get that far, and the explosion of further
> possibilities it would open up, make it too heavy-weight for my
> current project, but I would definitely like to explore that
> possibility further.

Ben (Reser) and I discussed this to death at one of the hackathons. It
turns out that checkpoints / local commits are, while not a
pre-requisite, definitely a logical first step on the road to stashes
and local branches. We even had a pretty good architectural model and UI
feature list written down ... I wonder where that's gone now.

The problem we encountered at the time is that the current working copy
schema is too limited to implement them effectively; it needs another
dimension of change tracking.

-- Brane

Re: [RFC] Shelving and Checkpointing

[Daniel Shahaf]

Sorry, pressed [Send] too early.

Thanks for the explanations.  More below.

Daniel Shahaf wrote on Tue, 11 Jul 2017 22:02 +:
> Julian Foad wrote on Tue, 11 Jul 2017 21:53 +0100:
> > Daniel Shahaf wrote:
> > > Julian Foad wrote:
> > >> Checkpointing
> > >>
> > >> Options:
> > >>
> > >> 1. further patch management built on a series of shelved changes
> > >> 2. local commits tightly integrated
> > >> 3. checkpoints are commits in a local repository
> > > 
> > > Can you explain these three options in more words?  AIUI #1 is
> > > "syntactic sugar to manage a patch series", like quilt(1), but I'm not
> > > sure I understand #2 and #3.
> > 
> > 2. What I was thinking there is to rewrite as much of our libs as needed 
> > to implement deeply integrated local branching in Svn client. The full 
> > extent of what that might entail or look like is unknown.

I don't understand the distinction apparent here between 'checkpointing'
and 'local branching'.  Are these two terms not synonymous?

> > 3. To store a series of checkpoints, create a temporary repos inside 
> > .svn/ and "relocate" the WC base to it. Then use it for all operations 
> > until the user comes to the point they want to commit to the real repo.

I think when wc-ng was invented, the idea was that a 'checkpoint' would
simply be another tree, alongside BASE and WORKING.  I suppose this
is option #4.

In general, we have at least five ways of representing a tree: as a
dumpstream, as an editor stream, as a FSFS revision file, as a wc.db
(+ pristine store), as a diff against an empty tree.  The 'temporary
repository' could store the changes in any of these forms.  I suppose the
question is which representation is best amenable to being replayed and to
resolving conflicts (when rebasing a checkpoint onto a younger start
revision).

Cheers,

Daniel

Re: [RFC] Shelving and Checkpointing

[Nathan Hartman]

On Tue, Jul 11, 2017 at 4:53 AM, Julian Foad  wrote:

> Nathan Hartman wrote:
>
>> [...] What if, instead of just a pristine copy, it actually created a
>> private local repository. Revision 1 of this repository would be the
>> pristine copy. [...]
>>
>
> That is exactly what I was thinking about when I wrote "Option 3...
> Checkpoints are commits in a local repository embedded in the WC" towards
> the end of that document. My feeling is that both the amount of effort
> required just to get that far, and the explosion of further possibilities
> it would open up, make it too heavy-weight for my current project, but I
> would definitely like to explore that possibility further.


The explosion of possibilities, to borrow your words, is IMO a disadvantage
of Option 3. I'm saying this despite having suggested this same idea.

I'd like to emphasize that I don't advocate turning Subversion into a DVCS.
Proponents of DVCSs seem to think that Subversion's centralized nature is a
disadvantage. I think the exact opposite. With centralized, you get Single
Source of Truth, a system that is easy to learn, easy to teach, programmers
and non programmers can use it successfully. I like that history is
immutable, never rewritten, because I think the primary responsibility is
to never lose information. You only have two places to know about: the
repository and your working copy.

What I do like is the idea of remaining centralized while having some local
capabilities. The stashing / checkpointing wishlist items are valid. I
mentioned the ability to roll back a svn update if something unexpected
happens (by doing an implied local commit of your current state as the
first step of svn update). Local feature branches could be useful for
one-off experiments, short-lived work done by one person, etc. In any case,
these features should all be optional. They'd be there if you want them,
but you never have to know about them if you're not interested.

In any event, I wouldn't go full blown DVCS where the concept of repository
vs. working copy goes away. I wouldn't turn a checkout into a full clone,
or a working copy into a repository.

Other than the addition of new features in the client, all else would
remain the same.

Why put a hidden repository in the working copy, then? The reason is that
while the client could use a different "lighter weight" method to manage
various versions of its working copy, that would reinvent the wheel. You'd
end up with a second incompatible version control system within Subversion.
If I understand correctly, the client already has server code in it,
because when you use the file:// URL scheme, the client is the server. So
why reinvent a new version control system within Subversion when it already
has everything it needs?

But here's the key: The difference between a normal repository and the
hidden one in the working copy is that the latter is a hidden
implementation detail, not something the user can access or control. For
example you can't checkout a working copy of it. Everything that concerns
that hidden repo, its creation, the directory and file layout within it...
all of that is controlled by Subversion itself. It's more of an internal
client-side filing system than an actual repository, although it uses a
repository as the underlying implementation. The user gets some new local
features, but not the dreaded "explosion of possibilities."

Kind regards,
Nathan

Re: [RFC] Shelving and Checkpointing

[Daniel Shahaf]

Julian Foad wrote on Tue, 11 Jul 2017 21:53 +0100:
> Daniel Shahaf wrote:
> > Julian Foad wrote:
> >> Checkpointing
> >>
> >> Options:
> >>
> >> 1. further patch management built on a series of shelved changes
> >> 2. local commits tightly integrated
> >> 3. checkpoints are commits in a local repository
> > 
> > Can you explain these three options in more words?  AIUI #1 is
> > "syntactic sugar to manage a patch series", like quilt(1), but I'm not
> > sure I understand #2 and #3.
> 
> 2. What I was thinking there is to rewrite as much of our libs as needed 
> to implement deeply integrated local branching in Svn client. The full 
> extent of what that might entail or look like is unknown.

I don't understand the distinction apparent here between 'checkpointing'
and 'local branching'.

> 3. To store a series of checkpoints, create a temporary repos inside 
> .svn/ and "relocate" the WC base to it. Then use it for all operations 
> until the user comes to the point they want to commit to the real repo.

I think when wc-ng was invented, the thinking was that a 'checkpoint'
would simply be another tree, alongside BASE and WORKING.  None


> 
> >> SHELVING
> >>
> >> git stashing <https://git-scm.com/book/en/v1/Git-Tools-Stashing>
> > 
> > For git, 'stashing' can also be implemented by a temporary branch («git
> > checkout -b foo && git commit -amm && git checkout master»), which
> > changes some of the values of the table.  I assume the same is true for hg.
> 
> Yes, I should note that mode of working as well. Thanks.
> 
> - Julian

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

Daniel Shahaf wrote:

Julian Foad wrote:

Checkpointing

Options:

1. further patch management built on a series of shelved changes
2. local commits tightly integrated
3. checkpoints are commits in a local repository


Can you explain these three options in more words?  AIUI #1 is
"syntactic sugar to manage a patch series", like quilt(1), but I'm not
sure I understand #2 and #3.


2. What I was thinking there is to rewrite as much of our libs as needed 
to implement deeply integrated local branching in Svn client. The full 
extent of what that might entail or look like is unknown.


3. To store a series of checkpoints, create a temporary repos inside 
.svn/ and "relocate" the WC base to it. Then use it for all operations 
until the user comes to the point they want to commit to the real repo.




SHELVING

git stashing <https://git-scm.com/book/en/v1/Git-Tools-Stashing>


For git, 'stashing' can also be implemented by a temporary branch («git
checkout -b foo && git commit -amm && git checkout master»), which
changes some of the values of the table.  I assume the same is true for hg.


Yes, I should note that mode of working as well. Thanks.

- Julian

Re: [RFC] Shelving and Checkpointing

[Daniel Shahaf]

On Mon, Jul 10, 2017 at 02:03:41PM +0100, Julian Foad wrote:
> Checkpointing
> 
> Options:
> 
> 
>1.
> 
>further patch management built on a series of shelved changes
>2.
> 
>local commits tightly integrated
>3.
> 
>checkpoints are commits in a local repository

Can you explain these three options in more words?  AIUI #1 is
"syntactic sugar to manage a patch series", like quilt(1), but I'm not
sure I understand #2 and #3.

> SHELVING
> 
> git stashing <https://git-scm.com/book/en/v1/Git-Tools-Stashing>
> 

For git, 'stashing' can also be implemented by a temporary branch («git
checkout -b foo && git commit -amm && git checkout master»), which
changes some of the values of the table.  I assume the same is true for hg.

> hg shelving <https://www.selenic.com/mercurial/hg.1.html#shelve>
> 
> bzr shelving
> <http://doc.bazaar.canonical.com/bzr.dev/en/user-guide/shelving_changes.html>
> 
> p4 shelving
> <https://www.perforce.com/perforce/r17.1/manuals/p4guide/chapter.basic-tasks.html#basic-tasks.recurring-other.changelist.shelve>
> 
> IntelliJ
> <https://www.jetbrains.com/help/idea/shelving-and-unshelving-changes.html>
> 
> NetBeans
> <http://docs.oracle.com/netbeans/nb82/netbeans/netbeans/NBDAG/version_control.htm#NBDAG2545>

Unfortunately the text/plain rendering of that table is quite lossy.
The part I quoted are the column headings; the row headings are
"general" "stored where?" "stored how?" "changelist integration" "pop
vs. apply" "access in other cmds" "merge & conflicts".

Cheers,

Daniel

Re: [RFC] Shelving and Checkpointing

[Julian Foad]

Thanks for your suggestion, Nathan.

Nathan Hartman wrote:

[...] What if, instead of just a pristine copy, it actually created a private local 
repository. Revision 1 of this repository would be the pristine copy. [...] if you type 
some other command instead of commit, or maybe prepend the word "local" or 
something, the commit would go into the local repository [...] Shelving, stashing, etc., 
all become local operations against this local souped up pristine copy replacement.


That is exactly what I was thinking about when I wrote "Option 3... 
Checkpoints are commits in a local repository embedded in the WC" 
towards the end of that document. My feeling is that both the amount of 
effort required just to get that far, and the explosion of further 
possibilities it would open up, make it too heavy-weight for my current 
project, but I would definitely like to explore that possibility further.


- Julian


References:
   [1] Shelving-Checkpointing Dev doc. (J Foad)
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#

Re: [RFC] Shelving and Checkpointing

[Nathan Hartman]

On Jul 10, 2017, at 8:59 AM, Julian Foad  wrote:
> 
> Dear Subversion Developers,
> 
> I am delighted to announce that I am working with Assembla to develop
> shelving and checkpointing functionality in Subversion. These have
> been on the wish list for many years, and are becoming ever more in
> demand since the popularity of git is making more users appreciate the
> benefits of local operations.

Hi all,

I just wanted to chime in and say that I'm very happy to hear that these local 
features are planned for Subversion. After ten years of using Subversion, I can 
say that it has been a joy to use from day one, owing in large part to its 
simplicity and reliability.

It occurred to me recently that in the distant past Subversion's purpose was to 
be a better CVS, that that goal has been achieved long ago, and that perhaps 
it's time for a Subversion 2.0, which could take it to the next level. I don't 
know much about future plans, such as what FSX is intended to achieve, but it 
occurred to me that perhaps Subversion could gain the ability to do local 
commits, which could be an optional step before doing the real commit back to 
the repository. As I thought about that, I realized that it might not have to 
require too much new machinery to achieve that.

I had this idea: When you checkout a working copy, Subversion creates the 
hidden .svn directory which contains the pristine copies among other things. 
What if, instead of just a pristine copy, it actually created a private local 
repository. Revision 1 of this repository would be the pristine copy. The user 
need not know that anything has changed and can continue working as before, 
committing back to the server. But, if you type some other command instead of 
commit, or maybe prepend the word "local" or something, the commit would go 
into the local repository rather than the remote one. Now this makes it 
possible to do locally anything you could do remotely such as creating multiple 
local feature branches. It also means that when you do an update, there could 
be an implied local commit first, which allows you to undo an update which 
perhaps messed something up. Shelving, stashing, etc., all become local 
operations against this local souped up pristine copy replacement.

Since you're only checking out one revision just as before, as opposed to 
cloning a whole repository, this idea comes with some big advantages that you 
don't get with git or hg, such as that you need not clone an entire gigantic 
repository. This is one of the big reasons that those systems are not a good 
choice for housing lots of big files that change infrequently, as every user 
would have to clone potentially gigabytes of data they're not interested in. In 
other words a svn checkout still does what it always did before, except that 
you have the option to work locally if you want to.

I didn't mention any of this before because:

(1) there are obviously a lot of holes in this idea, such as, what happens to 
this private local repo when you do a svn switch? Or the pesky issue of how to 
add such a major new feature without inheriting a ton of obscure commands a la 
git?

And (2) although I am a developer and very fond of Subversion, I cannot take 
this upon myself because of other obligations now and in the foreseeable 
future. So I felt it would be rude to dump a crazy idea like this on everyone 
else. Julian's post prompted me to break my silence, not knowing if what I've 
suggested is remotely realistic.

I realize that this suggests a major backwards-incompatible change to the 
working copy format and indeed the entire client side. But this was a request 
for comment. :-) Hopefully someone finds my input valuable.

Kind regards,
Nathan 

> References:
>   [1] Shelving-Checkpointing Dev doc. (J Foad)
> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#

[RFC] Shelving and Checkpointing

[Julian Foad]

Dear Subversion Developers,

I am delighted to announce that I am working with Assembla to develop
shelving and checkpointing functionality in Subversion. These have
been on the wish list for many years, and are becoming ever more in
demand since the popularity of git is making more users appreciate the
benefits of local operations.

Based on our many discussions in the past, along with a fresh look at
how other VCS's implement these features, I have written down the
requirements and started working on a design in the document
Shelving-Checkpointing Dev [1]. I will follow up with an HTML email
containing a copy of today's version. I invite you to leave comments
directly in Google Docs, or send them by reply here, as you prefer.

I will be the project lead on this within Assembla. I will regularly
demonstrate small increments of functionality, which will also act as
calls for feedback. We want to keep the scope quite small to get it
done as quickly as possible.

Please read through and if you are able to contribute with any
suggestions or practical help, that would be wonderful. Thanks!

- Julian


References:
   [1] Shelving-Checkpointing Dev doc. (J Foad)
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#