Re: Checkpointing

[Thomas Singer]

Imagine working on one larger feature (or even multiple features). You 
already have created a couple of local commits, but are not yet 
finished. Following use cases come to my mind:


1) some other developer commits new revisions
- it should be possible to continue working on your feature-queue 
keeping it based on the old revision, but sooner or later it comes the 
time to integrate the new revision(s)
- this means to create a new queue on the new revision and cherry-pick 
all local commits of the old queue onto this new revision (could this be 
done with purely local information or would each cherry-pick require a 
possibly slow communication with the server?)
- with each cherry-pick, a conflict might occur - aborting should abort 
the creation of the new queue and application of the old queue, 
resulting in the deletion of the partly finished new queue keeping the 
old one

- each queue should be rebased independently onto the new revisions

2) you have to make a quick-fix revision
- you need to switch from your local feature-queue to the latest 
revision and fix the bug a coworker requests to get fixed
- you may now switch back to your queue to proceed with the work, but 
there is a new revision now, so you should base your work sooner or 
later on that revision


3) you want to rework your local commits, e.g. change order, squash some 
commits, change the message

- create a new queue based on the same revision
- create new local commits by cherry-picking commits from the other 
(old) queue, maybe amending some local commits
- after the new queue is ready (verify to diff it with the old queue) 
the old queue can be deleted
- often enough I find it useful to be able to make one of my first 
feature commits public, so the x first local commits should be possible 
to be become revisions => the queue becomes shorter and based on the new 
revision(s)


4) you need to fork an existing queue at any local commit
- switching to one of the local commits of your current queue you detect 
that it contains a flaw
- creating now a local commit to fix would mean to first create a new 
queue based on the same revision, apply all previous local commits (no 
conflict risk) - the old queue would be kept



It might be useful to be able to "store" somehow one or another queue on 
the server, e.g. for backup reasons, so no change is lost if my hard 
disk crashes or my code fails and cleans the disk. (Creating a new, real 
feature branch with revisions in the repository I don't like because 
then they would be cut in stone and this would force me to create nice 
and clean commits. But because we are not without error, such code will 
contain back and forth changes and hence hard to read.)


As long as I'm working on a non-trivial feature/refactoring, I prefer to 
have complete control over my commits, I even like to commit completely 
unstable and incomplete code with Git - because I have the possibility 
to clean up later.


--
Best regards,
Thomas Singer



On 02/07/2019 18:32, Nathan Hartman wrote:

On Sun, Jun 30, 2019 at 11:22 AM Thomas Singer 
wrote:


With "rebasing" I mean, that such list of "local commits" needs to be
re-applied (on demand, not automatically) onto a different revision.
Something like a continues series of cherry-picking (with the
possibility to get a conflict in each step; and a possibility to
continue after conflict resolution or abort). This means to me, that at
least cherry-picking needs to be possible from a revision or a "local
commit".



Could you describe how you would like to use this capability? E.g.,
give an example use case?

Re: Checkpointing

[Nathan Hartman]

On Sun, Jun 30, 2019 at 11:22 AM Thomas Singer 
wrote:

> With "rebasing" I mean, that such list of "local commits" needs to be
> re-applied (on demand, not automatically) onto a different revision.
> Something like a continues series of cherry-picking (with the
> possibility to get a conflict in each step; and a possibility to
> continue after conflict resolution or abort). This means to me, that at
> least cherry-picking needs to be possible from a revision or a "local
> commit".
>

Could you describe how you would like to use this capability? E.g.,
give an example use case?

Re: Checkpointing

[Branko Čibej]

On 30.06.2019 16:56, Nathan Hartman wrote:
> On Sat, Jun 29, 2019 at 7:21 AM Branko Čibej  > wrote:
>
> As Mark explained, it will do none of the above unless someone
> steps up
> and writes the code.
>
> For reference, what Nathan described was discussed here on the
> list and
> in person during hackathons years ago, yet nothing happened until
> Julian
> started writing code (and even then, what Julian is doing is a limited
> subset of the "ideal").
>
> If there's no interest amongst people to take the time to write
> the code
> ... well, we can all tell tall stories about the future, but that
> won't
> change it one bit.
>
>
> I know.
>
> We all know.
>
> I understand the frustration I see here.
>
> I understand that you've seen these wonderful discussions time and
> again and then nothing happened. And you've seen it so many times that
> you've become inoculated to the idea that it could change.
>
> But it will change, because:
>
> There was a wise man named Albert Einstein, and I have no idea if he
> actually said this or not but he's widely credited with saying that
> the definition of insanity is to do the same thing over and over and
> expect different results.
>
> Telling the closed dev@ community that we need new developers didn't
> work until now and I don't expect it to start working miraculously.
> I'm sorry to use an expletive -- marketing -- but that's what we need
> with the outside world. No business can sustain itself without telling
> the world what it's all about and there's no reason to believe that an
> open source project is any different. There's still a profit motive.
> In a business, it's profit in money; in an open source project, it's
> profit in mindshare and participation. So we need to get out there and
> drum up some new business, BUT:
>
> There's a bit of a chicken and egg problem here. If we entice new
> potential devs to join dev@ and they come here and see discussions of
> decline, defeat, and despair,


Who's despairing? Stating facts is neither decline, defeat, nor despair. :)

I would actually love to see all of you who are so enthusiastic in this
thread to come aboard. Especially as one of the things the developers
here were traditionally lacking is real-world user experience on
non-trivial projects (I count Subversion itself as "trivial" in terms of
the complexity of our version control workflow).


> they'll get turned off and go somewhere
> else. People want to be part of something successful! We need those
> who join to see discussions of all the cool things Subversion WILL do.
>
> Of course it won't do any of it until after the code gets written. For
> the code to get written we need devs. To get devs we need to change
> our thinking from despair to planning for a great future. So let's have
> some positive discussions over here!
>
> I'm going to search for those old discussions -- and the ones about
> what the command line syntax should be like -- and I'll be back later
> with some concrete thoughts.


Perfect.

-- Brane

Re: Checkpointing

[Thomas Singer]

Hi Nathan,

Thanks for your detailed answers.


- will it support "rebasing" such a local history onto the latest
updated commit?



It will have to support "rebasing" which is what "svn update" already
does today. Otherwise you couldn't commit your work!


With "rebasing" I mean, that such list of "local commits" needs to be 
re-applied (on demand, not automatically) onto a different revision. 
Something like a continues series of cherry-picking (with the 
possibility to get a conflict in each step; and a possibility to 
continue after conflict resolution or abort). This means to me, that at 
least cherry-picking needs to be possible from a revision or a "local 
commit".


Tom

Re: Checkpointing

[Nathan Hartman]

On Sat, Jun 29, 2019 at 7:21 AM Branko Čibej  wrote:

> As Mark explained, it will do none of the above unless someone steps up
> and writes the code.
>
> For reference, what Nathan described was discussed here on the list and
> in person during hackathons years ago, yet nothing happened until Julian
> started writing code (and even then, what Julian is doing is a limited
> subset of the "ideal").
>
> If there's no interest amongst people to take the time to write the code
> ... well, we can all tell tall stories about the future, but that won't
> change it one bit.
>

I know.

We all know.

I understand the frustration I see here.

I understand that you've seen these wonderful discussions time and
again and then nothing happened. And you've seen it so many times that
you've become inoculated to the idea that it could change.

But it will change, because:

There was a wise man named Albert Einstein, and I have no idea if he
actually said this or not but he's widely credited with saying that
the definition of insanity is to do the same thing over and over and
expect different results.

Telling the closed dev@ community that we need new developers didn't
work until now and I don't expect it to start working miraculously.
I'm sorry to use an expletive -- marketing -- but that's what we need
with the outside world. No business can sustain itself without telling
the world what it's all about and there's no reason to believe that an
open source project is any different. There's still a profit motive.
In a business, it's profit in money; in an open source project, it's
profit in mindshare and participation. So we need to get out there and
drum up some new business, BUT:

There's a bit of a chicken and egg problem here. If we entice new
potential devs to join dev@ and they come here and see discussions of
decline, defeat, and despair, they'll get turned off and go somewhere
else. People want to be part of something successful! We need those
who join to see discussions of all the cool things Subversion WILL do.

Of course it won't do any of it until after the code gets written. For
the code to get written we need devs. To get devs we need to change
our thinking from despair to planning for a great future. So let's have
some positive discussions over here!

I'm going to search for those old discussions -- and the ones about
what the command line syntax should be like -- and I'll be back later
with some concrete thoughts.

Re: Checkpointing

[Nathan Hartman]

On Fri, Jun 28, 2019 at 1:12 PM Thomas Singer 
wrote:

> - so the working copy can have checked out multiple commits or one
> checkpoint?
>

The working copy has always provided one "view." I say "view" for lack
of a better word but I mean the checked out directory of your data
that is under version control. Currently, that "view" can be changed
through updating to a different revision, switching to a different
branch, changing depth on a sparse/shallow checkout, etc; but it's
still one view. As I imagine it, checkpoints would add the ability to
update to a checkpoint, or revert to the last checkpoint.

This begs a few important questions, such as:

When someone does "svn revert" with no additional parameters, what are
we reverting to? Do we revert to BASE as we always have? Or to the
last checkpoint? That requires further study...


> - will it support multiple histories ("branches") planned, e.g. for
> different features?
>

Subversion already provides server-side branches. For local work,
earlier I described multiple "arrays" of checkpoints. We need a better
name than "array" but the idea is that a checkpoint is like a local
commit, and the array is a local linear history of such commits. And
you can have multiple arrays. You could say that each array is like a
local branch. But I would rather think of each array as being for a
certain subject. So, for example if I'm editing code to write a new
feature, I might have a checkpoint array called "feature" and a second
checkpoint array called "unrelated" just as an example. I'd work on my
feature, making checkpoints to the "feature" array as I go. Suppose
that while I'm working I come across typos in comments. I could fix
those typos and make a checkpoint of that un the "unrelated" array.
When I'm ready to commit, I'll commit the typo fixes separately from
the feature work. That would make it much easier to have one subject
per commit.

Anyway that's how I imagine it. If you have other thoughts, I'd love
to hear them!

- will it support "rebasing" such a local history onto the latest
> updated commit?


It will have to support "rebasing" which is what "svn update" already
does today. Otherwise you couldn't commit your work!

Re: Checkpointing

[Branko Čibej]

On 28.06.2019 19:12, Thomas Singer wrote:
> Hi Nathan,
>
> Thanks for your explanation. This sounds quite interesting. I have
> some questions though:
>
> - so the working copy can have checked out multiple commits or one
> checkpoint?
>
> - will it support multiple histories ("branches") planned, e.g. for
> different features?
>
> - will it support "rebasing" such a local history onto the latest
> updated commit?
>

As Mark explained, it will do none of the above unless someone steps up
and writes the code.

For reference, what Nathan described was discussed here on the list and
in person during hackathons years ago, yet nothing happened until Julian
started writing code (and even then, what Julian is doing is a limited
subset of the "ideal").

If there's no interest amongst people to take the time to write the code
... well, we can all tell tall stories about the future, but that won't
change it one bit.

-- Brane

Re: Checkpointing

[Thomas Singer]

Hi Nathan,

Thanks for your explanation. This sounds quite interesting. I have some 
questions though:


- so the working copy can have checked out multiple commits or one 
checkpoint?


- will it support multiple histories ("branches") planned, e.g. for 
different features?


- will it support "rebasing" such a local history onto the latest 
updated commit?


--
Best regards,
Thomas Singer

Re: Checkpointing v1 design

[Julian Foad]

Julian Foad wrote:

[...]
This aims to streamline the manual method, which I use myself, of 
using 'svn diff' and 'svn patch' to create and apply patches named 
'feature-v1.patch', 'feature-v2.patch', etc.


Now on trunk:

$ svn help | grep "shel[fv]"
   shelf-diff
   shelf-drop
   shelf-list (shelves)
   shelf-log
   shelf-save
   shelve
   unshelve

- Julian

Re: Checkpointing v1 in diagrams

[Julian Foad]

And here are the said diagrams...

Julian Foad wrote:

Here are a couple of diagrams to show how Checkpointing v1 behaves so far.

1. Shelve, Unshelve

Shows a difference between how Shelving-v1 worked and how it works in 
the new sheving-and-checkpointing design. Note that Unshelve no longer 
deletes the shelf. Shelving again would create a second version "foo 2".


2. Checkpoint, Revert, Restore

Shows how checkpointing and restoring works. Note that a 'revert' is 
needed before restoring.


(SVG format, drawn in Inkscape.)

- Julian

Re: Checkpointing v1 design

[Julian Foad]

Julian Foad wrote:

[...]
This aims to streamline the manual method, which I use myself, of using 
'svn diff' and 'svn patch' to create and apply patches named 
'feature-v1.patch', 'feature-v2.patch', etc.


An initial version is now committed to 'shelve-checkpoint' branch.

- Julian

Re: Checkpointing v1 design -- terminology

[Paul Hammant]

>
> $ svn save --revert -v2 "add hair color to person page"
>>
>> .. does the save and THEN drops the CL and it's changes in working copy -
>> back to no changes but not necessarily up to date.
>>
>
> That sounds like you're proposing an alternative syntax for what the
> 'shelve' command does -- except I'm not sure what you want the 'v2' option
> to do here; was that accidental?
>
>
Maybe alternate, sure.

$ svn savelist

add hair color to person page; v1
add hair color to person page; v1.2
add hair color to person page; whatever_passes_fora_versn_string

$ svn save -v v2 -m "add hair color to person page"
# Overwrites v2, perhaps would need a --force.

$ svn save -v three
# Created a new 'three' version of (implicit) "add hair color to person
page"

$ svn save -v three --revert
# As above but also deletes the changelist and any changes to working copy
versus last 'svn up'

$ svn restore -v three
# Creates a CL with all those changes on them.

Oh and shelve/save/checkpoint would all reuse impl as they are pretty
close. As does "Pull-request" (PR).

$ svn pull-request -m "add hair color to person page"
# Goes into codereview for Rietveld, Assembla, RhodeCode (etc),
# Possibly needing a plugin, or CURL configuration in .subversion/

- Paul

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 10.11.2017 14:20, Julian Foad wrote:
> Julian Foad wrote:
>> [...] To achieve such rollbacks, the user would have to first revert,
>> which involves deciding what to revert.
>>
>> And that seems OK to me.
>
> The larger point is that this kind of work flow, and the difficulty of
> working with changes in overlapping subtrees, is analogous to some
> existing techniques. For example, if one tries to use two different
> branches to manage two different change-sets, and work with them both
> in one WC by switching part of the WC to one branch and another part
> to another branch, and the changes involve overlapping subdirectories,
> maybe like this...
>
>  svn switch "^/branches/#1" A/B
>  svn switch "^/branches/#2" A/B/F A/D
>  svn switch "^/branches/#1"   A/D/G
>
> ... it seems to me that more or less the same issues arise.

To be fair, I've yet to hear of a VCS that can handle this sort of
complexity and make it easy for users. Git cops out by having
all-or-nothing branches, commits, etc. But add submodules and you're in
the same leaky boat again.

-- Brane

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Julian Foad wrote:
[...] To achieve such rollbacks, the user would have to first revert, which 
involves deciding what to revert.


And that seems OK to me.


The larger point is that this kind of work flow, and the difficulty of 
working with changes in overlapping subtrees, is analogous to some 
existing techniques. For example, if one tries to use two different 
branches to manage two different change-sets, and work with them both in 
one WC by switching part of the WC to one branch and another part to 
another branch, and the changes involve overlapping subdirectories, 
maybe like this...


 svn switch "^/branches/#1" A/B
 svn switch "^/branches/#2" A/B/F A/D
 svn switch "^/branches/#1"   A/D/G

... it seems to me that more or less the same issues arise.

Looking more specifically at one of the details, we note that 'svn 
switch URL PATH' recursively overrides the current URL on PATH and all 
its descendents (hence having to write the above example as three 
steps), whereas 'svn update' doesn't, and we have yet to define whether 
the proposed 'restore' command would do one or the other or something 
different. But if it's broadly analogous it seems reasonable.


- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Paul Hammant wrote:
I'd be happier with save-cl rather than ci-save.  Or just 'save', as I'm 
not sure what you're saving if not a change list.  Incidentally CL looks 
pretty close to CI in lower case and that's already used.


Spaces allowed in FOO name?

$ svn save -v2 "add hair color to person page"


Yes. The name obeys existing rules for a changelist name, for forward 
compatibility, and that does allow spaces.



Also..

$ svn save --revert -v2 "add hair color to person page"

.. does the save and THEN drops the CL and it's changes in working copy 
- back to no changes but not necessarily up to date.


That sounds like you're proposing an alternative syntax for what the 
'shelve' command does -- except I'm not sure what you want the 'v2' 
option to do here; was that accidental?



Revert files in (active) changelist FOO:
   svn revert --cl=FOO  # existing syntax


I don't know what this one does Julian?


I abbreviated a bit; the actual existing syntax for reverting the files 
tagged as changelist 'FOO' is:


  svn revert --cl=FOO -R .

(assuming all the files you need to revert are under '.').

- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Julian Foad wrote:
You can work out the 'rollbacks' from there (I haven't time right now); 


(That sounded like a crass cop-out, but I seriously didn't have a moment 
more to spare to think about it right then.)


The 'rollback' commands as written in steps 6 and 7 would both error. To 
achieve such rollbacks, the user would have to first revert, which 
involves deciding what to revert.


And that seems OK to me.

- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

I shudder to think of the interactions and side effects of partial
checkpoints. Consider our standard Greek tree and the following sequence:


First think about saving checkpoints the manual way, to set our initial 
expectations fairly low:


  svn diff foo-paths > foo-4.patch
  svn diff bar-paths > bar-17.patch

Then let's look at how we can improve on that.


  1. 
  2. svn checkpoint save "#1" A/B A/D/G
  3. 
  4. svn checkpoint save "#2" A/B/F A/D


I assume (1) these named changelists '#1' and '#2' were not defined (so 
not checkpointed) before step 1; and (2) you intend these 'save' 
commands to act with depth 'infinity' on the specified subtrees.


So you are concerned about how to handle overlapping subtrees -- in 
particular the case where a directory is associated with one changelist, 
and a subdirectory within it is (already or later) associated with another.



  5. 


I'll interpret that as 'do normal Subversion operations' rather than 
'execute arbitrary SQL statements on wc.db' :-)



  6. svn checkpoint rollback "#1"
  7. svn checkpoint rollback "#2"


OK, now is a good time to think about possible semantics. Here is a 
starting point.


  * A subtree is associated with a changelist either explicitly by 
attaching the changelist name to its root directory, or if not 
explicitly then implicitly by inheritance from its parent. A subtree 
association applies only to parts of the subtree that are not associated 
with a different changelist.


(Therefore a directory can be associated with only one changelist, and 
an explicit assignment overrides any association that would otherwise be 
inherited from its parents. After step 4 of your example everything in 
A/B is associated with '#1' *except* everything in A/B/F which is 
associated with '#2'.)


  * 'Saving' a changelist 'NAME' records the explicit attachment of 
NAME to particular PATHS.


  * Restoring a changelist 'NAME' restores the explicit attachment of 
NAME to particular PATHS. If some paths are already associated with 
changelist NAME, behaviour is client-defined, e.g. error, append to 
exiting assignments, replace existing assignments, with or without a 
warning. Let's say the initial CLI behaviour is error, for simplicity. 
(In other words the user has to revert before restoring, as I suggested 
a little earlier in this thread.)


  * The syntax '... save NAME PATHS...' first assigns changelist NAME 
to PATH..., replacing any previous assignment for PATH..., and then 
saves all paths belonging to NAME.


With this, your step 2 would save the whole subtree at A/B (and the one 
at A/D/G) in a first checkpoint of '#1', while your step 4 would 
unassign A/B/F from changelist '#1' and assign it to changelist '#2' 
instead.


You can work out the 'rollbacks' from there (I haven't time right now); 
I acknowledge it won't do magic for you -- but I'm sure you wouldn't 
want it to.


How does that strike you from a pragmatic point of view?

- Julian

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 07.11.2017 16:15, Julian Foad wrote:
> Branko Čibej wrote:
>> On 06.11.2017 17:30, Julian Foad wrote:
>>> The key point I am making here is the "topic" we're talking about in
>>> these commands is the "changelist" rather than the "savepoint" which
>>> is merely a stored version of a changelist.
>>
>> All our current commands that use changelists use them as an alias for a
>> set of targets. [...]
>>
>> Your proposal elevates changelists to a status that they do not
>> currently have in the client, and makes their usage inconsistent with
>> the rest of the command line. You might as well propose
>>
>>  svn changelist commit
>>  svn changelist diff
>>  svn changelist switch
>>
>> and so on. Whether that would be a good idea is not the issue here: the
>> issue is consistency, and (possibly) backwards-compatibility.
>
> I absolutely am considering elevating the status of changelists to
> that level, yes! That makes a very good model in my opinion.
>
>> And the
>> rather important detail that changelists cannot be associated with
>> directories.
>
> Yes, that's noted as a needed enhancement.
>
>> Changelists were always a sort of afterthought ...
>
> Indeed, that made me sad from their beginning. And what better
> opportunity to at last make a plan to improve them?

Noted. +1. Enjoy the corner cases. :)


>> On another note: changelists with checkpoints make no sense IMO; a
>> checkpoint should save and restore the state of the whole working copy.
>> Shelved changes are different. Their internal representation may be the
>> same, but the semantics are not.
>
> Why do you think 'checkpointing' your work should be across the whole
> WC whereas 'shelving' it should not? I can see how 'checkpoint your
> whole WC' has a utility, sure, but no more so than 'shelve your whole
> WC'. I think it makes a great deal of sense to checkpoint a unit of
> work rather than the whole WC, and to work towards elevating
> 'changelist' to fulfil that role.

I shudder to think of the interactions and side effects of partial
checkpoints. Consider our standard Greek tree and the following sequence:

 1. 
 2. svn checkpoint save "#1" A/B A/D/G
 3. 
 4. svn checkpoint save "#2" A/B/F A/D
 5. 
 6. svn checkpoint rollback "#1"
 7. svn checkpoint rollback "#2"


Um ... is that a rebase, a reorder, or a pretzel? Or does rolling back
to the state before #1 imply rolling back #2 as well? If the latter,
then checkpoints form a stack and represent the whole WC state.

-- Brane

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

On 06.11.2017 17:30, Julian Foad wrote:

The key point I am making here is the "topic" we're talking about in
these commands is the "changelist" rather than the "savepoint" which
is merely a stored version of a changelist.


All our current commands that use changelists use them as an alias for a
set of targets. [...]

Your proposal elevates changelists to a status that they do not
currently have in the client, and makes their usage inconsistent with
the rest of the command line. You might as well propose

     svn changelist commit
     svn changelist diff
     svn changelist switch

and so on. Whether that would be a good idea is not the issue here: the
issue is consistency, and (possibly) backwards-compatibility.


I absolutely am considering elevating the status of changelists to that 
level, yes! That makes a very good model in my opinion.



And the
rather important detail that changelists cannot be associated with
directories.


Yes, that's noted as a needed enhancement.


Changelists were always a sort of afterthought ...


Indeed, that made me sad from their beginning. And what better 
opportunity to at last make a plan to improve them?



unless you plan to
elevate them to first-class concepts across the whole command-line, I
recommend that the shelve/save/whatnot commands simply accept a
--changelist option and not complicate further. That this doesn't
preclude populating a changelist when a shelved change is restored.

On another note: changelists with checkpoints make no sense IMO; a
checkpoint should save and restore the state of the whole working copy.
Shelved changes are different. Their internal representation may be the
same, but the semantics are not.


Why do you think 'checkpointing' your work should be across the whole WC 
whereas 'shelving' it should not? I can see how 'checkpoint your whole 
WC' has a utility, sure, but no more so than 'shelve your whole WC'. I 
think it makes a great deal of sense to checkpoint a unit of work rather 
than the whole WC, and to work towards elevating 'changelist' to fulfil 
that role.


- Julian

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 06.11.2017 17:30, Julian Foad wrote:
> The key point I am making here is the "topic" we're talking about in
> these commands is the "changelist" rather than the "savepoint" which
> is merely a stored version of a changelist.

All our current commands that use changelists use them as an alias for a
set of targets. Instead of 'svn commit foo bar baz' you'd say 'svn
commit --changelist fbb'.

Your proposal elevates changelists to a status that they do not
currently have in the client, and makes their usage inconsistent with
the rest of the command line. You might as well propose

    svn changelist commit
    svn changelist diff
    svn changelist switch

and so on. Whether that would be a good idea is not the issue here: the
issue is consistency, and (possibly) backwards-compatibility. And the
rather important detail that changelists cannot be associated with
directories.

Changelists were always a sort of afterthought ... unless you plan to
elevate them to first-class concepts across the whole command-line, I
recommend that the shelve/save/whatnot commands simply accept a
--changelist option and not complicate further. That this doesn't
preclude populating a changelist when a shelved change is restored.

On another note: changelists with checkpoints make no sense IMO; a
checkpoint should save and restore the state of the whole working copy.
Shelved changes are different. Their internal representation may be the
same, but the semantics are not.

-- Brane

Re: Checkpointing v1 design -- terminology

[Paul Hammant]

I'd be happier with save-cl rather than ci-save.  Or just 'save', as I'm
not sure what you're saving if not a change list.  Incidentally CL looks
pretty close to CI in lower case and that's already used.

Spaces allowed in FOO name?

$ svn save -v2 "add hair color to person page"

Also..

$ svn save --revert -v2 "add hair color to person page"

.. does the save and THEN drops the CL and it's changes in working copy -
back to no changes but not necessarily up to date.

> Revert files in (active) changelist FOO:
>   svn revert --cl=FOO  # existing syntax

I don't know what this one does Julian?

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Julian Foad wrote:
'Roll back' means retrieving an old version of a shelved change and 
could be named like one or a permutation of these:


   svn unshelve --checkpoint=3 foo
   svn shelf restore foo@3
   svn revert --savepoint=3 --cl=foo
   svn changelist restore -v3 foo

(I am deliberately bringing 'changelist' into the mix, as I believe 
integration with 'changelist' is where we need to take this.)


The integration with changelists that I am thinking of is basically that 
changelist FOO corresponds to the working (unshelved) version of a 
shelved change named FOO. Unshelving or restoring FOO will apply the 
saved change FOO and assign the affected files to changelist FOO; and 
saving or shelving FOO will use the changelist FOO as the set of files 
to save or shelve.


With this changelist integration we could have... (I know we can't 
compatibly have exactly this syntax; see below for alternatives)...


Shelve = save and revert:
  svn cl shelve FOO

Checkpoint = save and don't revert:
  svn cl save FOO

Unshelve / restore / roll back (all possible names for same thing):
  svn cl unshelve [-v VERSION_NUM] FOO
  svn cl restore [-v VERSION_NUM] FOO
  (etc.)

Revert files in (active) changelist FOO:
  svn revert --cl=FOO  # existing syntax
  svn cl revert FOO

Commit the (possibly shelved) changelist FOO:
  svn commit --cl=FOO  # existing syntax
  svn cl commit FOO


Possible different ways to write the topic noun and the verb:

  svn clsave  FOO [PATH...]
  svn cl save FOO [PATH...]
  svn cl --save   FOO [PATH...]
  svn changelist save FOO [PATH...]
  svn changelist-save FOO [PATH...]

  svn save   --cl=FOO [PATH...]


The key point I am making here is the "topic" we're talking about in 
these commands is the "changelist" rather than the "savepoint" which is 
merely a stored version of a changelist.


- Julian

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

[...] I've often wished we'd
made our argument parser slightly more flexible in that respect, so that
[...] we could write

     svn [topic] command [options args]



FWIW, the reason I'm slightly in favour of "savepoint" vs. "checkpoint"
has been discussed before: the abbreviation "sp" is not taken yet, [...]


Ack.



I am thinking for example that we might want to 'send' this change to
a review system or 'commit' it to the svn server (ok we'd use 'svn
commit ' for that) or export it to a patch or email (git
format-patch). The main actions on a Perforce changelist are done by
first-class commands 'p4 submit', 'p4 shelve', etc. while the 'p4
change[list]' command has (option-letter) verbs for 'input' and
'output' actions.

Any suggestions for the 'shelve' terminology problem? Change to
'stash' and incur false expectations based on git stash? Is there
something else that, like 'stash', can be used as a noun and a verb
and sounds good?


IIUC the shelving prototype is a lot closer to Perforce's than Git's,
right? Then thinking about shelves is better than about stashes. Why not
do the same as with (save|check)points: the [topic] should be a noun,
the [command] a verb, so:

     svn shelf save ...
     svn shelf restore ...


Next observation: 'shelves' and 'checkpoints' are not two alternative 
kinds of thing; rather savepoints/checkpoints are versions of a 
(shelved) change.


So 'save this point' and 'roll back to a saved point' need not be in 
grouped in their own 'savepoint' topic namespace but would perhaps be 
better described as actions on a (potentially shelved) change.


'Roll back' means retrieving an old version of a shelved change and 
could be named like one or a permutation of these:


  svn unshelve --checkpoint=3 foo
  svn shelf restore foo@3
  svn revert --savepoint=3 --cl=foo
  svn changelist restore -v3 foo

(I am deliberately bringing 'changelist' into the mix, as I believe 
integration with 'changelist' is where we need to take this.)


I was particularly looking at the case where we have some local 
modifications already and we want to 'roll back' to an earlier version 
of the files. I had been thinking about how Subversion should decide 
which modifications in the WC it should revert before restoring the old 
version. E.g. revert(union(files in requested old version, files in 
most-recently-restored version)). That seems prone to unintended 
consequences as the 'most recently restored version isn't necessarily 
known at all, or correctly known in cases where the user restored 
something and then manually reverted and applied some other change. Also 
the set of files that was previously recorded may not be the same as the 
set of files currently undergoing editing, as it is common to start 
affecting other files. (To handle this sort of case, we need the 
'changelist' mechanism augmented by the notion of assigning new changes 
to a 'default changelist'.)


Instead of all that uncertainty, let the user be responsible for 
reverting the current modifications currently in progress, and then roll 
back using a simple form of 'unshelve' that does not attempt to revert 
anything first. For added safety, 'unshelve' should warn (e.g. require 
'force' to override) if it attempts to modify any already-modified 
files. (This was already a suggested enhancement.)


- Julian

Re: Checkpointing v1 design -- terminology

[Daniel Shahaf]

Branko Čibej wrote on Sat, 04 Nov 2017 13:51 +0100:
> Traditionally we've striven to appear marginally literate in the text we
> produce, that includes command structure. There's no harm in trying to
> adhere to that tradition, especially as it does not cost us more than
> some chattering on the developers' list.
> 
> "Checkpoint" is not a verb. Written English has deteriorated to the
> toddler level in the last several decades, but I see no reason for us to
> follow suit.

I agree that we should use correct English but I don't see how using
"checkpoint" as a verb would be a problem in any way.

Let's just agree to disagree.

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 04.11.2017 00:21, Daniel Shahaf wrote:
> But I don't understand what's the problem that we're trying to solve
> here.  Why is it a problem that "shelf" is a noun and "shelve" a verb?
> And for that matter, why couldn't we use "checkpoint" as a verb?

Traditionally we've striven to appear marginally literate in the text we
produce, that includes command structure. There's no harm in trying to
adhere to that tradition, especially as it does not cost us more than
some chattering on the developers' list.

"Checkpoint" is not a verb. Written English has deteriorated to the
toddler level in the last several decades, but I see no reason for us to
follow suit.

-- Brane

Re: Checkpointing v1 design -- terminology

[Branko Čibej]

On 03.11.2017 22:51, Julian Foad wrote:
> Branko Čibej wrote:
>>>    * main commands are 'checkpoint' to save a new version and
>>> 'rollback' to revert to version N
>>
>> The main issue I have with this is that "checkpoint" is not a verb. (I
>> know, neither are 'changelist' or 'auth').
>>
>> How about 'save' (to create a savepoint) and 'restore' instead?
>>
>> As to why I'm nitpicking at this point: whatever we choose will be
>> pretty much set in stone, in the UI and the API, for the conceivable
>> future.
>
> Thanks for helping with the terminology. I agree it's important.
>
> The prototype CLI I'm working on (in the 'shelve-checkpoint' branch)
> actually has 'svn checkpoint save' and 'svn checkpoint rollback'. Do
> you think mean to suggest plain 'svn save' and 'svn restore'?

No, the two-keyword approach is actually better. I've often wished we'd
made our argument parser slightly more flexible in that respect, so that
instead of

    svn command [options args]

we could write

    svn [topic] command [options args]

which would be useful for topics such as "(save|check)point",
"changelist", "auth", "branch" and so on. I'm absolutely in favour of
the second form for complex cases.

[...]

> A reason for choosing a noun (phrase) such as 'checkpoint',
> 'savepoint', 'changelist' or 'changeset' followed by a verb is to
> allow us to add verbs without cluttering the main command name space.

Precisely. +1

FWIW, the reason I'm slightly in favour of "savepoint" vs. "checkpoint"
has been discussed before: the abbreviation "sp" is not taken yet, but
"cp" is; I suspect it would save "power users" of the command line a ton
of grief if they could write "svn sp s[quash] 0:-3" instead of "svn cp
squash --from 0 --to -3" and then get a syntax error. :)


> I am thinking for example that we might want to 'send' this change to
> a review system or 'commit' it to the svn server (ok we'd use 'svn
> commit ' for that) or export it to a patch or email (git
> format-patch). The main actions on a Perforce changelist are done by
> first-class commands 'p4 submit', 'p4 shelve', etc. while the 'p4
> change[list]' command has (option-letter) verbs for 'input' and
> 'output' actions.
>
> Any suggestions for the 'shelve' terminology problem? Change to
> 'stash' and incur false expectations based on git stash? Is there
> something else that, like 'stash', can be used as a noun and a verb
> and sounds good?

IIUC the shelving prototype is a lot closer to Perforce's than Git's,
right? Then thinking about shelves is better than about stashes. Why not
do the same as with (save|check)points: the [topic] should be a noun,
the [command] a verb, so:

    svn shelf save ...
    svn shelf restore ...

I'm not sure about the abbreviation here ... "sh" sounds tempting but is
also a well-known abbreviation for "shell".

-- Brane

Re: Checkpointing v1 design

[Vincent Lefevre]

On 2017-11-03 19:39:51 +0100, Branko Čibej wrote:
> On 03.11.2017 15:54, Julian Foad wrote:
> > After playing with two pre-prototypes and discussing a wide variety of
> > ideas on this list, I have given detailed thought to a v1
> > checkpointing design with the following properties:
> >
> >   * local (not server)
> >   * built on shelving v1 (patches)
> >   * main commands are 'checkpoint' to save a new version and
> > 'rollback' to revert to version N
> 
> The main issue I have with this is that "checkpoint" is not a verb. (I
> know, neither are 'changelist' or 'auth').
> 
> How about 'save' (to create a savepoint) and 'restore' instead?

Wouldn't it be better if these related commands have a common prefix?
A bit like property-related commands start with "prop".

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Checkpointing v1 design -- terminology

[Daniel Shahaf]

Julian Foad wrote on Fri, Nov 03, 2017 at 21:51:37 +:
> Branko Čibej wrote:
> > >    * main commands are 'checkpoint' to save a new version and
> > > 'rollback' to revert to version N
> > 
> > The main issue I have with this is that "checkpoint" is not a verb. (I
> > know, neither are 'changelist' or 'auth').
> > 
> > How about 'save' (to create a savepoint) and 'restore' instead?
> > 
> > As to why I'm nitpicking at this point: whatever we choose will be
> > pretty much set in stone, in the UI and the API, for the conceivable future.
> 
> Thanks for helping with the terminology. I agree it's important.
> 
> The prototype CLI I'm working on (in the 'shelve-checkpoint' branch)
> actually has 'svn checkpoint save' and 'svn checkpoint rollback'. Do you
> think mean to suggest plain 'svn save' and 'svn restore'? I'm not sure about
> that. They could deserve to be first-class 'svn' commands if we consider
> them first-class companions to 'commit', 'revert' and 'diff'.
> 
> The analogy to 'save' a file or the work in almost any application is apt.
> (For the converse we should then consider 'open' or 'load', but neither of
> these sound good.)

"edit"?  Inspired by Vim's :edit command; goes well with the "prototype
commit" mental model.

> Any suggestions for the 'shelve' terminology problem? Change to 'stash' and
> incur false expectations based on git stash? Is there something else that,
> like 'stash', can be used as a noun and a verb and sounds good?

pile, pocket, fold, store, house, dock, box, rack, hold, stow.

But I don't understand what's the problem that we're trying to solve
here.  Why is it a problem that "shelf" is a noun and "shelve" a verb?
And for that matter, why couldn't we use "checkpoint" as a verb?

Cheers,

Daniel

Re: Checkpointing v1 design -- terminology

[Julian Foad]

Branko Čibej wrote:

   * main commands are 'checkpoint' to save a new version and
'rollback' to revert to version N


The main issue I have with this is that "checkpoint" is not a verb. (I
know, neither are 'changelist' or 'auth').

How about 'save' (to create a savepoint) and 'restore' instead?

As to why I'm nitpicking at this point: whatever we choose will be
pretty much set in stone, in the UI and the API, for the conceivable future.


Thanks for helping with the terminology. I agree it's important.

The prototype CLI I'm working on (in the 'shelve-checkpoint' branch) 
actually has 'svn checkpoint save' and 'svn checkpoint rollback'. Do you 
think mean to suggest plain 'svn save' and 'svn restore'? I'm not sure 
about that. They could deserve to be first-class 'svn' commands if we 
consider them first-class companions to 'commit', 'revert' and 'diff'.


The analogy to 'save' a file or the work in almost any application is 
apt. (For the converse we should then consider 'open' or 'load', but 
neither of these sound good.)


Saving the work-in-progress is also like a deferred not-quite commit, so 
some variant of the 'commit' command such as 'svn commit [--]save' might 
be appropriate.


A reason for choosing a noun (phrase) such as 'checkpoint', 'savepoint', 
'changelist' or 'changeset' followed by a verb is to allow us to add 
verbs without cluttering the main command name space. I am thinking for 
example that we might want to 'send' this change to a review system or 
'commit' it to the svn server (ok we'd use 'svn commit ' for 
that) or export it to a patch or email (git format-patch). The main 
actions on a Perforce changelist are done by first-class commands 'p4 
submit', 'p4 shelve', etc. while the 'p4 change[list]' command has 
(option-letter) verbs for 'input' and 'output' actions.


Any suggestions for the 'shelve' terminology problem? Change to 'stash' 
and incur false expectations based on git stash? Is there something else 
that, like 'stash', can be used as a noun and a verb and sounds good?


- Julian

Re: Checkpointing v1 design

[Branko Čibej]

On 03.11.2017 15:54, Julian Foad wrote:
> After playing with two pre-prototypes and discussing a wide variety of
> ideas on this list, I have given detailed thought to a v1
> checkpointing design with the following properties:
>
>   * local (not server)
>   * built on shelving v1 (patches)
>   * main commands are 'checkpoint' to save a new version and
> 'rollback' to revert to version N


The main issue I have with this is that "checkpoint" is not a verb. (I
know, neither are 'changelist' or 'auth').

How about 'save' (to create a savepoint) and 'restore' instead?

As to why I'm nitpicking at this point: whatever we choose will be
pretty much set in stone, in the UI and the API, for the conceivable future.

-- Brane

Re: Checkpointing v1 design

[Julian Foad]

Listing shelved and applied change-sets could look something like this:

[[[
$ svn shelves
applied:
  X (at version 1)
  thing (at version 10)
shelved:
  foo (1 version)
  bar (4 versions)
  old-thing (1 version)
  new-changes (18 versions)
]]]

or integrated with changelists:

[[[
$ svn changelist --list
--- Applied Changelists:
  X (checkpoint 1, 1 file)
  thing (checkpoint 10, 2 files)
--- Shelved Changelists:
  foo (checkpoint 1, 14 files)
  bar (checkpoint 4, 1 file)
  old-thing (checkpoint 1, 2 files)
  new-changes (checkpoint 18, 3 files)
]]]

Listing checkpoints:

[[[
$ svn checkpoint list [--cl=]thing
thing-1 1230 mins old874 bytes
 (no log message)
 hello.txt |1 +
 1 file changed, 1 insertion(+)
...
thing-10  10 mins old  12345 bytes
 log message: Blah blah blah...
 hello.txt |1 +
 bar.txt   |   12 
 2 files changed, 13 insertions(+)
]]]

- Julian

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Tue, Aug 1, 2017 at 9:19 AM, Julian Foad  wrote:
> Keep in mind that a revision -- or any versioned state -- can be
> thought of in two ways: as a complete snapshot of the tree, or as
> representing a change relative to the previous snapshot.
>
> First let us be sure we understand that updating *is* rebasing. To
> explain this, think about a plain WC with no checkpointing. The WC
> has a base tree and a working tree. The work you do in a working copy
> is to create a change based on the base. An 'update' is a request to
> update the base and also to adjust the working version so that it
> represents the 'same' change that it previously did, except the
> change is now against the new base. Hence update is 'rebasing' your
> local change. The adjustment of the old change (against the old base)
> to a new change (against the new base) is accomplished by a merge.
>
> Illustration: a plain old WC based on r20 is like a one-commit local
> branch based on r20.
>
> [ repo -(r10)--(r20)--(r30)--(r40)---
> [||
> [||
> [ WC   (BASE)-(WORK)
>
> [(X) represents a tree snapshot. WORK means the working/on-disk state
> in the WC, including what libsvn_wc calls 'actual'.]
>
> Updating to r30 involves merging (r20:30) with (rBASE:WORK) to create
> (WORK'), and setting the new base to r30.
>
> [ repo -(r10)--(r20)--(r30)--(r40)---
> [   ||
> [   ||
> [ WC  (BASE')-(WORK')

I appreciate that you've taken the time and effort to explain this,
complete with graphs. I suppose I should have known that an update
rebases the working copy onto the new revision, but for whatever reason
I was under the impression that it was the other way around, that the
incoming changes get applied on top of the working copy.

The only drawback of the rebase is that update is a one-way operation,
with no 'undo' mechanism. It's the only place in Subversion where you
could potentially lose a very particular configuration of your files,
which may have taken some tricky manipulations to arrive at, and the
nature of update (being the action that immediately precedes a commit)
is such that your intention was probably to save those bits, not change
them. Currently in such a situation, the user should mitigate that by a
svn copy to a new branch, or adopt a policy of branching for everything
and avoid this issue altogether. Personally I've been known to copy the
contents of my working copy to a temp directory before performing an
update, just in case. :-) But the premise of version control is that
you should never fear losing anything. That makes update the perfect
use case and showcase for a checkpoint feature, even if the user never
takes advantage of checkpoints otherwise.

Referring to this last graph:

> (b) Keep the existing checkpoints based on the older base, and bring
> in the new base to be used only for new checkpoints. For this
> illustration, assume we had no outstanding working changes before
> running the update. We merge the incoming changes with the *complete*
> checkpoint series and record the result as (Pn').
>
> [ repo -(r10)--(r20)--(r30)--(r40)---
> [|| ||
> [|| ||
> [|  (P0)  (P0')
> [|\   \
> [ WC | \   \
> [|  \   \
> [| (P1)-...-(Pn)--(Pn'=BASE)
> [ \
> [  \_(WORK')
>
> The WC now contains two "base" snapshots (P0 and P0') copied from the
> repository, and (n) checkpoint snapshots, and a merged checkpoint.

If checkpoints are implemented using a hidden repo, is it feasible to
place two directories, BASE and WORK, at its root, and save the BASE
with each checkpoint? If so, Subversion's atomic repo-wide commits
would automatically solve the issue of keeping each checkpoint
synchronized with its base, since both would be saved together. The
repo's temporal compression would mean that very little additional data
would be saved, in comparison to the size of BASE.

If this is feasible, then I would treat checkpoints as a sort of time
line, rather than a private branch. That is, each action against the
checkpoints, whether to create a new checkpoint, perform an update, or
rollback to an earlier checkpoint, would create a new checkpoint with
the contents of the current BASE and WORK. So, rolling *back* (to an
earlier, in time, checkpoint) could actually roll your BASE *forward*.
The beauty is that since both BASE and WORK are saved in the hidden
repo, no server connection is needed to roll back or forward through
the point of a BASE change.

Here's an illustration of what might happen over the course of a
workday:

Time  | Action   | Hid

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

Nathan Hartman wrote:
> Julian Foad wrote:
> Performing an 'update' with a checkpoint series is a bigger ask than it
> might at first seem. In effect, it requires rebasing the series of
> checkpoints on the new base, which gets ugly because of the need to
> handle conflicts (which is ugly enough already in the existing
> single-depth WC).
> 
> Why does update with a checkpoint require rebasing? [...]

[tl;dr: Update fundamentally involves rebasing the working change.
Rebasing all the checkpoints as well is one approach but has drawbacks.
We examine other approaches.]


==

Keep in mind that a revision -- or any versioned state -- can be thought
of in two ways: as a complete snapshot of the tree, or as representing a
change relative to the previous snapshot.

First let us be sure we understand that updating *is* rebasing. To
explain this, think about a plain WC with no checkpointing. The WC has a
base tree and a working tree. The work you do in a working copy is to
create a change based on the base. An 'update' is a request to update
the base and also to adjust the working version so that it represents
the 'same' change that it previously did, except the change is now
against the new base. Hence update is 'rebasing' your local change. The
adjustment of the old change (against the old base) to a new change
(against the new base) is accomplished by a merge.

Illustration: a plain old WC based on r20 is like a one-commit local
branch based on r20.

[ repo -(r10)--(r20)--(r30)--(r40)---
[||
[||
[ WC   (BASE)-(WORK)

[(X) represents a tree snapshot. WORK means the working/on-disk state in
the WC, including what libsvn_wc calls 'actual'.]

Updating to r30 involves merging (r20:30) with (rBASE:WORK) to create
(WORK'), and setting the new base to r30.

[ repo -(r10)--(r20)--(r30)--(r40)---
[   ||
[   ||
[ WC  (BASE')-(WORK')


==

Now, what is a checkpoint series? We are exploring a couple of different
definitions.

1. The saved-patches definition: a series of patches, each representing
the state of the WC at the time it was saved, enabling an earlier state
of the WC to be recovered. (The 'option 1' design implements the patches
part of this, and may later be extended to record the WC base.)

[ repo -(r10)--(r20)--(r30)--(r40)---
[||
[||
[ WC   (BASE)
[ _(P1)
[  \\\_...
[   \\_(Pn)
[\_(WORK)

2. The local-branch definition: a series of changes, rather like
revisions in a repository (and implemented as such in 'option 3'
design). The zero'th checkpoint "P0" is (a copy of) what was the
original WC base tree. Then there are successive snapshots P1, P2... Pn
and then a working change which is currently being edited. This behaves
like you would expect from a private, local branch in Subversion based
on the original base point from the original repository.

[ repo -(r10)--(r20)--(r30)--(r40)---
[||
[||
[ WC(P0)-(P1)-...-(Pn=BASE)
[   \
[\_(WORK)


Each of these two definitions of checkpointing suggests its own
conceptual approaches to updating.

1. For saved-patches checkpointing, we might update the WC base and
working version, and start saving any new checkpoints relative to the
new base.

[ repo -(r10)--(r20)--(r30)--(r40)---
[|| ||
[|| ||
[ WC   (BASE)  (BASE')
[_(P1) \\_... <= (Pn+1) will go here
[ \\\_...   \
[  \\_(Pn)   \
[   \ \
[\_(WORK)  \_(WORK')

A consequence of different checkpoints having different bases is that
rolling back (or forward) to a checkpoint based on a different base
would require one of:

  * apply the patch using fuzzy matching
  * (if online) rebase the desired checkpoint at this time
  * (if online) update the WC base to the checkpoint's recorded base

An alternative would be that updating attempts to rebase every recorded
checkpoint patch, but see 2(a) below.

2. For local-branch checkpointing, there are at least two ways we could
update:

(a) Update P0 and successively rebase all the changes represented by the
checkpoints: rewrite P1 (by merging) to be based on P0, rebase P2 on P1,
and so on.

[ repo -(r10)--(r20)--(r30)--(r40)---
[   ||
[   ||
[ WC  (P0')-(P1')-...-(Pn'=BA

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Fri, Jul 28, 2017 at 8:19 AM, Julian Foad  wrote:

> Nathan Hartman wrote:
> All considered, it feels to me like we have neither the collective will
> nor the resources to take this in the direction of full local branching.
> I can't commit myself to that. The present work should strive to keep
> that option open, largely because a successful open-source software
> system is one that others can adapt to their needs later on.
>
> Rather, for me, this line of thought helps to clarify the differences
> between what we are and are not creating.
>

Agreed. Just to emphasize, I don't suggest going the route of full
local branching today. That involves challenges that require some
serious thought, not only implementation but also user perspective:
making it easy to learn and easy to use, like the rest of Subversion.


> > (2) [...] "svn update" should perform an automatic implicit
> > "checkpoint save" [...] If an update turns ugly, you will be able to do
> > that 3-way diff I was just talking about.
>
> Or you could revert the update. Yes, I would love to have this feature too.
>
> Performing an 'update' with a checkpoint series is a bigger ask than it
> might at first seem. In effect, it requires rebasing the series of
> checkpoints on the new base, which gets ugly because of the need to
> handle conflicts (which is ugly enough already in the existing
> single-depth WC).
>

Why does update with a checkpoint require rebasing? There must be a
large gap in my understanding here. Forgive me if I misunderstood, but
I thought a checkpoint is a snapshot of what the working copy looked
like at the time it was taken, to provide a sort of high-level
undo/redo capability. Suppose I did a checkout at 8:00 AM and then did
some work, taking snapshots of what my working copy looked like at
9:00 AM, 11:00 AM, and 2:00 PM. At 2:30 PM I invoke svn update. I
would expect to end up with four snapshots: 9:00 AM, 11:00 AM, 2:00
PM, 2:30 PM; and I would expect the working copy, after the update, to
be the same as though I checked out at 8:00 AM, worked until 2:30 PM,
and did the update, exactly as it occurs today. In other words, I
would expect my snapshots to remain untouched and I would expect the
update to be the "next step" in the working copy contents' linear
evolution. I could potentially continue working and taking snapshots,
and make subsequent updates as well. The implementation would probably
have to record the BASE at the time of each checkpoint, so that if you
rollback to it, passing through an "update" point, it could restore
that as the working copy's BASE along with restoring the working
copy's contents to those of the snapshot.


> Initially, I suggest 'update' and 'switch' will be incompatible with
> checkpoints. The user must first squash the checkpoints.
>

Agreed. This is reasonable for a first implementation.

As an enhancement, on running 'update', Subversion could 'squash' the
> checkpoints and shelve the resulting patch (but without reverting the
> WC). That would be enough to allow better manual 'undo' if the update
> went badly and if the user can remember the previous base state
> (revision/revisions). If the shelving function would also save a
> description of the base state, so much the better.
>

I don't like the idea of mixing checkpoints and updates with shelving.
It seems too kludgy to me because shelving is a way to temporarily
interrupt work and come back to it later, whereas updating is a linear
step forward, not an interruption. I propose this alternative: "svn
checkpoint squash" will in effect discard all checkpoints and take a
new checkpoint of the working copy. Following my earlier example of
four checkpoints, if I squash at 4:00 PM, I'll end up with just one
checkpoint with the 4:00 PM timestamp, identical to my working copy.
If "update" performs this squash operation (or simply takes a
checkpoint if none exists), then I believe it's a more logical
solution.


> > [...] I suggest ability to specify a log message [...]
>
> A log message option is intentionally omitted for now. My thinking is
> that specifying a log message reinforces the idea that each checkpoint
> commit will eventually become a central-repo commit, and with that in
> mind, the user's expectations would be a little different.


Fair enough. I'm in 100% agreement that this implementation should
focus on simplicity and reliability.

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

Nathan Hartman wrote:
> Some thoughts (probably too many)...

Not at all, this is fantastic, right where I want us to be, and very
helpful.

> I agree that 'init' and 'finish' should happen automatically.
> [...] As for when the 'finish' should occur, [...] probably not
> until the results have been squashed and committed.
> 
> As far as what to name the commands, if the "checkpoint" command is
> changed to "local" and the "save" subcommand is changed to "commit" then
> you get:
> 
> 1. svn local commit
> 2. svn local revert
> 3. svn local rollback
> 4. svn local list|log
> 5. svn local squash
> 6. svn commit   <--- to the central repository
> 
> I think users will immediately start getting ideas about what this
> means. Is that a good thing? It depends on what the goals are for
> checkpoints, how checkpoints are intended to fit into the larger
> workflow, and strategy for the future in general.

Excellent: this shows really clearly one way of thinking about
checkpointing.

> If checkpoints are intended to be merely a convenience with a limited
> purpose, then I would call the command "checkpoint" and "checkpoint
> save" to avoid confusion and make documentation, teaching, and learning
> easier.
> 
> If checkpoints are intended to have a much wider application, eventually
> providing an answer to git's/DVCSs' ability to work offline and/or to do
> personal work on projects to which one lacks commit privileges, then I
> would use the shorter "local" [...] and try to keep the sub-sub-commands
> as close in semantics and function as possible to Subversion's original
> command set.

Agreed.

> [...] Working on the
> local machine essentially means working on a private branch. With that
> in mind, to answer questions such as: whether a status or diff should be
> against the original base or against the checkpoint, such questions
> should always be answered consistently with how server side operations
> are carried out today.

Agreed.

> [...] if this feature is supposed to have such broad scope [...]
> users will have immediate sky-high expectations [...]; if that
> kind of future appeals to enough people, then the present work should
> [maintain] such a parallel between this new "local branching" and the
> original server-side branching.

All considered, it feels to me like we have neither the collective will
nor the resources to take this in the direction of full local branching.
I can't commit myself to that. The present work should strive to keep
that option open, largely because a successful open-source software
system is one that others can adapt to their needs later on.

Rather, for me, this line of thought helps to clarify the differences
between what we are and are not creating.

> Some more thoughts...
> 
> (1) It will eventually be necessary to create a new "URL specifier" or
> some sort of syntax to direct an operation into the checkpoint repo.
> This will make it possible to diff between any permutation of working
> copy, central repository, and local checkpoints, which will be crucial
> once users begin to use checkpoints as local branches. I can see a use
> case where a 3-way diff between all three is useful.

I'm not yet sure how important is diffing (2-way and 3-way) between
central repo and checkpoints. If we treat checkpointing as intermediate
WC states rather than as local branching then providing a syntax for
these kinds of operation is arguably less important. It still would be
of some use, just as 'rBASE' is useful, but not that 'rWORKING' has
always been missing from the UI despite a filed request for it, so there
is no syntax to request "diff -rWORKING:BASE" for example.


> (2) [...] "svn update" should perform an automatic implicit
> "checkpoint save" [...] If an update turns ugly, you will be able to do
> that 3-way diff I was just talking about.

Or you could revert the update. Yes, I would love to have this feature too.

Performing an 'update' with a checkpoint series is a bigger ask than it
might at first seem. In effect, it requires rebasing the series of
checkpoints on the new base, which gets ugly because of the need to
handle conflicts (which is ugly enough already in the existing
single-depth WC).

Initially, I suggest 'update' and 'switch' will be incompatible with
checkpoints. The user must first squash the checkpoints.

As an enhancement, on running 'update', Subversion could 'squash' the
checkpoints and shelve the resulting patch (but without reverting the
WC). That would be enough to allow better manual 'undo' if the update
went badly and if the user can remember the previous base state
(revision/revisions). If the shelving function would also save a
description of the base state, so much the better.

I do intend to give more thought to the more advanced possibilities in
this direction, though, like rebasing, just to see what it leads to.


> (3) Eventually, squashing should be optional;

By which you mean it should be possible to replay all the local commits
separately.

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Thu, Jul 27, 2017 at 7:56 AM, Julian Foad  wrote:

> What can we learn from this mock-up? Here are some of my thoughts

(snip)

> If I checkpoint my changes, do I then want to see 'status' and 'diff'
>
against the original base or against the checkpoint? Both are useful.
>
(snip)

> Probably 'checkpoint init' should not be needed, and instead running
> 'checkpoint save' should initialize the checkpointing repo if not
> already done.
>
> Probably 'checkpoint finish|uninit' should not be needed, and instead
> should happen automatically when the user either commits or reverts the
> entire set of changes.
>

Some thoughts (probably too many)...

I agree that 'init' and 'finish' should happen automatically. Creation and
destruction of a hidden repository (or any data structure) is an
implementation detail that could change in the future and should not be
leaked. As you've already said, the 'init' should occur the first time a
checkpoint operation is done. As for when the 'finish' should occur, that
is a question I've been grappling with... I would say, probably not until
the results have been squashed and committed.

As far as what to name the commands, if the "checkpoint" command is changed
to "local" and the "save" subcommand is changed to "commit" then you get:

1. svn local commit
2. svn local revert
3. svn local rollback
4. svn local list|log
5. svn local squash
6. svn commit   <--- to the central repository

I think users will immediately start getting ideas about what this means.
Is that a good thing? It depends on what the goals are for checkpoints, how
checkpoints are intended to fit into the larger workflow, and strategy for
the future in general.

If checkpoints are intended to be merely a convenience with a limited
purpose, then I would call the command "checkpoint" and "checkpoint save"
to avoid confusion and make documentation, teaching, and learning easier.

If checkpoints are intended to have a much wider application, eventually
providing an answer to git's/DVCSs' ability to work offline and/or to do
personal work on projects to which one lacks commit privileges, then I
would use the shorter "local" (unless someone can think of a better,
shorter, and more descriptive name) and try to keep the sub-sub-commands as
close in semantics and function as possible to Subversion's original
command set. Even though the word "local" is technically a command, we
pretend it's more of a mode specifier that redirects what would normally be
server side operations to the local machine instead. Working on the local
machine essentially means working on a private branch. With that in mind,
to answer questions such as: whether a status or diff should be against the
original base or against the checkpoint, such questions should always be
answered consistently with how server side operations are carried out
today. Of course, if this feature is supposed to have such broad scope,
there is the (not insignificant) issue that users will have immediate
sky-high expectations that will take significant additional time and effort
to fulfill, so clearly this is not by any means an immediate goal; it's a
*possible* future direction, and if that kind of future appeals to enough
people, then the present work should try to be compatible with that future
by maintaining (to the extent possible) such a parallel between this new
"local branching" and the original server-side branching.

Some more thoughts...

(1) It will eventually be necessary to create a new "URL specifier" or some
sort of syntax to direct an operation into the checkpoint repo. This will
make it possible to diff between any permutation of working copy, central
repository, and local checkpoints, which will be crucial once users begin
to use checkpoints as local branches. I can see a use case where a 3-way
diff between all three is useful.

(2) Once checkpoints are working reliably, I suggest that "svn update"
should perform an automatic implicit "checkpoint save" (or whatever the
command is called) before attempting to fetch from the repository and merge
with the working copy. This will add a big safety net to what can be a
dreaded process. If an update turns ugly, you will be able to do that 3-way
diff I was just talking about.

(3) Eventually, squashing should be optional; that said, it's not feasible
to checkout a WC from a trunk that has lots of activity on it, work locally
for weeks, and then expect to replay the local commits onto the trunk
successfully. So I suggest this alternative: You can checkout a working
copy and make as many checkpoints as you want. When ready to commit, one of
the following happens:

(a) you squash, update, and commit (which will turn into an infinite
loop if item #1 above is implemented and update creates a checkpoint...
hmmm... not good)

(b) you can replay the commits to the server only if no other commits
have been made in the meantime (to trunk or whichever directory originates
the checkout)

(c) you do not wish to s

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

What can we learn from this mock-up? Here are some of my thoughts.

* Status and diff against last incremental change vs. against the
'commit base'.

* The 'init' and 'finish' work flow should be simpler.

* s/w architecture: exporting from WC, changing the WC base, ...


== Status and diff against last incremental change vs. against the
'commit base'

If I checkpoint my changes, do I then want to see 'status' and 'diff'
against the original base or against the checkpoint? Both are useful.

This is completely analogous to working on a branch. Sometimes I want to
review just my latest change on the branch, and sometimes I want to
review what I have changed in total on the branch, in other words what I
will be merging back to the parent branch. In this regard, checkpointing
is like a local branch.

For backward compatibility with Subversion's user interface, we have to
choose one base or the other as the 'default' diff/status base. Ideally
we will make both bases support exactly the same enquiry features, and
the selection of the default base will be merely a user-interface
convenience.

(Long ago I drafted a patch to make it easy to run a diff against the
merge base, when working on a branch, because doing this useful task
manually was fiddly. The UI looked like 'svn diff --against-merge-base'
or some such option, nothing more needed to be specified, and it invoked
the merge base-finding code and then ran a diff against the found base.)

Diff and status are the obvious commands which compare against a base,
where choosing the base will be appropriate. I need to check what other
commands it might also apply to.

Commands that run back through history, especially 'log' and 'blame',
also need some adjustment. In the mock-up right now, these just report
what is in the local repo. We might instead prefer 'log' and 'blame' to
show revisions from the original repository (assuming it's online), so
they keep their original meaning and ignore checkpoints. Or we might
prefer them to show both the checkpoints and the original repository
revisions.


== The 'init' and 'finish' work flow should be simpler.

Probably 'checkpoint init' should not be needed, and instead running
'checkpoint save' should initialize the checkpointing repo if not
already done.

Probably 'checkpoint finish|uninit' should not be needed, and instead
should happen automatically when the user either commits or reverts the
entire set of changes.

The 'checkpoint squash' command is provided only for experimentation.
Squashing back to a plain WC with local mods is something that happens
as part of 'finish', and would be needed within a future 'checkpoint
commit' that commits everything in one revision. I don't think it is
needed separately.


== s/w architecture: exporting from WC, changing the WC base, ...

Some APIs that would be useful do not currently exist.

First, I wanted to copy the WC base into a new repository. Ideally the
WC will expose standard 'editor' API to export the base, and we would
pipe this into a commit editor. At the moment, the 'export' command
instead uses the WC's 'walk status' API with a set of callbacks that
write the exported data onto disk, where as 'export' from a repository
uses a standard 'editor' with a different set of callbacks that write
the exported data onto disk.

The 'squash' code is implemented by updating the WC back to r1 in the
checkpoints repo, and then merging all the checkpoints repo content
(r1:HEAD) into the WC working state, without mergeinfo. Instead, it
would be nice to have an API that directly updates the WC base back to
r1 while adjusting the 'scheduling' of each path (add/del/modify) so as
to keep the WC working state constant. (That is, 'rebasing' the WC.)


Your thoughts, anyone?

In particular, how well is the work flow shaping up, if we assume we
will eliminate the 'init' and teach 'revert' and 'commit' to do the
'finish' actions automatically?

- Julian


On 26/07/17 16:13, Julian Foad wrote:
> I committed an initial prototype for checkpointing backed by a local
> repo embedded in the WC (the "option 3" design), on the
> "shelve-checkpoint3" branch.
> 
> http://svn.apache.org/r1803046 and follow-ups.
> 
> Here is the help output:
> [[[
> $ svn checkpoint --help
> checkpoint: Checkpoint the local changes.
> usage: 0. checkpoint init
>1. checkpoint save
>2. checkpoint revert
>3. checkpoint rollback NUMBER
>4. checkpoint list|log
>5. checkpoint squash
>6. checkpoint finish|uninit
[...]
> 
> If you read this far and get a chance to try it out, I'd love to hear
> your thoughts on the overall shape and the big-picture possibilities. (I
> am not so concerned about details at this stage, but mention those too
> and hopefully they will spark further thoughts.)

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Julian Foad]

Nathan Hartman wrote:
> Julian Foad wrote:
> I committed an initial prototype for checkpointing backed by a local
> repo embedded in the WC (the "option 3" design), on the
> "shelve-checkpoint3" branch.
> 
> http://svn.apache.org/r1803046 and follow-ups.
> 
> Woohoo! Very exciting! 
> 
> I didn't quite understand the difference between uninit and finish,

They're synonyms. I was just trying out different names to see how they
sound.

> but
> that's okay. I'll re-read more carefully soon. More important to try it.
> I'll be back with thoughts...

Looking forward to hearing them. Thanks!

- Julian

Re: Checkpointing mock-up option 3 (backed by a local repo)

[Nathan Hartman]

On Wed, Jul 26, 2017 at 11:13 AM, Julian Foad  wrote:

> I committed an initial prototype for checkpointing backed by a local
> repo embedded in the WC (the "option 3" design), on the
> "shelve-checkpoint3" branch.
>
> http://svn.apache.org/r1803046 and follow-ups.
>

Woohoo! Very exciting!

I didn't quite understand the difference between uninit and finish, but
that's okay. I'll re-read more carefully soon. More important to try it.
I'll be back with thoughts...

Re: Checkpointing - is not Quilt

[Julian Foad]

Branko Čibej wrote:

In order to use checkpoints as a sort of local commit, you do have to be
able to push a series of commits to the server. You'd distinguish
explicit checkpoints (with log messages) and implicit/automatic
checkpoints; only the explicit ones would constitute separate commits.


That would be lovely. It is beyond what I see as basic checkpointing 
functionality (which is more like a local 'undo' mechanism while 
composing one commit).



In order to support that, you'd have to have a way to rebase the whole
checkpoint stack onto current HEAD (or at least to a non-conflicting
state). Ideally you'd also support operations such as checkpoint
reordering, (local) squashing, removal, etc.


Yes. I'll aim to design in such a way as to make those kinds of things 
possible in future extensions.


I like the idea of being able to have both explicit and 
implicit/automatic/lightweight checkpoints and distinguish them. The 
more I think about automatic checkpoints, the more I see this as 
bringing to Subversion's WC the 'auto-save' and 'undo' features that 
nearly all user-friendly editor applications offer. I can see a GUI 
wanting to make frequent automatic checkpoints so that the user doesn't 
have to remember to do so.


- Julian

Re: Checkpointing - is not Quilt

[Branko Čibej]

On 17.07.2017 13:42, Julian Foad wrote:
> Initially I wrote that 'checkpointing' should allow committing the
> result either all at once or as a corresponding series of commits.
> Having thought more, I believe the use case for a series of commits
> falls out of scope.
>
> Definitions:
>
>   Quilt:
>
> Preparing a series of patches that will be applied (perhaps
> committed) sequentially in a given order.
> Each patch represents a self-contained logical change, likely with
> its own log message.
> Later patches may depend on earlier ones.
> Ability to jump to any patch in the series and modify it in the
> context of the earlier patches having been applied and the later ones
> not.
> After such a modification, updating (as in rebasing) the later
> patches accordingly.
> Ability to commit the series as a corresponding series of commits,
> or all in one commit.
>
>   Checkpointing:
>
> Saving intermediate, unfinished, working states during the
> preparation of a single logical change.
> The change will be committed as one revision when finished.
> Changing a state that has been checkpointed may be accomplished
> either by rolling back to an earlier state and then re-doing all
> subsequent changes in the desired way, or by leaving that checkpoint
> in place and making a further checkpoint after changing the working
> state in the desired way.
> Like an "undo stack", with or without "redo".


In order to use checkpoints as a sort of local commit, you do have to be
able to push a series of commits to the server. You'd distinguish
explicit checkpoints (with log messages) and implicit/automatic
checkpoints; only the explicit ones would constitute separate commits.

In order to support that, you'd have to have a way to rebase the whole
checkpoint stack onto current HEAD (or at least to a non-conflicting
state). Ideally you'd also support operations such as checkpoint
reordering, (local) squashing, removal, etc.

-- Brane