Re: How SVN works wrt copies of versions - Was: Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-12-05 Thread Josef Wolf 
On Sun, Dec 03, 2006 at 04:20:10AM +0100, EdorFaus wrote:
> On 11/30/06, Tin Newsom <[EMAIL PROTECTED]> wrote:

> > If you are keeping a copy of the current versions locally then diffing and
> > sending only the diffs would be easy... but to my knowledge svn only keeps
> > the diffs between versions at the repository. Someone who knowns please
> As far as I know, this is essentially correct - the repository contains the 
> original file, and the diff for each version since then. This is, afterall, 
> enough to recreate any version.

Actually, it's a little bit more tricky.  They don't keep consecutive
diffs but diffs to get to the next lower 2^N revision.  With this method
they can retrieve a specific revision with O(log N) instead of O(N).
This costs more disk space than storing consecutive diffs.  But since
the repository is on the server, that should not harm in this application.
After all, disks are cheap nowadays.

Google for "delta combiner" or "skip deltas" if you are interested in
details on this.

But the delta's are only internal storage format.  For transport, always
diffs against the "base revision" (see below) are used.  Using apache
for repository serving, those diffs can even be compressed.

> I believe this is exactly what svn does. It certainly keeps a copy of the 
> last-committed (or updated-to) version in every working copy, along with the 
> version being worked on. This allows it to do various things while offline.

Exactly.  It keeps the "base revision" (they call it "text base").  This
is the revision your local files are based on.  You can see which revision
this is with "svn st -v"

When I was lurking at svn mailing lists years ago (it was around svn-0.17
or something), once a month came the request either to remove the text
base completely or to compress it.  I bet the requests are still beeing
made once a month.

> No, but I believe this is how CVS works - SVN on the other hand only
> transmits the diffs in either direction, [ ... ]

Yupp...

> > If you don't keep a repository locally then one way or another you will
> > have to obtain a copy of the old version in order to just sync changes
> > somehow..
> This being, I believe, one of the main reasons SVN does keep a copy (of the 
> latest-known-committed version) locally.

svn-1.4 includes a svnsync utility.  This is meant to replicate/sync
repositories.  Form the description on subversion's homepage I deduce
that svnsync is just a reimplementation of svk.  But I haven't done a
look at it yet, so I can be wrong.

> Some other good reasons are things SVN can then do offline, e.g.
>   - generating diffs (what exactly did I change this time?)
>   - reverting a file (nono, that's no good, let me start over...)
> 
> It can then also more easily update the working copy with new code from the 
> repository, by seeing local diffs vs diffs in the repository.
> 
> It only keeps one previous version in the working copy though, so for 
> comparing with or reverting to earlier versions, it must still be online.

AFAICS, with svn-1.4, this is no longer true.  You can keep a local
copy of the repos at the cost of disk space.  You do your ordinary
work against this local copy.  When you are at home, you sync the
local repos against the main repos with svnsync.  Too bad you can't
attach 1.8 inch USB disks to Neo due to USB beeing non-powered :-(

> > Its an interesting thought.. does anyone know for sure if SVN sends back
> > only the changes and how it does that if it doesn't fetch the previous
> > version for comparison?

SVN have even more selling points:

 - Can be served by apache with SSL, so you get encryption+authentication
 - Can use apache's mod_deflate to compress the diffs before sending them
 - You can setup apache so that you can browse/download newest revision
   with ordinary web-browser.

___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community

How SVN works wrt copies of versions - Was: Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-12-02 Thread EdorFaus 
On 11/30/06, Tin Newsom <[EMAIL PROTECTED]> wrote:

> If you are keeping a copy of the current versions locally then diffing and
> sending only the diffs would be easy... but to my knowledge svn only keeps
> the diffs between versions at the repository. Someone who knowns please

As far as I know, this is essentially correct - the repository contains the 
original file, and the diff for each version since then. This is, afterall, 
enough to recreate any version.

> correct me.  If I understand what you are saying and based on my
> knowledge...
> You either keep an old version for diffing purposes and replace it with the
> current version when you do the commit.  All changes happen to the original
> not the "old copy"

I believe this is exactly what svn does. It certainly keeps a copy of the 
last-committed (or updated-to) version in every working copy, along with the 
version being worked on. This allows it to do various things while offline.


> My understanding of svn is that it transmits the entire file and the svn
> server does the diff and only stores the differences between the files at
> the repository.  I could be wrong about that though.

No, but I believe this is how CVS works - SVN on the other hand only transmits 
the diffs in either direction, which is one of its selling points over CVS - 
iirc one of the things the SVN developers didn't like about CVS that caused 
them to make SVN in the first place.
(Note: this is from my admittedly faulty memory, so might not be correct.)

> 
> If you don't keep a repository locally then one way or another you will have
> to obtain a copy of the old version in order to just sync changes somehow..

This being, I believe, one of the main reasons SVN does keep a copy (of the 
latest-known-committed version) locally.

Some other good reasons are things SVN can then do offline, e.g.
- generating diffs (what exactly did I change this time?)
- reverting a file (nono, that's no good, let me start over...)

It can then also more easily update the working copy with new code from the 
repository, by seeing local diffs vs diffs in the repository.

It only keeps one previous version in the working copy though, so for 
comparing with or reverting to earlier versions, it must still be online.

> 
> Its an interesting thought.. does anyone know for sure if SVN sends back
> only the changes and how it does that if it doesn't fetch the previous
> version for comparison?

See above. :)

Just FYI, that it keeps a copy of the last version locally is something I've 
gotten quite familiar with through the use of grep... When I want to find 
something in the working copy, I usually  grep whatever | grep -v .svn  to 
avoid getting duplicates or finding things I've already changed locally :P

> 
> --Tim
> 
> btw.. I tend to over explain things so if I don't go far enough please ask
> me (I am trying to cut back)  ;-)
> Also, feel free to correct me if I state something wrong.. I like to be
> correct in my understanding and if I don't have all the necessary
> information I would like to know what I am missing out on.  ;-)


Seconded. (iow, please apply the same to me.)

--EdorFaus

___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community

Re: Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-11-30 Thread Ole Tange 

On 11/30/06, Robert Michel <[EMAIL PROTECTED]> wrote:


A little bit the same with normal files. How do you think that a file
will make a diff with itself (the modification) before it is modificated
and save the modicated version + all diffs since last replication with
the server.


Wouldn't rsync solve that? Use the backup option and -e ssh.

/Ole

___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community

Re: Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-11-29 Thread Tim Newsom 

So here is a document talking about some of the things we have mentioned...

From the link to nokia...


http://research.nokia.com/tr/NRC-TR-2006-005.pdf

Interesting, they have thought of quite a bit of this already.

--Tim.

On 11/29/06, Jeff Andros <[EMAIL PROTECTED]> wrote:




On 11/29/06, Tim Newsom <[EMAIL PROTECTED]> wrote:
>
> Web services are XML data transfer.  The problem with xml is that it is
> wordy for data (size) but good for parsing.  What I mean by that is that its
> not the most efficient way to transfer data ( ok thats obvious) but its a
> defined format and easy to parse just about anywhere.. just slow.


ah,  but XML has a lot of good tools to offload  the pretty portions onto
other servers... and we can pull a Google and use one-letter tag names to
cut down on space.  to see what I mean check out here:
www.asu.edu/clubs/paddle  all the stuff
that makes the site pretty is in the xsl-t page... I'm pretty sure we can
cut down on the bloat by setting up our schema for minimum size

If you are keeping a copy of the current versions locally then diffing and
> sending only the diffs would be easy... but to my knowledge svn only keeps
> the diffs between versions at the repository. Someone who knowns please
> correct me.  If I understand what you are saying and based on my
> knowledge...
> You either keep an old version for diffing purposes and replace it with
> the current version when you do the commit.  All changes happen to the
> original not the "old copy"

 Now that you mention it, I'm not really that sure either... but I think
you're right.


> --Tim
>
> btw.. I tend to over explain things so if I don't go far enough please
> ask me (I am trying to cut back)  ;-)
> Also, feel free to correct me if I state something wrong.. I like to be
> correct in my understanding and if I don't have all the necessary
> information I would like to know what I am missing out on.  ;-)


amen, brother, amen

On 11/29/06, Jeff Andros <[EMAIL PROTECTED]> wrote:
>
> >
> > 
> > >
> > >
> > > Without thougths like that you will have an incredible GPRS traffic
> > > =
> > > costs.
> > >
> > > > ICS is really simple so
> > > > we could host that from the device as well. If Apache isn't small
> > > enough,
> > > > even stripped down, there are several server apps that are
> > > optimised for
> > > > this kind of environment
> >
> >
> > I don't do a lot of data on my phone at the moment mostly I use
> > bluetooth for this kind of thing, so I hadn't thought about costs... the
> > other thing we could try is an XML transfer of just the data, offloading the
> > formatting onto another server... the cool thing about this is we could run
> > both directly from the phone or from an intermediary depending on
> > preferences and the particular situation.
> >
> > When you use an intermediary server, that will handle the heavy
> > lifting on building the page and you could have a nice ajax-y interface,
> > direct from the phone you could remotely store an XSL-T stylesheet to give
> > you the frontend.  this minimizes the data that needs to go back and forth.
> > sending files back and forth I'd rather use something else, but in a pinch
> > we could use an svn client which does send only the file diffs back and
> > forth, plus storing the whole machine's drive on subversion gives us a nice
> > backup in case someone throws you into the pool with your phone in your
> > pocket (it's all fun and games until someone loses thier {email | phonebook
> > | files | blackmail photos of drunk friends})  you could also use this like
> > so:
> >
> > {user to phone} request update/commit cycle from phone to repository
> > {user on desktop} update local copy of phone filesystem
> > {user on desktop} make appropriate changes to files
> > {user on desktop} commit to repository
> > {user to phone} request update from repository
> >
> > where you are not on your phone, and are making commands from a
> > browser
> >
> >
> > Apache/whichever servers we're running handles the encryption, and now
> > you can get to the current version of your system through websvn from
> > anywhere
> >
> > --Jeff
> >
> > ___
> > OpenMoko community mailing list
> > community@lists.openmoko.org
> > http://lists.openmoko.org/cgi-bin/mailman/listinfo/community
> >
> >
> >
>
>
> --
> -- Tim
> ___
> OpenMoko community mailing list
> community@lists.openmoko.org
> http://lists.openmoko.org/cgi-bin/mailman/listinfo/community
>
>
>


--
--Jeff
___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community






--
-- Tim
___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community

Re: Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-11-29 Thread Jeff Andros 

On 11/29/06, Tim Newsom <[EMAIL PROTECTED]> wrote:


Web services are XML data transfer.  The problem with xml is that it is
wordy for data (size) but good for parsing.  What I mean by that is that its
not the most efficient way to transfer data ( ok thats obvious) but its a
defined format and easy to parse just about anywhere.. just slow.



ah,  but XML has a lot of good tools to offload  the pretty portions onto
other servers... and we can pull a Google and use one-letter tag names to
cut down on space.  to see what I mean check out here:
www.asu.edu/clubs/paddle  all the stuff
that makes the site pretty is in the xsl-t page... I'm pretty sure we can
cut down on the bloat by setting up our schema for minimum size

If you are keeping a copy of the current versions locally then diffing and

sending only the diffs would be easy... but to my knowledge svn only keeps
the diffs between versions at the repository. Someone who knowns please
correct me.  If I understand what you are saying and based on my
knowledge...
You either keep an old version for diffing purposes and replace it with
the current version when you do the commit.  All changes happen to the
original not the "old copy"


Now that you mention it, I'm not really that sure either... but I think
you're right.



--Tim

btw.. I tend to over explain things so if I don't go far enough please ask
me (I am trying to cut back)  ;-)
Also, feel free to correct me if I state something wrong.. I like to be
correct in my understanding and if I don't have all the necessary
information I would like to know what I am missing out on.  ;-)



amen, brother, amen

On 11/29/06, Jeff Andros <[EMAIL PROTECTED]> wrote:


>
> 
> >
> >
> > Without thougths like that you will have an incredible GPRS traffic =
> > costs.
> >
> > > ICS is really simple so
> > > we could host that from the device as well. If Apache isn't small
> > enough,
> > > even stripped down, there are several server apps that are optimised
> > for
> > > this kind of environment
>
>
> I don't do a lot of data on my phone at the moment mostly I use
> bluetooth for this kind of thing, so I hadn't thought about costs... the
> other thing we could try is an XML transfer of just the data, offloading the
> formatting onto another server... the cool thing about this is we could run
> both directly from the phone or from an intermediary depending on
> preferences and the particular situation.
>
> When you use an intermediary server, that will handle the heavy lifting
> on building the page and you could have a nice ajax-y interface, direct from
> the phone you could remotely store an XSL-T stylesheet to give you the
> frontend.  this minimizes the data that needs to go back and forth.  sending
> files back and forth I'd rather use something else, but in a pinch we could
> use an svn client which does send only the file diffs back and forth, plus
> storing the whole machine's drive on subversion gives us a nice backup in
> case someone throws you into the pool with your phone in your pocket (it's
> all fun and games until someone loses thier {email | phonebook | files |
> blackmail photos of drunk friends})  you could also use this like so:
>
> {user to phone} request update/commit cycle from phone to repository
> {user on desktop} update local copy of phone filesystem
> {user on desktop} make appropriate changes to files
> {user on desktop} commit to repository
> {user to phone} request update from repository
>
> where you are not on your phone, and are making commands from a browser
>
>
> Apache/whichever servers we're running handles the encryption, and now
> you can get to the current version of your system through websvn from
> anywhere
>
> --Jeff
>
> ___
> OpenMoko community mailing list
> community@lists.openmoko.org
> http://lists.openmoko.org/cgi-bin/mailman/listinfo/community
>
>
>


--
-- Tim
___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community






--
--Jeff
___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community

Re: Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-11-29 Thread Tim Newsom 

Web services are XML data transfer.  The problem with xml is that it is
wordy for data (size) but good for parsing.  What I mean by that is that its
not the most efficient way to transfer data ( ok thats obvious) but its a
defined format and easy to parse just about anywhere.. just slow.

If you are keeping a copy of the current versions locally then diffing and
sending only the diffs would be easy... but to my knowledge svn only keeps
the diffs between versions at the repository. Someone who knowns please
correct me.  If I understand what you are saying and based on my
knowledge...
You either keep an old version for diffing purposes and replace it with the
current version when you do the commit.  All changes happen to the original
not the "old copy"

OR

You would have to have the repository local and commit the changes locally
but that doesn't help with the storage part.  The only way to sync up would
be through the dock.

My understanding of svn is that it transmits the entire file and the svn
server does the diff and only stores the differences between the files at
the repository.  I could be wrong about that though.

If you don't keep a repository locally then one way or another you will have
to obtain a copy of the old version in order to just sync changes somehow..

Its an interesting thought.. does anyone know for sure if SVN sends back
only the changes and how it does that if it doesn't fetch the previous
version for comparison?

--Tim

btw.. I tend to over explain things so if I don't go far enough please ask
me (I am trying to cut back)  ;-)
Also, feel free to correct me if I state something wrong.. I like to be
correct in my understanding and if I don't have all the necessary
information I would like to know what I am missing out on.  ;-)



On 11/29/06, Jeff Andros <[EMAIL PROTECTED]> wrote:




>
>
> Without thougths like that you will have an incredible GPRS traffic =
> costs.
>
> > ICS is really simple so
> > we could host that from the device as well. If Apache isn't small
> enough,
> > even stripped down, there are several server apps that are optimised
> for
> > this kind of environment


I don't do a lot of data on my phone at the moment mostly I use bluetooth
for this kind of thing, so I hadn't thought about costs... the other thing
we could try is an XML transfer of just the data, offloading the formatting
onto another server... the cool thing about this is we could run both
directly from the phone or from an intermediary depending on preferences and
the particular situation.

When you use an intermediary server, that will handle the heavy lifting on
building the page and you could have a nice ajax-y interface, direct from
the phone you could remotely store an XSL-T stylesheet to give you the
frontend.  this minimizes the data that needs to go back and forth.  sending
files back and forth I'd rather use something else, but in a pinch we could
use an svn client which does send only the file diffs back and forth, plus
storing the whole machine's drive on subversion gives us a nice backup in
case someone throws you into the pool with your phone in your pocket (it's
all fun and games until someone loses thier {email | phonebook | files |
blackmail photos of drunk friends})  you could also use this like so:

{user to phone} request update/commit cycle from phone to repository
{user on desktop} update local copy of phone filesystem
{user on desktop} make appropriate changes to files
{user on desktop} commit to repository
{user to phone} request update from repository

where you are not on your phone, and are making commands from a browser


Apache/whichever servers we're running handles the encryption, and now you
can get to the current version of your system through websvn from anywhere

--Jeff

___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community






--
-- Tim
___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community

Fwd: Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-11-29 Thread Jeff Andros 





Without thougths like that you will have an incredible GPRS traffic =
costs.

> ICS is really simple so
> we could host that from the device as well. If Apache isn't small
enough,
> even stripped down, there are several server apps that are optimised for
> this kind of environment



I don't do a lot of data on my phone at the moment mostly I use bluetooth
for this kind of thing, so I hadn't thought about costs... the other thing
we could try is an XML transfer of just the data, offloading the formatting
onto another server... the cool thing about this is we could run both
directly from the phone or from an intermediary depending on preferences and
the particular situation.

When you use an intermediary server, that will handle the heavy lifting on
building the page and you could have a nice ajax-y interface, direct from
the phone you could remotely store an XSL-T stylesheet to give you the
frontend.  this minimizes the data that needs to go back and forth.  sending
files back and forth I'd rather use something else, but in a pinch we could
use an svn client which does send only the file diffs back and forth, plus
storing the whole machine's drive on subversion gives us a nice backup in
case someone throws you into the pool with your phone in your pocket (it's
all fun and games until someone loses thier {email | phonebook | files |
blackmail photos of drunk friends})  you could also use this like so:

{user to phone} request update/commit cycle from phone to repository
{user on desktop} update local copy of phone filesystem
{user on desktop} make appropriate changes to files
{user on desktop} commit to repository
{user to phone} request update from repository

where you are not on your phone, and are making commands from a browser


Apache/whichever servers we're running handles the encryption, and now you
can get to the current version of your system through websvn from anywhere

--Jeff
___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community

Please think about incrementel sync Re: Another idea for an application for the Neo: Instant sync to web page duplicating info on phone

2006-11-29 Thread Robert Michel 
Salve Jeff,*!

When it will be only adresses, todo, memo - syncML is build to do
this. But please via a secure network - Siemens offer this unencrypted.
*help*help*help*help* 

On Wed, 29 Nov 2006, Jeff Andros wrote:
> >> for uploading sets of data they could be implelented in almost any
> >lauguage
> >> and used natively like normal objects.
> >> Perl, php, python, java, c# can all do that and it means the backend
> >does not
> >> have to be rewritten every time someone wants to use it in a language.
> >>
> >> Also, you could then provide a bundle of code for the server backend
> >that
> >> handles the web service requests which anyone could run and point their
> >own
> >> phone at their server.
> 
> 
> It may be easier if the phone has an accessable IP address... I'm not quite
> sure how GPRS works, some one who knows let me know... 
It has an IP and this makes me worry that the area of IPs for GPRS are
known and someone would send you anwanted packets "just for fun" to
encrease your mobil bill. If I had a chance I would take an option that
the operater will protect my mobil will a firewall and only two
connections to my two server would be allowed.
(I have big doubts that this would work with beeing abroad with a
GPRS roaming partner some day).
So everybody should track his GPRS costs automaticaly.

> but we could set up a
> embedded web server on the openmoko device itself.  

This will be an option, but will encrease the GPRS traffic = costs.

For having everyfile *save* on the server 
http://www.drbd.org/

But to lower the traffic costs the transmission of data should be
reduces as much as possible and wait until it is realy needed,
or a cheap connection (USB, data call...) would allow to sync.


Just let the server ask if since he last asked the Neo some
configuration has changed and transmit it only incremental.

A little bit the same with normal files. How do you think that a file
will make a diff with itself (the modification) before it is modificated
and save the modicated version + all diffs since last replication with
the server.
The server could ask the Neo for seperate direction pathes (tree parts)
for informations about changes. A list of the files that have been changed
inbetween (on inode level?) will be transmitted to the server.
When you open such a file on the server, the server will ask for the
diff. Transmission bzip2 compressed and via VPN

Without thougths like that you will have an incredible GPRS traffic =
costs.

> ICS is really simple so
> we could host that from the device as well. If Apache isn't small enough,
> even stripped down, there are several server apps that are optimised for
> this kind of environment

A webserver will change the layout of the neo... 
I like to propose the alternative Frontend, to have an OpenMoko/NSLU2 at home
with syncronisation and to use NX to forward the X to your webbrowser...

Greetings,
rob

___
OpenMoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/cgi-bin/mailman/listinfo/community