Re: [OSGeo-Discuss] Re: "git" like for geodata management [SEC=UNCLASSIFIED]

[Raj Singh]

Arnulf, that's exactly what the OGC Geosynchronization spec describes! Servers 
advertise their changes in a GeoRSS feed -- where the  element 
describes the change in WFS-T XML. It's nicely loosely coupled, because as you 
eloquently state, clients can decide what they want to do with the "suggested" 
modifications, accept them all, just use those in a certain area, or those from 
a "trusted" agency, etc.

---
Raj



On Sep 24, at 4:08 AM, Seven (aka Arnulf) wrote:

> This might be too far off the original idea... But: It might be interesting 
> to look into a more disconnected way of "managing" geospatial data with the 
> Web. As in having "servers" advertize latest changes in geometry and 
> attributes using GeoRSS and "clients" picking these up and integrating them 
> (potentially including processing, changes) into their "cache" or "copy" of 
> the data. Which might automagically turn into a "server" itself advertizing 
> changes following the same pattern. This idea grew on me over the past years 
> after seeing Ward Cunningham give his presentation at Wikimania 2005 in 
> Fankfurt on bits of information floating around the Web [1]. Maybe now the 
> time is ready to implement some of these thoughts in our domain.
> 
> I am currently seeding these ideas to the European National Mapping Agencies 
> through the ESDIN project. Yes I know, it sounds a bit remote but hey, it is 
> a nice vision... :-)
> 
> Have fun,
> Arnulf.
> 
> [1] Slides: http://c2.com/doc/wikimania/
> Screener: http://www.archive.org/details/WIKIMANIA_TAPE_019_SCREENER.mov
> Low res video runs 30 minutes
> 
> 
> On 09/24/2010 01:40 AM, Bruce Bannerman wrote:
>> Ragi,
>> 
>> I agree. I think that we have a way to go yet to have something
>> comparable to the ArcSDE / ArcGIS Multi-versioning and version conflict
>> detection functionality.
>> 
>> The advantage that the ArcSDE solution has is that edits are made
>> directly within the database. This works well within an Enterprise
>> environment as described by Fabio earlier in this thread.
>> 
>> I may be wrong, but I think that git works on files, but I haven’t used
>> it myself. Can git detect changes to the spatial representation of a
>> feature within a binary file?
>> 
>> Also, speaking as someone who implemented an ArcSDE/ArcGIS
>> Multi-versioned edit scenario several years ago, the ESRI solution is
>> far from perfect. It imposes very strict environment management on the
>> system managers, e.g.:
>> 
>>* All versions of the software used (client and server) must be at
>>  precisely the same version, service pack and patch;
>>* The environment can only use software that implements the
>>  ArcObjects environment (from experience, this rules out the use of
>>  the ArcSDE Java and C API’s);
>>* Editors must be well trained and knowledgeable in using both
>>  ArcGIS and Multi-versioned processes;
>>* The Organisation needs to think through their maintenance
>>  processes to get best advantage of the functionality; and
>>* It doesn’t remove the need for data maintenance people to talk to
>>  each other about work that is going on, as the software cannot
>>  resolve all conflicts. For example, if two editors make changes to
>>  the spatial representation of a feature, which one is correct? The
>>  software will detect the conflict, but the editors (or their
>>  managers) will need to resolve the issue of which version of the
>>  feature’s spatial representation is correct.
>> 
>> 
>> 
>> Bruce
>> 
>> 
>> On 24/09/10 4:05 AM, "Ragi Burhum"  wrote:
>> 
>>Hi Noli,
>> 
>>thanks for the link. That is definitely a step in the right
>>direction, but it is hardly comparable to git ArcSDE versioning at that.
>> 
>>The article and sample code you describe above generates hashes for
>>all rows and tables in the db and compares them to the target db. So
>>1 million rows in a db, regardless if the two dbs are identical,
>>would cause 1 million hashes to go over the wire. Every single time
>>you ask to sync you pay the price.
>> 
>>Git and ArcSDE keep track of changesets, and when it is time to
>>synchronize, they exchange that changeset and apply it. One insert?
>>That is all that needs to be sent.
>> 
>>Another issue is that there is nothing about conflict resolution
>>there (what happens when you delete one row in one db and modify it
>>in another one?). There is also the problem of allowing multiple
>>versions of the data in the same db (Like having multiple heads).
>> 
>>Regardless, thank you for the link,
>> 
>>- Ragi
>> 
>> 
>>Date: Thu, 23 Sep 2010 13:22:17 +1000
>>From: Noli Sicad 
>>Subject: Re: [OSGeo-Discuss] Re: "git" like for geodata management
>>To: OSGeo Discussions 
>>Message-ID:
>>
>>Content-Type: text/plain; charset=ISO-8859-1
>> 
>>PostgreSQL Synchronization Tool --- psync [1]
>> 
>>

Re: [OSGeo-Discuss] Re: "git" like for geodata management [SEC=UNCLASSIFIED]

[Peter Batty]

Hi,

Yes Smallworld did a lot of innovation in the area of version management and
long transactions some 20 years ago, I was quite involved in all that. The
technical paper you mention is at archive.org, a great resource for finding
web pages that have disappeared!
http://web.archive.org/web/20080519210011/http://cfis.savagexi.com/pages/technical_paper_4

This paper, which I was a co-author of, talks about long transactions as
well as query architectures:
http://web.archive.org/web/20070828091638/cfis.savagexi.com/pages/technical_paper_8

The whole list of Smallworld technical papers is at
http://web.archive.org/web/20080519210016/cfis.savagexi.com/pages/technical_papers

ESRI, Oracle and others adopted a number of the version management ideas
that Smallworld pioneered, but (IMHO) still don't have nearly as robust an
implementation as Smallworld does. If anyone has questions on this area, I'm
happy to try to answer them. If you're not familiar with Smallworld, there's
a bit more background on my blog at http://bit.ly/cDUid1

Cheers,
Peter.

On Fri, Sep 24, 2010 at 3:55 AM, STEPHEN STANTON wrote:

>
> I'm a bit surprised no-one mentions Smallworld. Weren't they at the
> forefront of version management almost 20 years ago?
>
> The following link to an interesting technical paper isn't currently
> working, but maybe it'll come back later:
> http://cfis.savagexi.com/pages/technical_paper_4
>
> Steve
>
> > On 09/24/2010 01:40 AM, Bruce Bannerman wrote:
> > > Ragi,
> > >
> > > I agree. I think that we have a way to go yet to have
> > something
> > > comparable to the ArcSDE / ArcGIS Multi-versioning and
> > version conflict
> > > detection functionality.
> > >
> > > The advantage that the ArcSDE solution has is that
> > edits are made
> > > directly within the database. This works well within
> > an Enterprise
> > > environment as described by Fabio earlier in this
> > thread.
> > >
> > > I may be wrong, but I think that git works on files,
> > but I haven’t used
> > > it myself. Can git detect changes to the spatial
> > representation of a
> > > feature within a binary file?
> > >
> > > Also, speaking as someone who implemented an
> > ArcSDE/ArcGIS
> > > Multi-versioned edit scenario several years ago, the
> > ESRI solution is
> > > far from perfect. It imposes very strict environment
> > management on the
> > > system managers, e.g.:
> > >
> > > * All versions of the software
> > used (client and server) must be at
> > >   precisely the same
> > version, service pack and patch;
> > > * The environment can only use
> > software that implements the
> > >   ArcObjects environment
> > (from experience, this rules out the use of
> > >   the ArcSDE Java and C
> > API’s);
> > > * Editors must be well trained
> > and knowledgeable in using both
> > >   ArcGIS and
> > Multi-versioned processes;
> > > * The Organisation needs to
> > think through their maintenance
> > >   processes to get best
> > advantage of the functionality; and
> > > * It doesn’t remove the need
> > for data maintenance people to talk to
> > >   each other about work
> > that is going on, as the software cannot
> > >   resolve all conflicts.
> > For example, if two editors make changes to
> > >   the spatial
> > representation of a feature, which one is correct? The
> > >   software will detect
> > the conflict, but the editors (or their
> > >   managers) will need to
> > resolve the issue of which version of the
> > >   feature’s spatial
> > representation is correct.
> > >
> > >
> > >
> > > Bruce
>
>
> ___
> Discuss mailing list
> Discuss@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: "git" like for geodata management [SEC=UNCLASSIFIED]

[STEPHEN STANTON]

I'm a bit surprised no-one mentions Smallworld. Weren't they at the forefront 
of version management almost 20 years ago?

The following link to an interesting technical paper isn't currently working, 
but maybe it'll come back later: 
http://cfis.savagexi.com/pages/technical_paper_4

Steve

> On 09/24/2010 01:40 AM, Bruce Bannerman wrote:
> > Ragi,
> >
> > I agree. I think that we have a way to go yet to have
> something
> > comparable to the ArcSDE / ArcGIS Multi-versioning and
> version conflict
> > detection functionality.
> >
> > The advantage that the ArcSDE solution has is that
> edits are made
> > directly within the database. This works well within
> an Enterprise
> > environment as described by Fabio earlier in this
> thread.
> >
> > I may be wrong, but I think that git works on files,
> but I haven’t used
> > it myself. Can git detect changes to the spatial
> representation of a
> > feature within a binary file?
> >
> > Also, speaking as someone who implemented an
> ArcSDE/ArcGIS
> > Multi-versioned edit scenario several years ago, the
> ESRI solution is
> > far from perfect. It imposes very strict environment
> management on the
> > system managers, e.g.:
> >
> >     * All versions of the software
> used (client and server) must be at
> >       precisely the same
> version, service pack and patch;
> >     * The environment can only use
> software that implements the
> >       ArcObjects environment
> (from experience, this rules out the use of
> >       the ArcSDE Java and C
> API’s);
> >     * Editors must be well trained
> and knowledgeable in using both
> >       ArcGIS and
> Multi-versioned processes;
> >     * The Organisation needs to
> think through their maintenance
> >       processes to get best
> advantage of the functionality; and
> >     * It doesn’t remove the need
> for data maintenance people to talk to
> >       each other about work
> that is going on, as the software cannot
> >       resolve all conflicts.
> For example, if two editors make changes to
> >       the spatial
> representation of a feature, which one is correct? The
> >       software will detect
> the conflict, but the editors (or their
> >       managers) will need to
> resolve the issue of which version of the
> >       feature’s spatial
> representation is correct.
> >
> >
> >
> > Bruce


___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: "git" like for geodata management [SEC=UNCLASSIFIED]

[Seven (aka Arnulf)]

This might be too far off the original idea... But: It might be 
interesting to look into a more disconnected way of "managing" 
geospatial data with the Web. As in having "servers" advertize latest 
changes in geometry and attributes using GeoRSS and "clients" picking 
these up and integrating them (potentially including processing, 
changes) into their "cache" or "copy" of the data. Which might 
automagically turn into a "server" itself advertizing changes following 
the same pattern. This idea grew on me over the past years after seeing 
Ward Cunningham give his presentation at Wikimania 2005 in Fankfurt on 
bits of information floating around the Web [1]. Maybe now the time is 
ready to implement some of these thoughts in our domain.


I am currently seeding these ideas to the European National Mapping 
Agencies through the ESDIN project. Yes I know, it sounds a bit remote 
but hey, it is a nice vision... :-)


Have fun,
Arnulf.

[1] Slides: http://c2.com/doc/wikimania/
Screener: http://www.archive.org/details/WIKIMANIA_TAPE_019_SCREENER.mov
Low res video runs 30 minutes


On 09/24/2010 01:40 AM, Bruce Bannerman wrote:

Ragi,

I agree. I think that we have a way to go yet to have something
comparable to the ArcSDE / ArcGIS Multi-versioning and version conflict
detection functionality.

The advantage that the ArcSDE solution has is that edits are made
directly within the database. This works well within an Enterprise
environment as described by Fabio earlier in this thread.

I may be wrong, but I think that git works on files, but I haven’t used
it myself. Can git detect changes to the spatial representation of a
feature within a binary file?

Also, speaking as someone who implemented an ArcSDE/ArcGIS
Multi-versioned edit scenario several years ago, the ESRI solution is
far from perfect. It imposes very strict environment management on the
system managers, e.g.:

* All versions of the software used (client and server) must be at
  precisely the same version, service pack and patch;
* The environment can only use software that implements the
  ArcObjects environment (from experience, this rules out the use of
  the ArcSDE Java and C API’s);
* Editors must be well trained and knowledgeable in using both
  ArcGIS and Multi-versioned processes;
* The Organisation needs to think through their maintenance
  processes to get best advantage of the functionality; and
* It doesn’t remove the need for data maintenance people to talk to
  each other about work that is going on, as the software cannot
  resolve all conflicts. For example, if two editors make changes to
  the spatial representation of a feature, which one is correct? The
  software will detect the conflict, but the editors (or their
  managers) will need to resolve the issue of which version of the
  feature’s spatial representation is correct.



Bruce


On 24/09/10 4:05 AM, "Ragi Burhum"  wrote:

Hi Noli,

thanks for the link. That is definitely a step in the right
direction, but it is hardly comparable to git ArcSDE versioning at that.

The article and sample code you describe above generates hashes for
all rows and tables in the db and compares them to the target db. So
1 million rows in a db, regardless if the two dbs are identical,
would cause 1 million hashes to go over the wire. Every single time
you ask to sync you pay the price.

Git and ArcSDE keep track of changesets, and when it is time to
synchronize, they exchange that changeset and apply it. One insert?
That is all that needs to be sent.

Another issue is that there is nothing about conflict resolution
there (what happens when you delete one row in one db and modify it
in another one?). There is also the problem of allowing multiple
versions of the data in the same db (Like having multiple heads).

Regardless, thank you for the link,

- Ragi


Date: Thu, 23 Sep 2010 13:22:17 +1000
From: Noli Sicad 
Subject: Re: [OSGeo-Discuss] Re: "git" like for geodata management
To: OSGeo Discussions 
Message-ID:

Content-Type: text/plain; charset=ISO-8859-1

PostgreSQL Synchronization Tool --- psync [1]

" The article introduces a method of synchronizing two PostgreSQL
databases. Although, this seems to be an easy task, no product
(slony,
londiste, ...) really satisfied the needs within the
maps.bremen.de  
project. Either they have special prerequsits that didn't apply for
our problem or they didn't support synchronizing of large objects.

Large objects are used to store tiles of a street/aerial map within
PostgreSQL. My GIS-server queries the database and gets the
tiles out.
By using this construction we are getting a flexible infrastructure
for updating and mainta

Re: [OSGeo-Discuss] Re: "git" like for geodata management [SEC=UNCLASSIFIED]

[Bruce Bannerman]

Ragi,

I agree. I think that we have a way to go yet to have something comparable to 
the ArcSDE / ArcGIS Multi-versioning and version conflict detection 
functionality.

The advantage that the ArcSDE solution has is that edits are made directly 
within the database. This works well within an Enterprise environment as 
described by Fabio earlier in this thread.

I may be wrong, but I think that git works on files, but I haven't used it 
myself. Can git detect changes to the spatial representation of a feature 
within a binary file?

Also, speaking as someone who implemented an ArcSDE/ArcGIS Multi-versioned edit 
scenario several years ago, the ESRI solution is far from perfect. It imposes 
very strict environment management on the system managers, e.g.:


 *   All versions of the software used (client and server) must be at precisely 
the same version, service pack and patch;
 *   The environment can only use software that implements the ArcObjects 
environment (from experience, this rules out the use of the ArcSDE Java and C 
API's);
 *   Editors must be well trained and knowledgeable in using both ArcGIS and 
Multi-versioned processes;
 *   The Organisation needs to think through their maintenance processes to get 
best advantage of the functionality; and
 *   It doesn't remove the need for data maintenance people to talk to each 
other about work that is going on, as the software cannot resolve all 
conflicts. For example, if two editors make changes to the spatial 
representation of a feature, which one is correct? The software will detect the 
conflict, but the editors (or their managers) will need to resolve the issue of 
which version of the feature's spatial representation is correct.


Bruce


On 24/09/10 4:05 AM, "Ragi Burhum"  wrote:

Hi Noli,

thanks for the link. That is definitely a step in the right direction, but it 
is hardly comparable to git ArcSDE versioning at that.

The article and sample code you describe above generates hashes for all rows 
and tables in the db and compares them to the target db. So 1 million rows in a 
db, regardless if the two dbs are identical, would cause 1 million hashes to go 
over the wire. Every single time you ask to sync you pay the price.

Git and ArcSDE keep track of changesets, and when it is time to synchronize, 
they exchange that changeset and apply it. One insert? That is all that needs 
to be sent.

Another issue is that there is nothing about conflict resolution there (what 
happens when you delete one row in one db and modify it in another one?). There 
is also the problem of allowing multiple versions of the data in the same db 
(Like having multiple heads).

Regardless, thank you for the link,

- Ragi


Date: Thu, 23 Sep 2010 13:22:17 +1000
From: Noli Sicad 
Subject: Re: [OSGeo-Discuss] Re: "git" like for geodata management
To: OSGeo Discussions 
Message-ID:

Content-Type: text/plain; charset=ISO-8859-1

PostgreSQL Synchronization Tool  --- psync [1]

" The article introduces a method of synchronizing two PostgreSQL
databases. Although, this seems to be an easy task, no product (slony,
londiste, ...) really satisfied the needs within the maps.bremen.de 
  
project. Either they have special prerequsits that didn't apply for
our problem or they didn't support synchronizing of large objects.

Large objects are used to store tiles of a street/aerial map within
PostgreSQL. My GIS-server queries the database and gets the tiles out.
By using this construction we are getting a flexible infrastructure
for updating and maintaining different versions of the maps.

Everything was working fine until the service needs to be spread over
three servers. How can we easily synchronize the databases? I really
found no really working solution that is clean and easy to use.  "

[1]http://www.codeproject.com/KB/database/psync.aspx 



Noli

On 9/23/10, Ragi Burhum  wrote:
Are you looking for an alternative to (1)ESRI's versioning, (2)ESRI's
disconnected editing, or a mix of both (3)git like? the scenario that you
described first was more like (2), but this one fits (1).

I would love to see something like (3), but truth of the matter, AFAIK,
there is nothing like that implemented for geo (yet).

On Sep 22, 2010, at 9:00 AM, discuss-requ...@lists.osgeo.org wrote:

On Wed, 2010-09-22 at 12:10 +0800, maning sambale wrote:
Any real world cases for this?

Imagine the following scenario:

* 50 ~ 70 digitizers
* 5 QA
* 1 Manager

Each QA has 10 digitizers assigned. After all the data is validated, the
manager merges it and generates the geodb.

All users work against the same DB, most of them linked. This causes
disconnections, duplicated data, and lots of random errors.

Also, they can't be forced to work on different DB's because they are
all working on the same project, at the same time.

This is the real scenario of GISWorking (http://www.gisworking.com/), a
company we are w