Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 20.09.2017 17:39, Even Rouault wrote: On mercredi 20 septembre 2017 17:32:27 CEST Sandro Mani wrote: > On 19.09.2017 19:15, Even Rouault wrote: > > On mardi 19 septembre 2017 18:30:50 CEST Sandro Mani wrote: > > > On 11.07.2017 14:31, Even Rouault wrote: > > > > We could potentially

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On mercredi 20 septembre 2017 17:32:27 CEST Sandro Mani wrote: > On 19.09.2017 19:15, Even Rouault wrote: > > On mardi 19 septembre 2017 18:30:50 CEST Sandro Mani wrote: > > > On 11.07.2017 14:31, Even Rouault wrote: > > > > We could potentially imagine to defer the repacking at > > > > > > > >

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 19.09.2017 19:15, Even Rouault wrote: On mardi 19 septembre 2017 18:30:50 CEST Sandro Mani wrote: > On 11.07.2017 14:31, Even Rouault wrote: > > We could potentially imagine to defer the repacking at > > leaveUpdateMode() time, actually in the close() method which is called > > when the

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On mardi 19 septembre 2017 18:30:50 CEST Sandro Mani wrote: > On 11.07.2017 14:31, Even Rouault wrote: > > We could potentially imagine to defer the repacking at > > leaveUpdateMode() time, actually in the close() method which is called > > when the reference counter of the update mode drops to

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 11.07.2017 14:31, Even Rouault wrote: We could potentially imagine to defer the repacking at leaveUpdateMode() time, actually in the close() method which is called when the reference counter of the update mode drops to zero (*), instead of doing it at the end of each operation. To come

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 14.07.2017 00:01, Martin Dobias wrote: Hi Sandro On Wed, Jul 12, 2017 at 11:15 AM, Sandro Mani wrote: The problem is that if one of the changes to be committed is refused by the provider, you're stuck not being able to commit the entire set of changes. And, since we

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi Sandro On Wed, Jul 12, 2017 at 11:15 AM, Sandro Mani wrote: > > The problem is that if one of the changes to be committed is refused by the > provider, you're stuck not being able to commit the entire set of changes. > And, since we are dealing with invalid geometries,

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 12.07.2017 11:41, Luigi Pirelli wrote: why not add the feature to a more detailed notification of what is wrong during commitChanges? Like point out a more detailed report of the failing feature/attribute/geometry? it would just a new signal and EditBuffer cache. Uhm but that would still

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

why not add the feature to a more detailed notification of what is wrong during commitChanges? Like point out a more detailed report of the failing feature/attribute/geometry? it would just a new signal and EditBuffer cache. just my 2c Luigi Pirelli

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 12.07.2017 01:06, Nyall Dawson wrote: Thanks for the insight, I'll look at that approach. Sandro I might have missed this in the earlier conversation, but why not explore the option suggest of using the layer's edit buffer? I believe this should automatically handle all the reassignment

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

> > Thanks for the insight, I'll look at that approach. > > Sandro I might have missed this in the earlier conversation, but why not explore the option suggest of using the layer's edit buffer? I believe this should automatically handle all the reassignment without any changes needed (except

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 11.07.2017 14:31, Even Rouault wrote: On mardi 11 juillet 2017 14:00:28 CEST Sandro Mani wrote: > On 10.07.2017 13:58, Even Rouault wrote: > > > I did some playing around and came up with some FeatureId mapping code > > > > > > which is pretty space efficient (basically requiring just

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On mardi 11 juillet 2017 14:00:28 CEST Sandro Mani wrote: > On 10.07.2017 13:58, Even Rouault wrote: > > > I did some playing around and came up with some FeatureId mapping code > > > > > > which is pretty space efficient (basically requiring just > > > > > > 3*nDeletedFeatures entries), see

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 10.07.2017 13:58, Even Rouault wrote: > I did some playing around and came up with some FeatureId mapping code > which is pretty space efficient (basically requiring just > 3*nDeletedFeatures entries), see below. > > The code however assumes that the gaps in the fid-sequence are filled

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On lundi 10 juillet 2017 15:01:56 CEST Régis Haubourg wrote: > Hi, > > This should clearly be restricted to Shapefiles. Other drivers don't have > > > this compaction logic and will let happily holes in the feature ids. > > What about Mapinfo TAB, which relies on DBF like shapefiles, and dbf

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi, This should clearly be restricted to Shapefiles. Other drivers don't have > this compaction logic and will let happily holes in the feature ids. What about Mapinfo TAB, which relies on DBF like shapefiles, and dbf files? Just asking, I didn't dig into that in depth. Régis

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

> I did some playing around and came up with some FeatureId mapping code > which is pretty space efficient (basically requiring just > 3*nDeletedFeatures entries), see below. > > The code however assumes that the gaps in the fid-sequence are filled by > decreasing the fids after the gap, which

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 06.07.2017 11:00, Sandro Mani wrote: On 06.07.2017 10:57, Even Rouault wrote: On jeudi 6 juillet 2017 10:45:21 CEST Sandro Mani wrote: > On 05.07.2017 14:56, Sandro Mani wrote: > >> That could potentially be done using the current OGR API. Basically > >> in pseudo code, to execute

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 06.07.2017 11:38, Martin Dobias wrote: Hi Sandro On Thu, Jul 6, 2017 at 11:00 AM, Sandro Mani wrote: It would be something I would call explicitly while the geometry checker is active, I think it is safe to assume that users don't expect that they can operate on the

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi Sandro On Thu, Jul 6, 2017 at 11:00 AM, Sandro Mani wrote: > > It would be something I would call explicitly while the geometry checker is > active, I think it is safe to assume that users don't expect that they can > operate on the shapefile while the geometry checker

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 07/06/2017 11:00 AM, Sandro Mani wrote: >> >> My feeling is that QgsOgrDataProvider is already complicated enough >> with its existing tricks. I'm not sure this temporary freeze repacking >> is a right move (how would you decide when you do it ? and I'm pretty >> sure users will not get it, or

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 06.07.2017 10:57, Even Rouault wrote: On jeudi 6 juillet 2017 10:45:21 CEST Sandro Mani wrote: > On 05.07.2017 14:56, Sandro Mani wrote: > >> That could potentially be done using the current OGR API. Basically > >> in pseudo code, to execute before executing REPACK > >> > >>

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On jeudi 6 juillet 2017 10:45:21 CEST Sandro Mani wrote: > On 05.07.2017 14:56, Sandro Mani wrote: > >> That could potentially be done using the current OGR API. Basically > >> in pseudo code, to execute before executing REPACK > >> > >> OGR_L_ResetReading(layer) > >> > >> char** ignored_fields

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 05.07.2017 14:56, Sandro Mani wrote: That could potentially be done using the current OGR API. Basically in pseudo code, to execute before executing REPACK OGR_L_ResetReading(layer) char** ignored_fields = CSLAddString(NULL, "OGR_GEOMETRY" ); OGR_L_SetIgnoredFields( layer,

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 05.07.2017 12:18, Even Rouault wrote: On mercredi 5 juillet 2017 12:04:04 CEST Sandro Mani wrote: > On 05.07.2017 11:30, Even Rouault wrote: > > My feeling is that, for good or worse, they are definitely people > > wanting to have QGIS and "put here the name of your favorite > >

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi Even, > On mercredi 5 juillet 2017 10:51:02 CEST Sandro Mani wrote: >> On 05.07.2017 08:51, Giovanni Manghi wrote: >> > as far as I remember this is not entirely true: if you go on redmine >> > and look for the (sometimes very long) tickets about this very big >> > issue at that time (now

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On mercredi 5 juillet 2017 12:04:04 CEST Sandro Mani wrote: > On 05.07.2017 11:30, Even Rouault wrote: > > My feeling is that, for good or worse, they are definitely people > > wanting to have QGIS and "put here the name of your favorite > > proprietary GIS software" running at the same time, and

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 05.07.2017 11:30, Even Rouault wrote: My feeling is that, for good or worse, they are definitely people wanting to have QGIS and "put here the name of your favorite proprietary GIS software" running at the same time, and possibly on the same datasets. This is apparently important in

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 7/5/17 1:31 AM, Nathan Woodrow wrote: > > I'd say we definitely do in the processing case. It'd be a significant > shortcoming to have to manually unload layers from projects before you > can run analysis on them. > > > Not sure about totally external applications though > > My comments

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On mercredi 5 juillet 2017 10:51:02 CEST Sandro Mani wrote: > On 05.07.2017 08:51, Giovanni Manghi wrote: > > as far as I remember this is not entirely true: if you go on redmine > > and look for the (sometimes very long) tickets about this very big > > issue at that time (now thankfully), is

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 05.07.2017 08:51, Giovanni Manghi wrote: as far as I remember this is not entirely true: if you go on redmine and look for the (sometimes very long) tickets about this very big issue at that time (now thankfully), is wasn't necessary to have the shapefile open in edit mode two applications

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi, >> > A common workflow that leads to this situation is when working on > shapefiles and then using them in a processing algorithm that involves an > external process. The processing situation could be avoided (detect if > potentially dangerous operations like a delete have been performed

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

> I'd say we definitely do in the processing case. It'd be a significant shortcoming to have to manually unload layers from projects before you can run analysis on them. > Not sure about totally external applications though My comments only relate to external applications using files we have

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 5 July 2017 at 09:10, Nathan Woodrow wrote: >> > A common workflow that leads to this situation is when working on shapefiles > and then using them in a processing algorithm that involves an external > process. The processing situation could be avoided (detect if

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

> A common workflow that leads to this situation is when working on shapefiles and then using them in a processing algorithm that involves an external process. The processing situation could be avoided (detect if potentially dangerous operations like a delete have been performed since the last

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 7/4/17 6:46 PM, Even Rouault wrote: > Another solution would be to defere the compaction only at layer > closing (that's actually the new behaviour of the OGR Shapefile driver > in recent GDAL versions to force a repack at closing). I'm not sure > why QGIS currently repacks it every time ? >

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

On 04.07.2017 18:46, Even Rouault wrote: Another solution would be to defere the compaction only at layer closing (that's actually the new behaviour of the OGR Shapefile driver in recent GDAL versions to force a repack at closing). I'm not sure why QGIS currently repacks it every time ?

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

> I don't know if it would be possible for OGR - in case of a repack() - > to create an internal map of new ids to original ids (or if it would be > possible for people to stop using shp :P). But I really don't know the > internals of ogr (or people) good enough to comment on the feasibility > of

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi Sandro, The commit that introduced this behavior was done around 2 years back. The reason for this was mainly interoperability with other applications as Even mentioned. https://github.com/qgis/QGIS/commit/7d7cdcd376c0d3fa60af1403a91e1e611b210174 Most relevant discussion around the issue can

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi Even Thanks for the reply. Yes I am indeed testing with shapefiles. Not sure why I never observed this behaviour previously, but indeed I have up to now never observed this myself, nor have received bug reports concerned with this kind breakage - and this is a pretty hard to miss issue

Re: [QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Sandro, I guess this might depend on the actual OGR driver used. Somehow I assume you use shapefiles here ? This is probably linked to shapefiles being repacked after the end of various operations such as addFeatures(), changeGeometryValues(), deleteFeatures(), so as to avoid issues with

[QGIS-Developer] QGIS/OGR: FeatureIds reassigned on write to data provider?

Hi I'm doing work on the geometry checker in QGIS master and noticed that, when deleting features from an ogr vector layer, FeatureIds appear to be reassigned to fill up the gap in the ids. This is a pretty serious blocker for the geometry checker since geometry checker errors are identified