Re: [gdal-dev] ogr2ogr with append fails to convert truncated fields
Added a ticket with the proposed implementation patch. Let me know how you like it. http://trac.osgeo.org/gdal/ticket/5021 Best regards, Tamas 2013/3/2 Even Rouault even.roua...@mines-paris.org Le vendredi 01 mars 2013 20:57:43, Tamas Szekeres a écrit : 2013/3/1 Even Rouault even.roua...@mines-paris.org There's even a 4th option that doesn't require any code change. Create a OGR VRT file that renames the source fields to the truncated shapefile field names (or the reverse : creates a OGR VRT that renames the truncated shapefile field names to the original field names, since OGR VRT now supports since a few versions CreateFeature()) This may probably work, but doesn't seem to be very user friendly. Actually I would prefer to include at least a flag to indicate that ogr2ogr should provide mapping the fields in the same order (don't lookup field index by name) for example: ogr2ogr -append -f ESRI Shapefile -nofieldreorder destination.shp [source] or with the previous fieldmap approach ogr2ogr -append -f ESRI Shapefile -fieldmap default destination.shp [source] I'm fine with -fieldmap. It will be just another advanced option (I'm culprit for a bunch of other options...). Perhaps same_order (or identity to be mathematically purist) instead of default, since it's not actually the default behaviour (the default behaviour is to match the field names to build the field map). Some care should be done to check that the specified map is of the right size (number of fields of the source layer), or if same_order/identity is specified that the number of fields in the source and target layers are identical. As ogr2ogr can accept several source layers, perhaps a word of caution in the doc of the option to mention that the specified map applies to all source layers, so if they don't have the same field definition, the option might not be relevant in that case. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] ogr2ogr with append fails to convert truncated fields
Le lundi 25 février 2013 19:53:22, Tamas Szekeres a écrit : So we seem to have 3 divergent options to provide a work around: 1. Expose fieldmap setting to let the user decide the order of the fields to be copied. 2. Modify GetFieldIndex to provide the name based lookup by the truncated name (for the shape driver specifically) 3. Provide an option to instruct ogr2ogr to consider the source and the destination has the same field structure (order). Which would be the most reasonable? I personally in favour of #3 or #1. #3 is easier to implement, but #1 is more customizable. With regards to #2 we might continue to encounter non trivial matches. There's even a 4th option that doesn't require any code change. Create a OGR VRT file that renames the source fields to the truncated shapefile field names (or the reverse : creates a OGR VRT that renames the truncated shapefile field names to the original field names, since OGR VRT now supports since a few versions CreateFeature()) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] ogr2ogr with append fails to convert truncated fields
2013/3/1 Even Rouault even.roua...@mines-paris.org There's even a 4th option that doesn't require any code change. Create a OGR VRT file that renames the source fields to the truncated shapefile field names (or the reverse : creates a OGR VRT that renames the truncated shapefile field names to the original field names, since OGR VRT now supports since a few versions CreateFeature()) This may probably work, but doesn't seem to be very user friendly. Actually I would prefer to include at least a flag to indicate that ogr2ogr should provide mapping the fields in the same order (don't lookup field index by name) for example: ogr2ogr -append -f ESRI Shapefile -nofieldreorder destination.shp [source] or with the previous fieldmap approach ogr2ogr -append -f ESRI Shapefile -fieldmap default destination.shp [source] Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] ogr2ogr with append fails to convert truncated fields
A simple temporary fix is to use the -sql option and use the AS keyword to set the alias of the original field name into the truncated name. On Sat, Mar 2, 2013 at 1:27 AM, Tamas Szekeres szeker...@gmail.com wrote: 2013/3/1 Even Rouault even.roua...@mines-paris.org There's even a 4th option that doesn't require any code change. Create a OGR VRT file that renames the source fields to the truncated shapefile field names (or the reverse : creates a OGR VRT that renames the truncated shapefile field names to the original field names, since OGR VRT now supports since a few versions CreateFeature()) This may probably work, but doesn't seem to be very user friendly. Actually I would prefer to include at least a flag to indicate that ogr2ogr should provide mapping the fields in the same order (don't lookup field index by name) for example: ogr2ogr -append -f ESRI Shapefile -nofieldreorder destination.shp [source] or with the previous fieldmap approach ogr2ogr -append -f ESRI Shapefile -fieldmap default destination.shp [source] Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Best regards, Chaitanya kumar CH. +91-9494447584 17.2416N 80.1426E ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] ogr2ogr with append fails to convert truncated fields
Hi All, Related to issue #3247 http://trac.osgeo.org/gdal/ticket/3247 we experienced, that if we use ogr2ogr to append source data to a shapefile destination (by using the -append flag) and field name truncation is taking place, the corresponding values are not copied. This is because when setting up the field index map (in ogr2ogr) the destination (truncated) field indexes are searched by the name of of the source fields (the long names). Since it's not always trivial to find the correct fields by names, how about exposing the field map itself as an option, something like: ogr2ogr -append -f ESRI Shapefile -fieldmap 0,1,2,3,4,5 destination.shp [source] Would that workaround be sufficient? Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] ogr2ogr with append fails to convert truncated fields
Le lundi 25 février 2013 16:57:08, Tamas Szekeres a écrit :
Hi All,
Related to issue #3247 http://trac.osgeo.org/gdal/ticket/3247 we
experienced, that if we use ogr2ogr to append source data to a shapefile
destination (by using the -append flag) and field name truncation is taking
place, the corresponding values are not copied.
This is because when setting up the field index map (in ogr2ogr) the
destination (truncated) field indexes are searched by the name of of the
source fields (the long names).
Since it's not always trivial to find the correct fields by names, how
about exposing the field map itself as an option, something like:
ogr2ogr -append -f ESRI Shapefile -fieldmap 0,1,2,3,4,5 destination.shp
[source]
Would that workaround be sufficient?
That would certainly work, although not particularly user friendly, but
shapefiles are not that user friendly with their various limitations.
Perhaps you could just work with spatialite DBs as intermediate and convert to
shapefile at the end if it is really needed ?
A potential alternative to -fieldmap would be to have :
class OGRLayer:
{
public:
virtual int GetFieldIndex(const char* pszFieldname);
}
int OGRLayer::GetFieldIndex(const char* pszFieldname)
{
return GetLayerDefn()-GetFieldIndex(pszFieldname);
}
int OGRShapeLayer::GetFieldIndex(const char* pszFieldname)
{
int idx = OGRLayer::GetFieldIndex(pszFieldname);
if( idx = 0 )
return idx;
/* otherwise implement here truncation logic to find best match */
/* caution: if there are several potential matches (for example
verylongcolumnname1 being truncated to verylongna and verylongcolumnname2
truncated to verylong~1) and you are looking for verylongcolumnXXX then return
-1 */
}
Best regards,
Tamas
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] ogr2ogr with append fails to convert truncated fields
Can we expect the fields to remain in the same order every time? Hi All, Related to issue #3247 http://trac.osgeo.org/gdal/ticket/3247 we experienced, that if we use ogr2ogr to append source data to a shapefile destination (by using the -append flag) and field name truncation is taking place, the corresponding values are not copied. This is because when setting up the field index map (in ogr2ogr) the destination (truncated) field indexes are searched by the name of of the source fields (the long names). Since it's not always trivial to find the correct fields by names, how about exposing the field map itself as an option, something like: ogr2ogr -append -f ESRI Shapefile -fieldmap 0,1,2,3,4,5 destination.shp [source] Would that workaround be sufficient? Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] ogr2ogr with append fails to convert truncated fields
So we seem to have 3 divergent options to provide a work around:
1. Expose fieldmap setting to let the user decide the order of the fields
to be copied.
2. Modify GetFieldIndex to provide the name based lookup by the truncated
name (for the shape driver specifically)
3. Provide an option to instruct ogr2ogr to consider the source and the
destination has the same field structure (order).
Which would be the most reasonable?
I personally in favour of #3 or #1. #3 is easier to implement, but #1 is
more customizable.
With regards to #2 we might continue to encounter non trivial matches.
Best regards,
Tamas
2013/2/25 Even Rouault even.roua...@mines-paris.org
Le lundi 25 février 2013 16:57:08, Tamas Szekeres a écrit :
Hi All,
Related to issue #3247 http://trac.osgeo.org/gdal/ticket/3247 we
experienced, that if we use ogr2ogr to append source data to a shapefile
destination (by using the -append flag) and field name truncation is
taking
place, the corresponding values are not copied.
This is because when setting up the field index map (in ogr2ogr) the
destination (truncated) field indexes are searched by the name of of the
source fields (the long names).
Since it's not always trivial to find the correct fields by names, how
about exposing the field map itself as an option, something like:
ogr2ogr -append -f ESRI Shapefile -fieldmap 0,1,2,3,4,5
destination.shp
[source]
Would that workaround be sufficient?
That would certainly work, although not particularly user friendly, but
shapefiles are not that user friendly with their various limitations.
Perhaps you could just work with spatialite DBs as intermediate and
convert to
shapefile at the end if it is really needed ?
A potential alternative to -fieldmap would be to have :
class OGRLayer:
{
public:
virtual int GetFieldIndex(const char* pszFieldname);
}
int OGRLayer::GetFieldIndex(const char* pszFieldname)
{
return GetLayerDefn()-GetFieldIndex(pszFieldname);
}
int OGRShapeLayer::GetFieldIndex(const char* pszFieldname)
{
int idx = OGRLayer::GetFieldIndex(pszFieldname);
if( idx = 0 )
return idx;
/* otherwise implement here truncation logic to find best match */
/* caution: if there are several potential matches (for example
verylongcolumnname1 being truncated to verylongna and verylongcolumnname2
truncated to verylong~1) and you are looking for verylongcolumnXXX then
return
-1 */
}
Best regards,
Tamas
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
