[gdal-dev] GDAL translate 16 bit to 8 bit
Hi, I'm working with a 16 bit SPOT 6 image that I want to convert from 16 bit pixel depth to 8 bit pixel depth. I say 16 bit, but if I load the file into Envi, it reports that uses only 12 bits. However gdalinfo reports 16 bit (see below). I've manage to do so on the multispectral image, but the result is poor colour wise. So I split the image into its four bands that is, I now have one image per band. When I now try to convert to 8 bit, I get a message that says: Warning 742327101_R1C1_band1_8bit.TIF not created. The command I'm using is: gdal_translate -ot Byte -of GTiff D:/SPOT 6/EightBit/IMG_SPOT6_MS_201304290812030_ORT_742327101_R1C1_band1.TIF 742327101_R1C1_band1_8bit.TIF gdalinfo for the original MS image gives the following result: Driver: GTiff/GeoTIFF Files: D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6_MS_201304290812030_ORT_742327101_R1C1.TIF D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6_MS_201304290812030_ORT_742327101_R1C1.tif.ovr D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6_MS_201304290812030_ORT_742327101_R1C1.TFW D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6_MS_201304290812030_ORT_742327101_R1C1.TIF.aux.xml Size is 7680, 7680 Coordinate System is: PROJCS[WGS 84 / UTM zone 34S, GEOGCS[WGS 84, DATUM[WGS_1984, SPHEROID[WGS 84,6378137,298.257223563, AUTHORITY[EPSG,7030]], AUTHORITY[EPSG,6326]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4326]], PROJECTION[Transverse_Mercator], PARAMETER[latitude_of_origin,0], PARAMETER[central_meridian,21], PARAMETER[scale_factor,0.9996], PARAMETER[false_easting,50], PARAMETER[false_northing,1000], UNIT[metre,1, AUTHORITY[EPSG,9001]], AUTHORITY[EPSG,32734]] Origin = (426894.000,6421830.000) Pixel Size = (6.000,-6.000) Metadata: AREA_OR_POINT=Area TIFFTAG_DATETIME=2013:12:05 14:23:06 TIFFTAG_IMAGEDESCRIPTION=B2, B1, B0, B3 TIFFTAG_XRESOLUTION=6 TIFFTAG_YRESOLUTION=6 Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 426894.000, 6421830.000) ( 20d13'23.42E, 32d20'16.91S) Lower Left ( 426894.000, 6375750.000) ( 20d13'10.50E, 32d45'13.25S) Upper Right ( 472974.000, 6421830.000) ( 20d42'46.12E, 32d20'24.35S) Lower Right ( 472974.000, 6375750.000) ( 20d42'41.34E, 32d45'20.81S) Center ( 449934.000, 6398790.000) ( 20d28' 0.35E, 32d32'49.70S) Band 1 Block=7680x128 Type=UInt16, ColorInterp=Red Min=0.000 Max=2768.000 Minimum=0.000, Maximum=2768.000, Mean=254.357, StdDev=82.827 Overviews: 3840x3840, 1920x1920, 960x960, 480x480, 240x240 Metadata: STATISTICS_COVARIANCES=6860.33555334015,5408.39177012214,3932.59000707709,8359.55872888383 STATISTICS_MAXIMUM=2768 STATISTICS_MEAN=254.35743888007 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=82.827142612432 Band 2 Block=7680x128 Type=UInt16, ColorInterp=Green Min=0.000 Max=1928.000 Minimum=0.000, Maximum=1928.000, Mean=272.449, StdDev=67.362 Overviews: 3840x3840, 1920x1920, 960x960, 480x480, 240x240 Metadata: STATISTICS_COVARIANCES=5408.39177012214,4537.57701249689,3512.4873688442,6910.89114933438 STATISTICS_MAXIMUM=1928 STATISTICS_MEAN=272.44932240804 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=67.361539564479 Band 3 Block=7680x128 Type=UInt16, ColorInterp=Blue Min=0.000 Max=2264.000 Minimum=0.000, Maximum=2264.000, Mean=291.925, StdDev=54.875 Overviews: 3840x3840, 1920x1920, 960x960, 480x480, 240x240 Metadata: STATISTICS_COVARIANCES=3932.59000707709,3512.4873688442,3011.23474438976,5111.02811591707 STATISTICS_MAXIMUM=2264 STATISTICS_MEAN=291.92457129584 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=54.874718626976 Band 4 Block=7680x128 Type=UInt16, ColorInterp=Undefined Min=0.000 Max=2699.000 Minimum=0.000, Maximum=2699.000, Mean=374.146, StdDev=115.219 Overviews: 3840x3840, 1920x1920, 960x960, 480x480, 240x240 Metadata: STATISTICS_COVARIANCES=8359.55872888383,6910.89114933438,5111.02811591707,13275.4755734648 STATISTICS_MAXIMUM=2699 STATISTICS_MEAN=374.14637317234 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=115.21925001259 gdalinfo for band 1 of the image that I'm trying to use for the
Re: [gdal-dev] RFC 15: Band Masks vs #5621
Hi Even, From: even.roua...@spatialys.com To: lucena_i...@hotmail.com Subject: Re: [gdal-dev] RFC 15: Band Masks vs #5621 Date: Wed, 20 Aug 2014 20:39:29 +0200 CC: gdal-dev@lists.osgeo.org I am trying to help a developer how wants to use VRTs as an intermediary between a GeoRaster and a PNG, in order to produce thumbnails. In that case, does the VRT needs to describe the mask band so that the PNG's driver will received the filtered image, with the zeroes regions masked out? You need to explicitely declare mask bands in the VRT. See the VRT documentation for that. But that will just declare the existence of the mask band, and will not do any processing such as zeroing masked regions. You are right. GDAL can copy the mask from one dataset to another, depending on the driver. But there is no way to indicate that you want the result of the masking, not in the API, not on command line tools. Maybe we should add that to the wish list. Regards, Ivan ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 15: Band Masks vs #5621
Selon Ivan Lucena lucena_i...@hotmail.com: Hi Even, From: even.roua...@spatialys.com To: lucena_i...@hotmail.com Subject: Re: [gdal-dev] RFC 15: Band Masks vs #5621 Date: Wed, 20 Aug 2014 20:39:29 +0200 CC: gdal-dev@lists.osgeo.org I am trying to help a developer how wants to use VRTs as an intermediary between a GeoRaster and a PNG, in order to produce thumbnails. In that case, does the VRT needs to describe the mask band so that the PNG's driver will received the filtered image, with the zeroes regions masked out? You need to explicitely declare mask bands in the VRT. See the VRT documentation for that. But that will just declare the existence of the mask band, and will not do any processing such as zeroing masked regions. You are right. GDAL can copy the mask from one dataset to another, depending on the driver. But there is no way to indicate that you want the result of the masking, not in the API, not on command line tools. Maybe we should add that to the wish list. Eh eh seems there is no limit to the wish list ;-) More seriously, I can imagine this could be done through a new VRT mechanism, that could be triggered by a new option of gdal_translate That would be mostly needed for per-band mask, since for per-dataset mask, you can already trick the mask to be considered as an alpha band (e.g. gdal_translate -b 1 -b 2 -b 3 -b mask in.tif out.tif will transform a RGB+mask into a RGBA dataset). For per-band mask, the value of the pixels that are masked should also be specified in the VRT syntax. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Adding a Commercial support section on gdal.org?
Charta is interested to be listed as experienced provider if gdal.org will ever add info about commercial support. Nevertheless, I think this item has to receive more attention, considering at least: * the license of GDAL. Using MIT license means being expecially open to commercial usage. I don't know if this is really consequential with selecting a group of commercial providers instead of being agnostic about usage of the code and technology created (I don't agree with this, but...); * usually support means choice between several tools and technologies, being a core contributor could mean some bias about recurring to a specific tool. Being listed as such could become a double-edged sword; * any evaluation aobut support should be based on the help provided to the user and not about the contribution to the project. The client shoud be able to choose if he needs broad or focused support, if he needs new developments or better integration and so on. Listing experiences or specialisation (geodatabases, remote sensing and so on) could be more useful. Even's case is noteworthy, because he is more than a Core contributor, but a project leader for GDAL. When his affiliation to École des Mines changed to his own company, Spatialys, I thought this was a very good move because he could more easily provide consulting services to commercial entities. Maybe his visibility as a prominent person in this community is already warranted, I understand that his proposal is addressed to others, but I think gdal-dev is about developing, although the list provides a lot of help to users, is this the right place? c On Thu, Aug 20, 2014 at 10:02:35 PM, Even Rouault wrote: Date: Wed, 20 Aug 2014 22:02:35 +0200 From: Even Rouault even.roua...@spatialys.com To: gdal-dev@lists.osgeo.org gdal-dev@lists.osgeo.org Subject: [gdal-dev] Adding a Commercial support section on gdal.org ? Message-ID: 201408202202.36022.even.roua...@spatialys.com Content-Type: Text/Plain; charset=us-ascii Hi, I'm wondering if there would be a concensus and interest to add a Commercial support section on gdal.org. A number of OSGeo projects have such page (see [1]), so that wouldn't be completely awkward to have one for GDAL as well. The OSGeo Service provider database reference 137 companies/individuals that have registered themselves as providing GDAL support ([2]) ! Pretty cool, but I'm wondering how a user not familiar with the project could effectively use that list to identify core contributors from casual advanced users. If we agree for adding a Commercial support section, the question is : on which criteria do we accept an organization/individual to be listed in the section ? We would want them to be as most objective and non debatable as possible. A simple criterion could be anyone who has commit rights (in trunk, not just in a sandbox or customer branch). There are currently 56 SVN committers. That could be strengthened with a minimum number of commits/lines changed during a period, but we perhaps don't need that level of complexity. We could possibly also extend that to entities that provide public support to users through gdal-dev or other public forums (gis.stackexchange, others?). Other suggestions ? Should we distinguish several categories of actors ? - QGIS makes a division between Core contributors vs Contributors. GeoServer has Core contributors, Experienced providers and Additional services (the last one is populated on service provider request). - On the other side, deegree, Geomoose or Geotools simply list them in a single section. The answer likely depends on the number of organizations that would be listed (I guess below 10 we don't need much structure). The difficulty here would be to establish the categories and criteria. So, could entities interested in being listed reply to this email so we can have a better idea of how many would be listed, and if we need more stricter criteria or several categories ? As far as I'm concerned, Spatialys would be interested. Best regards, Even [1] Non exhaustive list of OSGeo projects with a commercial support section : http://geoserver.org/support/ http://www.geomoose.org/info/commercial_support.html http://docs.geotools.org/latest/userguide/welcome/support.html http://wiki.deegree.org/deegreeWiki/GettingSupport http://qgis.org/en/site/forusers/commercial_support.html [2] OSGeo Service Provider catalog with entities declaring GDAL expertise : http://www.osgeo.org/search_profile?SET=1MUL_TECH[]=00013 -- Spatialys - Geospatial professional services http://www.spatialys.com -- -- Carlo A. Bertelli Charta servizi e
Re: [gdal-dev] Issues building ECW/JP2ECW driver as a plugin on Ubuntu
Am 21.08.2014 07:55, schrieb Luke: I cross-posted to the UbuntuGIS mailing list (http://article.gmane.org/gmane.comp.gis.osgeo.ubuntu/941), but have had no response. Unfortunately, yes. There seems to be little activity on the list recently. Are there any other drivers in the GDAL tree that support being built as plugins in a linux environment that I could have a look at and try and base a patch on? The bin\gdalplugins folder contains GEOR, MG4Lidar, MrSID, FILEGDB, OCI and SOSI. I guess your driver should work in a similar way. I don't have any skills in C coding and compiling, so I can't help you on that. Greetings, André Joost ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Adding a Commercial support section on gdal.org?
Adding a support section like some other OSGeo projects is a good idea (in my opinion). This is a way for people to use the project webpage to find those who are most substantially contributing to the project and know the most about GDAL/OGR. On Thu, Aug 21, 2014 at 5:46 AM, berte...@charta.acme.com wrote: Charta is interested to be listed as experienced provider if gdal.org will ever add info about commercial support. Nevertheless, I think this item has to receive more attention, considering at least: * the license of GDAL. Using MIT license means being expecially open to commercial usage. I don't know if this is really consequential with selecting a group of commercial providers instead of being agnostic about usage of the code and technology created (I don't agree with this, but...); * usually support means choice between several tools and technologies, being a core contributor could mean some bias about recurring to a specific tool. Being listed as such could become a double-edged sword; If you are looking for GDAL/OGR support (on the gdal.org web page) you have already made your technology choice. If you are looking for support in selecting a technology you should not be looking on gdal.org but rather collecting references or otherwise doing research on appropriate support in whatever domain you're in. gdal.org is not the the proper place to look for support on javascript libraries or other non-GDAL/OGR topics. Someone may be better off with advice to use something other than GDAL/OGR but gdal.org is not the place to get that advice (the people and companies listed there may give that advice if approached but they would best know what is out of the GDAL/OGR scope). * any evaluation aobut support should be based on the help provided to the user and not about the contribution to the project. The client shoud be able to choose if he needs broad or focused support, if he needs new developments or better integration and so on. Listing experiences or specialisation (geodatabases, remote sensing and so on) could be more useful. A prudent customer might evaluate support by referrals and reviews based on help provided to the user but gdal.org is in no place to evaluate that. gdal.org is certainly in a place to evaluate and recommend those who have a history of competently contributing to the project (tickets, patches, commits, emails correctly answered, relevant presentations at FOSS4G, etc). The OSGeo Service providers directory, http://www.osgeo.org/search_profile?SET=1MUL_TECH[0]=00013 already provides a mechanism for self listing anything the person listing wants. This remains a valuable resource for researching support options but gives no indication of who is competently contributing to the project. The PostGIS support page lists focus area of support, http://postgis.net/support Best regards, Eli Even's case is noteworthy, because he is more than a Core contributor, but a project leader for GDAL. When his affiliation to École des Mines changed to his own company, Spatialys, I thought this was a very good move because he could more easily provide consulting services to commercial entities. Maybe his visibility as a prominent person in this community is already warranted, I understand that his proposal is addressed to others, but I think gdal-dev is about developing, although the list provides a lot of help to users, is this the right place? c On Thu, Aug 20, 2014 at 10:02:35 PM, Even Rouault wrote: Date: Wed, 20 Aug 2014 22:02:35 +0200 From: Even Rouault even.roua...@spatialys.com To: gdal-dev@lists.osgeo.org gdal-dev@lists.osgeo.org Subject: [gdal-dev] Adding a Commercial support section on gdal.org ? Message-ID: 201408202202.36022.even.roua...@spatialys.com Content-Type: Text/Plain; charset=us-ascii Hi, I'm wondering if there would be a concensus and interest to add a Commercial support section on gdal.org. A number of OSGeo projects have such page (see [1]), so that wouldn't be completely awkward to have one for GDAL as well. The OSGeo Service provider database reference 137 companies/individuals that have registered themselves as providing GDAL support ([2]) ! Pretty cool, but I'm wondering how a user not familiar with the project could effectively use that list to identify core contributors from casual advanced users. If we agree for adding a Commercial support section, the question is : on which criteria do we accept an organization/individual to be listed in the section ? We would want them to be as most objective and non debatable as possible. A simple criterion could be anyone who has commit rights (in trunk, not just in a sandbox or customer branch). There are currently 56 SVN committers. That could be strengthened with a minimum number of commits/lines changed during a period, but we perhaps don't need that
Re: [gdal-dev] Adding a Commercial support section on gdal.org?
Folks, This is a somewhat sticky area, which is why I started just with just the self-registration mechanism on the OSGeo site in the past. A scenario that I could support would be a section somewhat like the postgis.net support list where being added to it needs to be voted on by the PSC. My criteria as a PSC member would be: - The organization has made significant contributions to the project (in code, docs, etc) - The organization has staff that I personally know to be competent GDAL/OGR developers. It is a slippery sort of thing of course. Subjective, and I would hate to be in the situation where I'm having to vote against an addition. If we were to pursue this I actually think an RFC with an initial list of entries, and some general principles would be appropriate (though additions wouldn't need an RFC - just a up/down vote). My perspective when consulting was that being active on the mailing list, and noting in my email signature that I was available for consulting was enough to give me some profile with those looking for someone. PS. as happy customer of Even's (at Planet Labs) I can strongly endorse him as a consultant! Best regards, Frank On Thu, Aug 21, 2014 at 8:37 AM, Eli Adam ea...@co.lincoln.or.us wrote: Adding a support section like some other OSGeo projects is a good idea (in my opinion). This is a way for people to use the project webpage to find those who are most substantially contributing to the project and know the most about GDAL/OGR. On Thu, Aug 21, 2014 at 5:46 AM, berte...@charta.acme.com wrote: Charta is interested to be listed as experienced provider if gdal.org will ever add info about commercial support. Nevertheless, I think this item has to receive more attention, considering at least: * the license of GDAL. Using MIT license means being expecially open to commercial usage. I don't know if this is really consequential with selecting a group of commercial providers instead of being agnostic about usage of the code and technology created (I don't agree with this, but...); * usually support means choice between several tools and technologies, being a core contributor could mean some bias about recurring to a specific tool. Being listed as such could become a double-edged sword; If you are looking for GDAL/OGR support (on the gdal.org web page) you have already made your technology choice. If you are looking for support in selecting a technology you should not be looking on gdal.org but rather collecting references or otherwise doing research on appropriate support in whatever domain you're in. gdal.org is not the the proper place to look for support on javascript libraries or other non-GDAL/OGR topics. Someone may be better off with advice to use something other than GDAL/OGR but gdal.org is not the place to get that advice (the people and companies listed there may give that advice if approached but they would best know what is out of the GDAL/OGR scope). * any evaluation aobut support should be based on the help provided to the user and not about the contribution to the project. The client shoud be able to choose if he needs broad or focused support, if he needs new developments or better integration and so on. Listing experiences or specialisation (geodatabases, remote sensing and so on) could be more useful. A prudent customer might evaluate support by referrals and reviews based on help provided to the user but gdal.org is in no place to evaluate that. gdal.org is certainly in a place to evaluate and recommend those who have a history of competently contributing to the project (tickets, patches, commits, emails correctly answered, relevant presentations at FOSS4G, etc). The OSGeo Service providers directory, http://www.osgeo.org/search_profile?SET=1MUL_TECH[0]=00013 already provides a mechanism for self listing anything the person listing wants. This remains a valuable resource for researching support options but gives no indication of who is competently contributing to the project. The PostGIS support page lists focus area of support, http://postgis.net/support Best regards, Eli Even's case is noteworthy, because he is more than a Core contributor, but a project leader for GDAL. When his affiliation to École des Mines changed to his own company, Spatialys, I thought this was a very good move because he could more easily provide consulting services to commercial entities. Maybe his visibility as a prominent person in this community is already warranted, I understand that his proposal is addressed to others, but I think gdal-dev is about developing, although the list provides a lot of help to users, is this the right place? c On Thu, Aug 20, 2014 at 10:02:35 PM, Even Rouault wrote: Date: Wed, 20 Aug 2014 22:02:35 +0200 From: Even Rouault even.roua...@spatialys.com To: gdal-dev@lists.osgeo.org gdal-dev@lists.osgeo.org Subject: [gdal-dev] Adding a Commercial
[gdal-dev] Notes on gdalbuildvrt.cpp
Assuming that VRTBuilder class inside gdalbuildvrt.cpp has been created to be reusable, I have some comments/questions that, depending on your input, may end up in pull requests. - *VRTBuilder::VRTBuilder, line 300* Instead of assigning the pointer create a copy to be consistent with the rest of the code. The memory is released twice, once at 1558 with CPLFree( panBandList ); and once at 347 with delete[] panBandList; - *VRTBuilder::pasDatasetProperties line 237* At line 1092 AnalyseRaster is called with a pointer inside pasDatasetProperties if (AnalyseRaster( hDS, dsFileName, pasDatasetProperties[i] )) However the array may be re-allocated at line 416; if the reallocation is not in place the psDatasetProperties pointer will be invalid. Maybe passing an index instead of a pointer can solve the problem. - *line 513-520* What would it take to support rotated, positive NS resolutions? - *line 543* Assumes that there is at least one band; The code below does not make that assumption and checks for that condition; GDALGetBlockSize() does not handle a NULL pointer. - *line 825, 920 * Why the GDALProxyPoolDatasetH? Why not opening it like a regular data set? When creating VRT files in C++ code this is the path to be taken? Regards, Nicu ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] GDAL Driver for MongoDB
Hi there, I wrote a GDAL driver for mongodb, and uploaded it in the githubhttps://github.com/mongogis/mongodb-gdal-driver. Feel free to grab and use the driver. Just let me know if you have any questions or suggestions. Thanks, shuai ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notes on gdalbuildvrt.cpp
Le jeudi 21 août 2014 19:32:34, Nicu Tofan a écrit : Assuming that VRTBuilder class inside gdalbuildvrt.cpp has been created to be reusable, Not particularly, but your remarks are interesting and beneficial to gdalbuildvrt itself. I have some comments/questions that, depending on your input, may end up in pull requests. - *VRTBuilder::VRTBuilder, line 300* Instead of assigning the pointer create a copy to be consistent with the rest of the code. The memory is released twice, once at 1558 with CPLFree( panBandList ); and once at 347 with delete[] panBandList; True, I could actually reproduce the issue with valgrind and passing the -b option of gdalbuildvrt. But there are code paths in VRTBuilder::AnalyseRaster() that allocate panBandList in some circumstances. So the right fix would be in the VRTBuilder constructor to do a copy of the passed panBandList if not NULL - *VRTBuilder::pasDatasetProperties line 237* At line 1092 AnalyseRaster is called with a pointer inside pasDatasetProperties if (AnalyseRaster( hDS, dsFileName, pasDatasetProperties[i] )) However the array may be re-allocated at line 416; if the reallocation is not in place the psDatasetProperties pointer will be invalid. Maybe passing an index instead of a pointer can solve the problem. I'm not sure there is an issue there. AnalyseRaster() will possibly relocate pasDatasetProperties, but as this is a member variable of the class, the next call of line 1092 will have the new address. Do you have a practical case where there would be an issue ? - *line 513-520* What would it take to support rotated, positive NS resolutions? Generally this is a sign of non georeferenced images. Perhaps some adaptations should be made if there are assumptions in the rest of the code about the sign of the y resolution - *line 543* Assumes that there is at least one band; The code below does not make that assumption and checks for that condition; GDALGetBlockSize() does not handle a NULL pointer. True. Should be moved after line 576 likely. - *line 825, 920 * Why the GDALProxyPoolDatasetH? Why not opening it like a regular data set? When creating VRT files in C++ code this is the path to be taken? If you are creating a VRT with more than 1024 sources, you would run out of file descriptors on Linux. GDALProxyPoolDatasetH is a trick to avoid having too many real datasets opened at the same time. If you don't have that constraint, you could use the real datasets, but this is needed in the general case of gdalbuildvrt. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL translate 16 bit to 8 bit
Le jeudi 21 août 2014 08:57:32, Hanlie Pretorius a écrit : Hi, I'm working with a 16 bit SPOT 6 image that I want to convert from 16 bit pixel depth to 8 bit pixel depth. I say 16 bit, but if I load the file into Envi, it reports that uses only 12 bits. However gdalinfo reports 16 bit (see below). I've manage to do so on the multispectral image, but the result is poor colour wise. So I split the image into its four bands that is, I now have one image per band. When I now try to convert to 8 bit, I get a message that says: Warning 742327101_R1C1_band1_8bit.TIF not created. Are you quoting the exact message ? I can't find any trace of it in the relevant parts of the source code. Did you check that you have write permissions in the directory where you launch the command ? Anyway for what you want to do you likely need to explore the -scale option of gdal_translate. But that's unrelated to the error The command I'm using is: gdal_translate -ot Byte -of GTiff D:/SPOT 6/EightBit/IMG_SPOT6_MS_201304290812030_ORT_742327101_R1C1_band1.TIF 742327101_R1C1_band1_8bit.TIF gdalinfo for the original MS image gives the following result: Driver: GTiff/GeoTIFF Files: D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE 1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6 _MS_201304290812030_ORT_742327101_R1C1.TIF D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE 1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6 _MS_201304290812030_ORT_742327101_R1C1.tif.ovr D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE 1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6 _MS_201304290812030_ORT_742327101_R1C1.TFW D:/SPOT 6/SANSA_SouthAfr_246_SO13012989_43_DS_SPOT6_201304290812030_FR1_FR1_SE1_SE 1_E021S32_03414/PROD_SPOT6_001/VOL_SPOT6_001_A/IMG_SPOT6_MS_001_A/IMG_SPOT6 _MS_201304290812030_ORT_742327101_R1C1.TIF.aux.xml Size is 7680, 7680 Coordinate System is: PROJCS[WGS 84 / UTM zone 34S, GEOGCS[WGS 84, DATUM[WGS_1984, SPHEROID[WGS 84,6378137,298.257223563, AUTHORITY[EPSG,7030]], AUTHORITY[EPSG,6326]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4326]], PROJECTION[Transverse_Mercator], PARAMETER[latitude_of_origin,0], PARAMETER[central_meridian,21], PARAMETER[scale_factor,0.9996], PARAMETER[false_easting,50], PARAMETER[false_northing,1000], UNIT[metre,1, AUTHORITY[EPSG,9001]], AUTHORITY[EPSG,32734]] Origin = (426894.000,6421830.000) Pixel Size = (6.000,-6.000) Metadata: AREA_OR_POINT=Area TIFFTAG_DATETIME=2013:12:05 14:23:06 TIFFTAG_IMAGEDESCRIPTION=B2, B1, B0, B3 TIFFTAG_XRESOLUTION=6 TIFFTAG_YRESOLUTION=6 Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 426894.000, 6421830.000) ( 20d13'23.42E, 32d20'16.91S) Lower Left ( 426894.000, 6375750.000) ( 20d13'10.50E, 32d45'13.25S) Upper Right ( 472974.000, 6421830.000) ( 20d42'46.12E, 32d20'24.35S) Lower Right ( 472974.000, 6375750.000) ( 20d42'41.34E, 32d45'20.81S) Center ( 449934.000, 6398790.000) ( 20d28' 0.35E, 32d32'49.70S) Band 1 Block=7680x128 Type=UInt16, ColorInterp=Red Min=0.000 Max=2768.000 Minimum=0.000, Maximum=2768.000, Mean=254.357, StdDev=82.827 Overviews: 3840x3840, 1920x1920, 960x960, 480x480, 240x240 Metadata: STATISTICS_COVARIANCES=6860.33555334015,5408.39177012214,3932.59000707709, 8359.55872888383 STATISTICS_MAXIMUM=2768 STATISTICS_MEAN=254.35743888007 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=82.827142612432 Band 2 Block=7680x128 Type=UInt16, ColorInterp=Green Min=0.000 Max=1928.000 Minimum=0.000, Maximum=1928.000, Mean=272.449, StdDev=67.362 Overviews: 3840x3840, 1920x1920, 960x960, 480x480, 240x240 Metadata: STATISTICS_COVARIANCES=5408.39177012214,4537.57701249689,3512.4873688442,6 910.89114933438 STATISTICS_MAXIMUM=1928 STATISTICS_MEAN=272.44932240804 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=67.361539564479 Band 3 Block=7680x128 Type=UInt16, ColorInterp=Blue Min=0.000 Max=2264.000 Minimum=0.000, Maximum=2264.000, Mean=291.925, StdDev=54.875 Overviews: 3840x3840, 1920x1920, 960x960, 480x480, 240x240 Metadata: STATISTICS_COVARIANCES=3932.59000707709,3512.4873688442,3011.23474438976,5 111.02811591707 STATISTICS_MAXIMUM=2264 STATISTICS_MEAN=291.92457129584 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=54.874718626976 Band 4 Block=7680x128 Type=UInt16, ColorInterp=Undefined Min=0.000 Max=2699.000
Re: [gdal-dev] Notes on gdalbuildvrt.cpp
Hello Even, - VRTBuilder::pasDatasetProperties line 237 DatasetProperty* psDatasetProperties This is a variable allocated on the stack; on function entry it contains a pointer somewhere inside pasDatasetProperties; Say pasDatasetProperties is 0x750 and psDatasetProperties is 0x7500100. Now on line pasDatasetProperties the buffer at 0x750gets reallocated, possibly moving to, say, 0x9000; psDatasetPropertiesshould now be 0x9100 but it is still 0x7500100 because it is stored in a variable on the stack - namelly the argument psDatasetPropertie. On line 457 psDatasetPropertie is used and is going to contain 0x750010. It may not trigger an access violation because the address will probably be inside the heap but it may generate some hard to find bugs that we all love and cherish. As a reflection of mine, this may be the result of the naming convention, as psDataset and pasDataset look the same to our animal brains. - Generally this is a sign of non georeferenced images... The bounding box becomes a bit more complicated in rotated images, too. I'm gonna look into it. - GDALProxyPoolDatasetH Got it, thanks! Regards, Nick PS Forgot to reply to the list, sorry. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notes on gdalbuildvrt.cpp
Le jeudi 21 août 2014 21:13:15, Nicu Tofan a écrit : Hello Even, - VRTBuilder::pasDatasetProperties line 237 DatasetProperty* psDatasetProperties This is a variable allocated on the stack; on function entry it contains a pointer somewhere inside pasDatasetProperties; Say pasDatasetProperties is 0x750 and psDatasetProperties is 0x7500100. Now on line pasDatasetProperties the buffer at 0x750gets reallocated, possibly moving to, say, 0x9000; psDatasetPropertiesshould now be 0x9100 but it is still 0x7500100 because it is stored in a variable on the stack - namelly the argument psDatasetPropertie. On line 457 psDatasetPropertie is used and is going to contain 0x750010. It may not trigger an access violation because the address will probably be inside the heap but it may generate some hard to find bugs that we all love and cherish. That shouldn't happen because of the return at line 453. But for clarity we might have a dedicated function for the processing done between line 412 and 454 since it doesn't need psDatasetProperties at all. As a reflection of mine, this may be the result of the naming convention, as psDataset and pasDataset look the same to our animal brains. Hungarian-style convention. ps is a Pointer to a single Structure pasX is a Pointer to an Array of Structure - Generally this is a sign of non georeferenced images... The bounding box becomes a bit more complicated in rotated images, too. I'm gonna look into it. Dealing with non-zero values in gt[2] and gt[4] is clearly out of scope of what a regular VRT can do (you need a warped VRT for that). Well you could still produce a regular VRT (with a rotated geotransform) if all the sources have the same rotation in their geotransform, but that's an unlikely situation. - GDALProxyPoolDatasetH Got it, thanks! Regards, Nick PS Forgot to reply to the list, sorry. -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notes on gdalbuildvrt.cpp
That shouldn't happen because of the return at line 453 Damn, I missed that. :) Hungarian-style convention Yes, I've read RFC8 https://trac.osgeo.org/gdal/wiki/rfc8_devguide; just thinking out loud. 2014-08-21 22:21 GMT+03:00 Even Rouault even.roua...@spatialys.com: Le jeudi 21 août 2014 21:13:15, Nicu Tofan a écrit : Hello Even, - VRTBuilder::pasDatasetProperties line 237 DatasetProperty* psDatasetProperties This is a variable allocated on the stack; on function entry it contains a pointer somewhere inside pasDatasetProperties; Say pasDatasetProperties is 0x750 and psDatasetProperties is 0x7500100. Now on line pasDatasetProperties the buffer at 0x750gets reallocated, possibly moving to, say, 0x9000; psDatasetPropertiesshould now be 0x9100 but it is still 0x7500100 because it is stored in a variable on the stack - namelly the argument psDatasetPropertie. On line 457 psDatasetPropertie is used and is going to contain 0x750010. It may not trigger an access violation because the address will probably be inside the heap but it may generate some hard to find bugs that we all love and cherish. That shouldn't happen because of the return at line 453. But for clarity we might have a dedicated function for the processing done between line 412 and 454 since it doesn't need psDatasetProperties at all. As a reflection of mine, this may be the result of the naming convention, as psDataset and pasDataset look the same to our animal brains. Hungarian-style convention. ps is a Pointer to a single Structure pasX is a Pointer to an Array of Structure - Generally this is a sign of non georeferenced images... The bounding box becomes a bit more complicated in rotated images, too. I'm gonna look into it. Dealing with non-zero values in gt[2] and gt[4] is clearly out of scope of what a regular VRT can do (you need a warped VRT for that). Well you could still produce a regular VRT (with a rotated geotransform) if all the sources have the same rotation in their geotransform, but that's an unlikely situation. - GDALProxyPoolDatasetH Got it, thanks! Regards, Nick PS Forgot to reply to the list, sorry. -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Adding a Commercial support section on gdal.org?
As Frank wrote, this is a slippery issue. Personally I could be comfortable with anything from self-registration to the highly selective approach described by Frank. To me, the important issue is making clear to a reader of the list what exactly the list means and how to use that to interpret the skills of those on the list. One way to use this list is as a reward to significant contributors to project. This would tend to point to those most familiar with the internals of the project, as well as having a broad commitment to the project and the notion of an open source community. Of course this requires a voting process, presumably by the PSC, which can be burdensome and stressful, as Frank notes. While I have found this project community to be generally welcoming, open source projects somewhat deservedly have a reputation for being insular and hard to crack. (For a great read, check out this article. Worth reading just for a remarkably intolerant response from Linus Torvalds on the merits of C++). A vetted list of names carries an implied endorsement, which is valuable to the reader, but carries a risk for the committee that chooses the list. (I'm not talking risk in the legal sense, though that could occur, I suppose. More the reflection on how the community chooses who to include or exclude.) As the other extreme, we allow anyone to register and hopefully provide some guidance in how to choose amongst them. For example, suggest that people search the archives of this mailing list to see how often the consultant participates. Put a star next to names who have commit privileges, perhaps the date the achieved this status, so you can tell how long they've been active. There are many ways to objectively identify the stronger contributors while remaining open. I am tempted to suggest even allowing endorsements, but policing that against spam, abuse, fraud is probably more work than it's worth. My choice leans to an open list of self-registrants with some objective measures of their participation, but I'll probably be content with whatever the community decides. On 8/21/2014 11:02 AM, Frank Warmerdam wrote: Folks, This is a somewhat sticky area, which is why I started just with just the self-registration mechanism on the OSGeo site in the past. A scenario that I could support would be a section somewhat like the postgis.net support list where being added to it needs to be voted on by the PSC. My criteria as a PSC member would be: - The organization has made significant contributions to the project (in code, docs, etc) - The organization has staff that I personally know to be competent GDAL/OGR developers. It is a slippery sort of thing of course. Subjective, and I would hate to be in the situation where I'm having to vote against an addition. If we were to pursue this I actually think an RFC with an initial list of entries, and some general principles would be appropriate (though additions wouldn't need an RFC - just a up/down vote). My perspective when consulting was that being active on the mailing list, and noting in my email signature that I was available for consulting was enough to give me some profile with those looking for someone. PS. as happy customer of Even's (at Planet Labs) I can strongly endorse him as a consultant! Best regards, Frank ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 47 and Threading
On Thu, Aug 21, 2014 at 11:36 AM, Jeff Lacoste jefflacosteg...@gmail.com wrote: Hi, Improving the thread safety of GDAL is a big improvement. I know this proposal is not claiming to 'fix' gdal thread safety but adress at least the cache safety. This is said, may be to help clarify more the proposal, we can state what the change would address and (may be more important) not address. Just to make it more specific about gain of caching per dataset instead of global one. Updated the RFC to hopefully reflect this better reflect some of this, please send more questions as you have them. Would this mean for ex. that batch translating datasets would gain from this and be done in a parallel way ? Since we can avoid avoid trashing a global trash. So instead of translation x datasets in a sequential manner now (with the proposed changes) this can be done in parallel ? Currently it is possible to do batch translating of datasets, but it is not efficient. The reason for this is that there is a point where each thread is attempting to add or remove gdalrasterblocks from the global cache. Once the global cache size is reached all the threads are often blocked for extended periods as each one attempts to clear the cache. The worst part is that this also prevents simple reading from the cache during this period, because other blocks can not efficiently use GDALRasterBand::TryGetLockedBlockRef, to pull existing blocks from the cache. The simple and first fix I made while doing this was simply to make it possible to operate with a per dataset cache (this was not selected as per band due to possible deadlock issues). Doing this allows for each dataset to have its own lock for its cache and the performance increases dramatically for operating on different datasets in parallel. If yes, what would the cache flushing strategy once the cach max is reached ? For ex. 5 running threads converting 5 datasets. We reach the max cache while the 5 threads are executing, this mean that threads will be blocked from executing as no cache is available until it has been released by other threads ? Jeff Lacoste In the case I described above each dataset has its own max cache. Therefore, reaching the maximum cache can only occur in a single dataset. Therefore if the maximum cache was reached on that cache, it would flush and not prevent other datasets and threads from operating on their own caches. However, this means that the person writing the code MUST be aware of the total cache sizes of all their datasets (I know this isn't ideal for many people). So I decided to take it a step farther and wanted to make it possible for a dataset using a driver such as memory to be able to do something such as: 5 running threads translating 1 dataset, 5 times. Also I wanted to be able to use one global cache and make it so that, the 5 threads translating 5 datasets would not lock as often with a global cache. To achieve this a separation of concerns had to occur, and this lead to the development of multiple mutexes. Three different data structure are at risk during threading within the cache: #1 - The Linked List of the cache and size of this linked list (this allows the cache to flush the least recently used GDALRasterBlock) #2 - The cache block array of a GDALRasterBand (this allows the GDALRasterBand to find its GDALRasterBlocks) #3 - The data stored within the GDALRasterBlock (this is the actual data stored in the block) If you are limiting the scope of our support of threading to only allow 1 thread to access any 1 dataset at a time #3 requires no protection. However, in any cache that is shared by more then 1 dataset (global cache), not only must the linked list be protected by the list global cache, but you must protect the cache block array in the raster bands. Since currently when items are removed from the cache, it simply removes blocks until it is below the cache limit, all raster bands can not read from their cache block arrays until all flushing has completed. Therefore, during my design I decided that each portion should be protected by its own mutex. In order to not require the linked list (LRU) mutex from being locked for extended periods, I mark blocks for deletion when it is to removed from the cache and it will be removed at the earliest safe period, which unless another the block is about to be used by another thread, will be right away. Otherwise it will occur once the other thread is done using that block. Therefore, this also means the mutex protecting the global cache in my code is also locked less. Therefore, even with out using a per dataset cache, the example of using 5 threads to translate 5 datasets should still be faster then the current configuration. TLDR; There are benefits in my design outside of just a non global cache for datasets. Blake Thompson ___ gdal-dev mailing list gdal-dev@lists.osgeo.org
Re: [gdal-dev] RFC 47 and Threading
On Thu, Aug 21, 2014 at 8:26 AM, Even Rouault even.roua...@spatialys.com wrote: I guess the silence in thread is due to people being impressed by the austerity of the topic... Something like that :) On the other side, I would be very pleased with having just the preliminary step of Blake's work, i.e. the possibility to choose a per-band block cache strategy instead of the global block cache. That should already address most scaling issues. I agree - from reading it seems like the major improvement is shifting away from a global, locking cache to a per-dataset cache. (ah. has been edited since you read it?) Which personally I think is worthwhile for any code/application which reads a variety of gdal datasources. Quick question - presumably for VRT datasets any source images currently share the global cache and are treated from this proposals' POV as their own datasets? As well as the VRT being a separate dataset? If so, seems like this could be quite a major win for users with VRTs for tiled images? While it's a lot of code change, it's mostly simple/repeated changes. Solution 2 (RW Mutex in GDALRasterBlock) Cons: It means that writing a driver is not as trivial as before and care must be taken in how locking is done within the driver in order to prevent deadlocks and maintain thread safety. I'm wary about making drivers more complex, there's a lot of them. And many are unlikely to be tested in multi-threaded anger during initial development (and MT issues can be hard to unit test). Can you explain a bit further what would be the impact for typical driver styles? Can we make the typical case (no/locked multithreaded access) really easy? What sort of driver code would be needed for drivers that can do MT reads? MT reads and writes? Rob :) OT: can we change gdal-dev to be reply-all by default? :) -- *Koordinates*PO Box 1604, Shortland St, Auckland 1140, New Zealand Phone +64-9-966 0433 koordinates.com https://koordinates.com/about ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
