Re: [gdal-dev] RE: Memory Error in _gdal.Band_ReadRaster(*args, **kwargs)

[Greg Coats]

It is my understanding that the default is
gdal_translate --config GDAL_CACHEMAX 40 ...
which equates to 40 MB of memory.
If you want to allow gdal_translate to use say 2 GB of memory for  
GDAL_CACHEMAX, enter

gdal_translate --config GDAL_CACHEMAX 2000 ...
In my experience, increasing GDAL_CACHEMAX substantially speeds up  
gdal_translate processing.

Greg

On Aug 14, 2009, at 4:08 PM, Paul Meems wrote:

I also noticed both gdal_translate and gdal_retile not use the full  
potential of my hardware. I'm running Vista 64Bit on a AMD3 2.6Ghz  
quadcore with 8GB RAM and only 2-2.5GB RAM was used by  
gdal_translate, leaving more than 5 GB free and only 35-55% of my  
CPU was used. Do I need to set some settings to speed things up?


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL WKT Raster cache: Problem, options to solve and doubts

[Tamas Szekeres]

Hi Jorge,

I really appreciate your efforts to sort these issues out. I consider these
issues would be crucial especially when the requested block number is large
or when using non regular (and possibly overlapped) blocking where each
IReadBlock may result in fetching multiple raster rows.

However I consider we should start considering how to finalize the GSoc
project at a well defined stage, which we could safely be incorporated in
the gdal project tree as a new driver. It would be required to write up some
documentation or test scripts to make sure the existing code is working
correctly, then we should also submit the final evaluations within the Aug
17-24 interval according to the gsoc timeline. By all means please calculate
the necessary efforts to implement this enhancement we have recently been
talking about, and if it cannot be fitted in the gsoc timeline then go ahead
finalizing the existing implementation by adding the driver documentation
and scripts.

So as to implement these enhancements after finishing the gsoc efforts
you're happily welcomed in the gdal team to contribute with the remaining
parts to make this driver really compelling to use. Keep up the good work,


Best regards,

Tamas




2009/8/14 Jorge Arévalo 

> Hello,
>
> I've asked several concepts related with RasterIO-related methods in
> Dataset and RasterBand. Thanks to your responses, I've a better
> understanding of the GDAL drivers' I/O method. But I've a couple of
> doubts I need to solve to finish the GSoC, although I'd like to
> continue developing the driver after it.
>
> Problem: In basic GDAL WKT Raster driver, each row of a raster table
> (one block, in regularly blocked rasters) means one server round. This
> is slow, and "sub-optimal".
>
> How to solve it?: IReadBlock executes an spatial query to get all the
> rows of a table that fits in a natural block, defined by nXOff, nYOff
> and the values fetched by GetBlockSize. In many raster formats, one
> natural block is a scanline, of size (nXRasterSize, 1). In WKT Raster
> format, if we have a regularly blocked raster,  "natural" block size
> will be equal to RASTER_COLUMNS-defined block size, and the query will
> return one block. To avoid one spatial query for each block petition,
> we should "force" the driver to get all the blocks covering the area
> requested in a IRasterIO call.
>
> How to implement it? My approach is based on implementing
> WKTRasterDataset::IRasterIO method (overriding GDALDataset::IRasterIO
> method). This method executes a spatial query that returns all the
> raster rows covering the area requested. Now, I have all the data of
> an image region. If the requested region dimensions match the buffer
> dimensions, I can copy all the pixels fetched in this way:
>
> pImage =
> {b1b1b1b1b1b1b1b1b1b1...|b2b2b2b2b2b2b2b2b2...|...|bnbnbnbnbnbnbnbnbn}
>
> Where bi are the bytes of the band i. A WKT Raster image has a
> non-interleaved format, all the band are consecutive. Is it correct? I
> mean, copy the data in pImage with this format.
>
> And if the region dimensions don't match the buffer dimensiones,
> should I raise an error and finish or delegate in base
> GDALDataset::IRasterIO implementation?
>
> Another question: After copying the data from fetched rows in pImage
> buffer, should I do anything more?
>
> Other part of my implementation is overriding
> GDALRasterBand::IRasterIO method. My method will simply call the
> WKTRasterDataset::IRasterIO method with only one band to read. Is it
> correct?
>
> Thanks in advance.
>
> Best regards,
> Jorge
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] RE: Memory Error in _gdal.Band_ReadRaster(*args, **kwargs)

[Paul Meems]

I've been in contact with Christian the author of gdal_retile and he told me
this problem is in the main GDAL library.
He also suggested using some of the other GDAL tools to see if the problem
persists.

So I started using gdal_translate to copy my ecw to a new ecw using this
statement:
gdal_translate -a_srs EPSG:28992 -of ECW  -co "LARGE_OK=YES" large.ecw
Large_translate.ecw

After 6 hours (!) I got a new ecw file, this time 4.2GB with this info:
Driver: ECW/ERMapper Compressed Wavelets
Files: Large_translate.ecw
Size is 173334, 16
Coordinate System is:
PROJCS["NUTM31",
GEOGCS["RIJKDRIEHOEKSMETING",
DATUM["RD",
SPHEROID["BESS1841",6377397.155,299.1528128]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",3],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",50],
PARAMETER["false_northing",0],
UNIT["Meter",1]]
Origin = (33000.000,381000.000)
Pixel Size = (0.1501455,-0.1523283)
Corner Coordinates:
Upper Left  (   33000.000,  381000.000) (  1d12'3.77"W,  3d26'16.85"N)
Lower Left  (   33000.000,  357000.000) (  1d12'0.46"W,  3d13'17.23"N)
Upper Right (   59000.100,  381000.000) (  0d58'3.16"W,  3d26'20.47"N)
Lower Right (   59000.100,  357000.000) (  0d58'0.03"W,  3d13'20.61"N)
Center  (   46000.050,  369000.000) (  1d 5'1.86"W,  3d19'48.81"N)
Band 1 Block=173334x1 Type=Byte, ColorInterp=Red
  Overviews: arbitrary
Band 2 Block=173334x1 Type=Byte, ColorInterp=Green
  Overviews: arbitrary
Band 3 Block=173334x1 Type=Byte, ColorInterp=Blue
  Overviews: arbitrary

I tried to retile this ecw file using this command:
gdal_retile -v -s_srs EPSG:28992 -of ECW -ps 17335 16000 -targetDir tiles
Large_translate.ecw >retile.txt
But got the same memory error.

Next I tried converting to GeoTiff using this command:
gdal_translate -a_srs EPSG:28992 -of GTiff  Large.ecw Large.tiff
After 2.5 hours I got a 77.5GB Tiff file with this info:
Driver: GTiff/GeoTIFF
Files: Large.tiff
Size is 173334, 16
Coordinate System is:
PROJCS["Amersfoort / RD New",
GEOGCS["Amersfoort",
DATUM["Amersfoort",
SPHEROID["Bessel 1841",6377397.155,299.152812809,
AUTHORITY["EPSG","7004"]],
AUTHORITY["EPSG","6289"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4289"]],
PROJECTION["Oblique_Stereographic"],
PARAMETER["latitude_of_origin",52.156160],
PARAMETER["central_meridian",5.3876389],
PARAMETER["scale_factor",0.079],
PARAMETER["false_easting",155000],
PARAMETER["false_northing",463000],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AUTHORITY["EPSG","28992"]]
Origin = (33000.000,381000.000)
Pixel Size = (0.1501455,-0.1523283)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (   33000.000,  381000.000) (  3d38'2.16"E, 51d24'21.02"N)
Lower Left  (   33000.000,  357000.000) (  3d38'31.86"E, 51d11'24.60"N)
Upper Right (   59000.100,  381000.000) (  4d 0'27.21"E, 51d24'39.10"N)
Lower Right (   59000.100,  357000.000) (  4d 0'50.58"E, 51d11'42.56"N)
Center  (   46000.050,  369000.000) (  3d49'27.95"E, 51d18'2.36"N)
Band 1 Block=173334x1 Type=Byte, ColorInterp=Red
Band 2 Block=173334x1 Type=Byte, ColorInterp=Green
Band 3 Block=173334x1 Type=Byte, ColorInterp=Blue

When I try to retile this tiff using:
gdal_retile -v -s_srs EPSG:28992 -of GTiff -ps 17335 16000 -targetDir tiles
Large.tiff >retile.txt
I got again the memory error.

I don't know anymore how to to this. How can I split my large ecw file into
smaller tiles?

I also noticed that even if I use the same EPSG code I get different
projection info, even differences when using only gdal_translate.
The projection is the same, they only have different naming. Why is that?

I also noticed both gdal_translate and gdal_retile not use the full
potential of my hardware. I'm running Vista 64Bit on a AMD3 2.6Ghz quadcore
with 8GB RAM and only 2-2.5GB RAM was used by gdal_translate, leaving more
than 5 GB free and only 35-55% of my CPU was used. Do I need to set some
settings to speed things up?

Thanks,

Paul



From: Paul Meems  gmail.com>
Subject: Memory Error in _gdal.Band_ReadRaster(*args,
**kwargs)
Newsgroups: 
gmane.comp.gis.gdal.devel
Date: 2009-08-10 21:22:06 GMT

I'm trying to split a large ECW file into smaller files.
I've downloaded and installed OSGeo4W with gdal16
I managed to split a small ecw file.
That ecw file had this info:

Driver: ECW/ERMapper Compressed Wavelets
Files: tilertest.ecw
Size is 6800, 7500
Coordinate System is `'
O

Re: [gdal-dev] Getting BigTiff support in Ubuntu 9.04 64bits

[Felix Schalck]

2009/8/14 Nikos Alexandris :
> Felix Schalck:
>> Dear Frank,
>> After some extensive research, I discovered this:
>>
>> qwer...@qwerkus-desktop:~$ ls -la /usr/local/bin/gdalwarp
>> -rwxr-xr-x 1 root root 95391 2009-08-14 20:20 /usr/local/bin/gdalwarp
>> qwer...@qwerkus-desktop:~$ ls -la /usr/bin/gdalwarp
>> -rwxr-xr-x 1 root root 31608 2009-03-10 00:32 /usr/bin/gdalwarp
>>
>> Looks like your were right: it is an installation problem: looks like
>> the installation script doesn't put gdal in the sam directory as the
>> ubuntu package manager.
>> Now, here comes another problem:
>>
>> By invoquing /usr/local/bin/gdalwarp, I have this error message:
>>
>> gdalwarp: error while loading shared libraries: libgdal.so.1: cannot
>> open shared object file: No such file or directory
>>
>> Although, by launching gdalwarp from the compilation directory (in
>> scr/apps), It works fine INCLUDING BIGTIFF option.
>> How can I fix the linking problem ?
>
> Felix,
> I use (ubuntu 9.04 / 64-bit), among others, the following
> in /etc/ld.so.conf:
>
> ---%<---%<--
> # share
> /usr/local/share/gdal
>
> # gdal
> /usr/local/lib/gdalplugins
> #/geo/osgeo/FWTools-2.0.6/bin_safe
> # I use the last line to read HDF files (like MODIS MOD09 products)
> ---%<---%<--
>
> No warranty though. I am still messing up my system from time to time
> and kill my time trying to fix it (or loading the lists with
> questions :-p).
>
> Maybe there should be a separate section in the wiki to explain
> step-by-step linking issues, shared libraries, etc.
>
> FYI, below I am copy-pasting notes with respect to "./configuration" for
> GDAL compilation [*].
>
> Kindest regards, Nikos
> ---
>
> [*] GDAL related compilation notes
>
> *Must* compile without GRASS support
>
> # enter in directory gdal_stable
> cd /usr/local/src/gdal_stable
>
> # download with svn ## or update with svn up
>
> # clean previous configurations/compilations
> make distclean
>
> # configure
> CFLAGS="-g -Wall" LDFLAGS="-s" ./configure \
> --with-png=internal \
> --with-libtiff=internal \
> --with-geotiff=internal \
> --with-jpeg=internal \
> --with-gif=internal \
> --with-ecw=yes \
> --with-expat=yes \
> --with-expat-inc=/usr/include/ \
> --with-expat-lib=/usr/lib \
> --with-sqlite3=yes \
> --with-geos=yes \
> --with-python \
> --with-libz=internal \
> --with-netcdf \
> --with-sqlite \
> --with-threads=yes \
> --without-grass  \
> --without-ogdi \
> --with-pg=/usr/bin/pg_config \
> --with-xerces=yes
>
> #compile & install & ldconfig
> make  &&  sudo make install  &&  sudo ldconfig
>
>
> Where is GDAL installed?
>
> # in /usr/local/bin
> /usr/local/bin/gdal-config \
> /usr/local/bin/gdal2tiles.py \
> /usr/local/bin/gdal2xyz.py \
> /usr/local/bin/gdal_contour \
> /usr/local/bin/gdal_grid \
> /usr/local/bin/gdal_merge.py \
> /usr/local/bin/gdal_rasterize \
> /usr/local/bin/gdal_retile.py \
> /usr/local/bin/gdal_translate \
> /usr/local/bin/gdaladdo \
> /usr/local/bin/gdalchksum.py \
> /usr/local/bin/gdalenhance \
> /usr/local/bin/gdalident.py \
> /usr/local/bin/gdalimport.py \
> /usr/local/bin/gdalinfo \
> /usr/local/bin/gdalmanage \
> /usr/local/bin/gdaltindex \
> /usr/local/bin/gdaltransform \
> /usr/local/bin/gdalwarp \
>
> # in /usr/local/share \
> /usr/local/share/gdal \
>
> # in /usr/local/lib \
> /usr/local/lib/libgdal.a \
> /usr/local/lib/libgdal.la \
> /usr/local/lib/libgdal.so \
> /usr/local/lib/libgdal.so.1 \
> /usr/local/lib/libgdal.so.1.12.3 \
>
> # in  /usr/local/include/
> /usr/local/include/gdal.h \
> /usr/local/include/gdal_alg.h \
> /usr/local/include/gdal_frmts.h \
> /usr/local/include/gdal_pam.h \
> /usr/local/include/gdal_priv.h \
> /usr/local/include/gdal_rat.h \
> /usr/local/include/gdal_version.h \
> /usr/local/include/gdal_vrt.h \
> /usr/local/include/gdalgrid.h \
> /usr/local/include/gdaljp2metadata.h \
> /usr/local/include/gdalwarper.h \
>
>

Very nice ! This helped a lot. No linking error anymore.
Many thanks,

Felix
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Strange things with gdalwarp ...

[Felix Schalck]

Hello,

As explained in my previous question, I'm trying to create a high
resolution topographic map of europe based on cgiar processed srtm
data. Of cours, the first step is to merge the 5*5° tif tiles into one
big tif, which can be achieved using gdalwarp's mosaic feature.

Simply pasting all the tiles togther trough following code:

gdalwarp --config GDAL_CACHEMAX 512 -wm 512  HDA1/srtm_*.ti HDA2/final.tif

takes about 20 hours. HDA1/2 are two different harddrives.

Now, a second script, pasting first "slices" of 5*30° together, and
than merging all the slices:

for i in 39 40 41 42 43; do
gdalwarp --config GDAL_CACHEMAX 512 -wm 512 HDA1/srtm_$i*.tif
HDA2/slice_0$((i-34)).tif
done
gdalwarp --config GDAL_CACHEMAX 512 -wm 512 HDA2/slice_*.tif HDA1/final.tif

takes less than on hour ! Filesize is similar.

So what is the trick here ? Is there some quality loss when using gdalwarp ?

Felix Schalck
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Getting BigTiff support in Ubuntu 9.04 64bits

[Nikos Alexandris]

Felix Schalck:
> Dear Frank,
> After some extensive research, I discovered this:
> 
> qwer...@qwerkus-desktop:~$ ls -la /usr/local/bin/gdalwarp
> -rwxr-xr-x 1 root root 95391 2009-08-14 20:20 /usr/local/bin/gdalwarp
> qwer...@qwerkus-desktop:~$ ls -la /usr/bin/gdalwarp
> -rwxr-xr-x 1 root root 31608 2009-03-10 00:32 /usr/bin/gdalwarp
> 
> Looks like your were right: it is an installation problem: looks like
> the installation script doesn't put gdal in the sam directory as the
> ubuntu package manager.
> Now, here comes another problem:
> 
> By invoquing /usr/local/bin/gdalwarp, I have this error message:
> 
> gdalwarp: error while loading shared libraries: libgdal.so.1: cannot
> open shared object file: No such file or directory
> 
> Although, by launching gdalwarp from the compilation directory (in
> scr/apps), It works fine INCLUDING BIGTIFF option.
> How can I fix the linking problem ?

Felix,
I use (ubuntu 9.04 / 64-bit), among others, the following
in /etc/ld.so.conf:

---%<---%<--
# share
/usr/local/share/gdal

# gdal
/usr/local/lib/gdalplugins
#/geo/osgeo/FWTools-2.0.6/bin_safe
# I use the last line to read HDF files (like MODIS MOD09 products)
---%<---%<--

No warranty though. I am still messing up my system from time to time
and kill my time trying to fix it (or loading the lists with
questions :-p).

Maybe there should be a separate section in the wiki to explain
step-by-step linking issues, shared libraries, etc.

FYI, below I am copy-pasting notes with respect to "./configuration" for
GDAL compilation [*].

Kindest regards, Nikos
---

[*] GDAL related compilation notes

*Must* compile without GRASS support

# enter in directory gdal_stable
cd /usr/local/src/gdal_stable

# download with svn ## or update with svn up

# clean previous configurations/compilations
make distclean

# configure
CFLAGS="-g -Wall" LDFLAGS="-s" ./configure \
--with-png=internal \
--with-libtiff=internal \
--with-geotiff=internal \
--with-jpeg=internal \
--with-gif=internal \
--with-ecw=yes \
--with-expat=yes \
--with-expat-inc=/usr/include/ \
--with-expat-lib=/usr/lib \
--with-sqlite3=yes \
--with-geos=yes \
--with-python \
--with-libz=internal \
--with-netcdf \
--with-sqlite \
--with-threads=yes \
--without-grass  \
--without-ogdi \
--with-pg=/usr/bin/pg_config \
--with-xerces=yes

#compile & install & ldconfig
make  &&  sudo make install  &&  sudo ldconfig


Where is GDAL installed?

# in /usr/local/bin
/usr/local/bin/gdal-config \
/usr/local/bin/gdal2tiles.py \
/usr/local/bin/gdal2xyz.py \
/usr/local/bin/gdal_contour \
/usr/local/bin/gdal_grid \
/usr/local/bin/gdal_merge.py \
/usr/local/bin/gdal_rasterize \
/usr/local/bin/gdal_retile.py \
/usr/local/bin/gdal_translate \
/usr/local/bin/gdaladdo \
/usr/local/bin/gdalchksum.py \
/usr/local/bin/gdalenhance \
/usr/local/bin/gdalident.py \ 
/usr/local/bin/gdalimport.py \
/usr/local/bin/gdalinfo \
/usr/local/bin/gdalmanage \
/usr/local/bin/gdaltindex \
/usr/local/bin/gdaltransform \
/usr/local/bin/gdalwarp \

# in /usr/local/share \
/usr/local/share/gdal \

# in /usr/local/lib \
/usr/local/lib/libgdal.a \
/usr/local/lib/libgdal.la \
/usr/local/lib/libgdal.so \
/usr/local/lib/libgdal.so.1 \
/usr/local/lib/libgdal.so.1.12.3 \

# in  /usr/local/include/
/usr/local/include/gdal.h \
/usr/local/include/gdal_alg.h \
/usr/local/include/gdal_frmts.h \
/usr/local/include/gdal_pam.h \
/usr/local/include/gdal_priv.h \
/usr/local/include/gdal_rat.h \
/usr/local/include/gdal_version.h \
/usr/local/include/gdal_vrt.h \
/usr/local/include/gdalgrid.h \
/usr/local/include/gdaljp2metadata.h \
/usr/local/include/gdalwarper.h \

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Getting BigTiff support in Ubuntu 9.04 64bits

[Felix Schalck]

Dear Frank,

After some extensive research, I discovered this:

qwer...@qwerkus-desktop:~$ ls -la /usr/local/bin/gdalwarp
-rwxr-xr-x 1 root root 95391 2009-08-14 20:20 /usr/local/bin/gdalwarp
qwer...@qwerkus-desktop:~$ ls -la /usr/bin/gdalwarp
-rwxr-xr-x 1 root root 31608 2009-03-10 00:32 /usr/bin/gdalwarp

Looks like your were right: it is an installation problem: looks like
the installation script doesn't put gdal in the sam directory as the
ubuntu package manager.
Now, here comes another problem:

By invoquing /usr/local/bin/gdalwarp, I have this error message:

gdalwarp: error while loading shared libraries: libgdal.so.1: cannot
open shared object file: No such file or directory

Although, by launching gdalwarp from the compilation directory (in
scr/apps), It works fine INCLUDING BIGTIFF option.
How can I fix the linking problem ?

Thanks for your patience,

Felix Schalck


2009/8/14 Frank Warmerdam :
> On Fri, Aug 14, 2009 at 1:09 PM, Felix Schalck wrote:
>> Well, I did the whole procedure again, after cleaning, and checked the
>> installed version with:
>>
>> $gdal-config
>>
>> Result:
>>
>> 1.6.2
>>
>> Looks like the correct version is installed, so that I'm quite lost.
>
> Felix,
>
> This is not really that conclusive as a way of ensuring you
> are using your new build but I don't know what else to suggest.
>
>> Notice that I found a similar problem on the mailing list archive here:
>>
>> http://n2.nabble.com/Trying-to-get-BIGTIFF-to-work-td2035106.html
>>
>> But the mail reamains unclear about the way to solve it. Looks like he
>> had to build everything against external libs; what options does this
>> mean for my ./configure command ?
>
> I don't see much in this discussion that implies external libraries
> are required for bigtiff support in 1.6.x.
>
> You could try a development snapshot from:
>
>  http://download.osgeo.org/gdal/daily
>
> These are quite stable.
>
> Best regards,
>
> --
> ---+--
> I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Programmer for Rent
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Definite problem with XERCES 3.0.x and GDAL

[Ethan Alpert]

 

I'm running 1.6.1 stable that I downloaded on 7/14 and xerces 3.0.1 on a
RHEL5.2 64 bit system.

 

When I convert a shapefile to GML and then try to convert back to
shapefile I was getting:

 

ERROR 1:  element missing value

 

And an invalid shapefile with no polygons. I'd get this error at various
other points too.

 

 

I spent some time in gdb and found that a XERCES method parseNext inside
of the GMLReader::NextFeature() was returning an empty coordinates block
even though my GML had coordinates.

 

I backed out of xerces 3.0 to xerces 2.8 and recompiled GDAL and reran
my test and now everything is working.

 

I'm not sure what the next steps are here but xerces 3.0 and 3.0.1 did
not work with OGR.

 

-e

 


This electronic communication and any attachments may contain confidential and 
proprietary 
information of DigitalGlobe, Inc. If you are not the intended recipient, or an 
agent or employee 
responsible for delivering this communication to the intended recipient, or if 
you have received 
this communication in error, please do not print, copy, retransmit, disseminate 
or 
otherwise use the information. Please indicate to the sender that you have 
received this 
communication in error, and delete the copy you received. DigitalGlobe reserves 
the 
right to monitor any electronic communication sent or received by its 
employees, agents 
or representatives.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Getting BigTiff support in Ubuntu 9.04 64bits

[Felix Schalck]

Many thanks for your reply: the configuration now shows

  LIBTIFF support:   internal (BigTIFF=yes)

I ran make & make install, but I'm still getting the following error
message, whie trying to merge my .tifs:

$gdalwarp -co bigtiff=yes *.tif europe_all.tif
Warning 6: Driver GTiff does not support bigtiff creation option
Creating output file that is 78000P x 36000L.
ERROR 6: A 78000 pixels x 36000 lines x 1 bands Int16 image would be
larger than 4GB
but this is the largest size a TIFF can be.  Creation failed.

Is there somthing else to reconfigure/recompile to get it work ? Is
there some wrong linking between the libs ?

Thanks for your time,

Felix Schalck

2009/8/14 Frank Warmerdam :
> Felix,
>
> Please use --with-libtiff=internal instead of --with-tiff=internal.
>
> On Fri, Aug 14, 2009 at 11:17 AM, Felix Schalck 
> wrote:
>> Hello,
>>
>> I'm trying to create a high resolution topographic map of europe in
>> grass, using cgiar-cis srtm datas. The data is shipped in 5*5° tiles
>> which I need to somhow paste together. On the grass mailing list, I
>> was told to use gdalwarp in order to achieve this. Unfortunately, my
>> current gdalwarp command (1.5) tells me the final TIF file will be
>> over 4GB large, so that I need to get BigTIFF support.
>>
>> To do so, I downloaded the 1.6 source from the main gdal site, and
>> launched ./configure with the following options:
>>
>> sudo CFLAGS="-O2 -march=k8 -pipe" ./configure --with-tiff=internal
>> --with-hide-internal-symbols --with-jpeg=/usr --enable-largefile
>> --with-libz=internal
>>
>> The command works fine, but the final message gives:
>>
>>  LIBTIFF support:           external (BigTIFF=no)
>>  LIBGEOTIFF support:        internal
>>
>> Somehow there must be a config option I'm missing; and I would be very
>> thankful if somebody in here could explain me what I'm doing wrong.
>>
>> regards,
>>
>> Felix Schalck
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
>
> --
> ---+--
> I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Programmer for Rent
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Getting BigTiff support in Ubuntu 9.04 64bits

[Frank Warmerdam]

Felix,

Please use --with-libtiff=internal instead of --with-tiff=internal.

On Fri, Aug 14, 2009 at 11:17 AM, Felix Schalck wrote:
> Hello,
>
> I'm trying to create a high resolution topographic map of europe in
> grass, using cgiar-cis srtm datas. The data is shipped in 5*5° tiles
> which I need to somhow paste together. On the grass mailing list, I
> was told to use gdalwarp in order to achieve this. Unfortunately, my
> current gdalwarp command (1.5) tells me the final TIF file will be
> over 4GB large, so that I need to get BigTIFF support.
>
> To do so, I downloaded the 1.6 source from the main gdal site, and
> launched ./configure with the following options:
>
> sudo CFLAGS="-O2 -march=k8 -pipe" ./configure --with-tiff=internal
> --with-hide-internal-symbols --with-jpeg=/usr --enable-largefile
> --with-libz=internal
>
> The command works fine, but the final message gives:
>
>  LIBTIFF support:           external (BigTIFF=no)
>  LIBGEOTIFF support:        internal
>
> Somehow there must be a config option I'm missing; and I would be very
> thankful if somebody in here could explain me what I'm doing wrong.
>
> regards,
>
> Felix Schalck
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Getting BigTiff support in Ubuntu 9.04 64bits

[Felix Schalck]

Hello,

I'm trying to create a high resolution topographic map of europe in
grass, using cgiar-cis srtm datas. The data is shipped in 5*5° tiles
which I need to somhow paste together. On the grass mailing list, I
was told to use gdalwarp in order to achieve this. Unfortunately, my
current gdalwarp command (1.5) tells me the final TIF file will be
over 4GB large, so that I need to get BigTIFF support.

To do so, I downloaded the 1.6 source from the main gdal site, and
launched ./configure with the following options:

sudo CFLAGS="-O2 -march=k8 -pipe" ./configure --with-tiff=internal
--with-hide-internal-symbols --with-jpeg=/usr --enable-largefile
--with-libz=internal

The command works fine, but the final message gives:

  LIBTIFF support:   external (BigTIFF=no)
  LIBGEOTIFF support:internal

Somehow there must be a config option I'm missing; and I would be very
thankful if somebody in here could explain me what I'm doing wrong.

regards,

Felix Schalck
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL WKT Raster cache: Problem, options to solve and doubts

[Frank Warmerdam]

2009/8/14 Jorge Arévalo :
>> Jorge,
>>
>> I would note that one round trip per block is not necessarily so terrible
>> if the block size is reasonably large (128x128 or larger for instance).
>>
>
> Yes, but don't you think it could be a good idea trying to avoid extra
> server rounds, in general?

Jorge,

It would be nice, but only as long as the complexity and risk is
appropriate to the benefit.

> Mmmm... I was studying the WCSDataset.cpp code and I thought that this
> might happen:
>
> - WCSDataset::IRasterIO calls GDALRasterBand::RasterIO
> - GDALRasterBand::RasterIO calls WCSRasterBand::IRasterIO
> - WCSRasterBand::IRasterIO calls WCSDataset::IRasterIO
> 
>
> But I'm sure I'm making a mistake, a misunderstanding, because WCS
> driver is a solid, well tested driver.

I believe the WCS code is careful about this, only calling into
WCSDataset::DirectRasterIO() for cases that can be handled
directly (checked via a Test method).  This care is exactly to
avoid a looping recursion.

> Anyway, my idea is only to call
> IReadBlock as the "last resource". For this reason, I'd like to take
> advantage of the GDAL Data Model with this simple algorithm:
>
> WKTRasterDataset::IRasterIO(...)
> {
>   // Fetch all raster rows covered by the area requested
>
>   If I have rows:
>        // Get the data of these rows and copy it into the pImage
> buffer (byte swapping if needed)
>        // Anything more? return CE_None? add blocks to rasterband
> cache? delegate in GDALDataset::RasterIO?
>  else:
>      // Delegate in GDALDataset::RasterIO? This will derive in
> WKTRasterRasterBand::IReadBlock or not??
> }
>
>
> WKTRasterRasterBand::IRasterIO(...)
> {
>  poDS->IRasterIO(current band);
> }
>
>

> What's the normal way in which a program calls GDAL IO system. Is
> usual to call RasterIO directly? maybe call IReadBlock directly? What
> should a driver expect? Should a driver expect something as "usual"?

Normally applications call RasterIO() regardless of the underlying
format.  Direct block reads are very rarely done.

> I suspect that depends of the driver format...
>
>> I would note that the optimized IRasterIO() implementation is
>> not really necessary for a successful project though it would
>> certainly be a crowning achievement for the summer.
>>
>
> Ok. I've quite a lot of code related with this concept (cache,
> IRasterIO...), but doesn't seem to works fine, for any reason. Maybe
> needs more discussion and/or thinking. I could put it off...

Well, please ensure that the more sophisticated machinery can
be easily disabled, getting back to a safely working block at
a time approach if needed.

Best regards,
-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL WKT Raster cache: Problem, options to solve and doubts

[Jorge Arévalo]

Hello,

2009/8/14 Mateusz Loskot :
> Jorge Arévalo wrote:
>> Hello,
>>
>> I've asked several concepts related with RasterIO-related methods in
>> Dataset and RasterBand. Thanks to your responses, I've a better
>> understanding of the GDAL drivers' I/O method. But I've a couple of
>> doubts I need to solve to finish the GSoC, although I'd like to
>> continue developing the driver after it.
>>
>> Problem: In basic GDAL WKT Raster driver, each row of a raster table
>> (one block, in regularly blocked rasters) means one server round. This
>> is slow, and "sub-optimal".
>
> Jorge,
>
> What you mean as one server round? Is it 1 SQL query per tile?

Yes, 1 sql select per IReadBlock call (1 tile in regularly blocked rasters).


>
>> How to solve it?: IReadBlock executes an spatial query to get all the
>> rows of a table that fits in a natural block, defined by nXOff, nYOff
>> and the values fetched by GetBlockSize. In many raster formats, one
>> natural block is a scanline, of size (nXRasterSize, 1).
>
> Yes, I found the natural block (=scalnline for many datasets)
> as inconvenient for querying regular blocking tables.
> Instead, I'd recommend to use tile (block) size as size of
> natural block, so you always query and process
> whole tile or number of tiles.
>

Yes, I use block size read from RASTER_COLUMNS to create the
RasterBands, as block size.


>> In WKT Raster
>> format, if we have a regularly blocked raster,  "natural" block size
>> will be equal to RASTER_COLUMNS-defined block size, and the query will
>> return one block.
>
> A! Perfect. This is what I've mentioned above.
> By the way, my thinking of WKT Raster in general may be a bit biased
> because I'm personally focused on regular blocking only.

Just like me... I've detected the points of the code where the
situation "non-regular blocking rasters" takes place, and, normally, I
raise an "Under development" error. This will be the bigger todo task
in this driver, I think.


>
>> To avoid one spatial query for each block petition,
>> we should "force" the driver to get all the blocks covering the area
>> requested in a IRasterIO call.
>
> Yes, this is the crux.
>
>> How to implement it? My approach is based on implementing
>> WKTRasterDataset::IRasterIO method (overriding GDALDataset::IRasterIO
>> method). This method executes a spatial query that returns all the
>> raster rows covering the area requested. Now, I have all the data of
>> an image region. If the requested region dimensions match the buffer
>> dimensions, I can copy all the pixels fetched in this way:
>>
>> pImage = 
>> {b1b1b1b1b1b1b1b1b1b1...|b2b2b2b2b2b2b2b2b2...|...|bnbnbnbnbnbnbnbnbn}
>>
>> Where bi are the bytes of the band i. A WKT Raster image has a
>> non-interleaved format, all the band are consecutive. Is it correct? I
>> mean, copy the data in pImage with this format.
>>
>> And if the region dimensions don't match the buffer dimensiones,
>> should I raise an error and finish or delegate in base
>> GDALDataset::IRasterIO implementation?
>
> What it means "region dimensions don't match the buffer"?
>
> I'd imagine something like this:
>
> 1. Query tiles that match the requested region (window) of raster
> coverage (table).
> 2. Fetch
> 3. Merge tiles into in-memory or on-disk file.


Yes, that's the point. With "region dimensions don't match the buffer"
I mean that nXSize, nYSize are different from nBufXSize, nBufYSize.
And maybe eBufType is different from band data type (for each band.
I'm talking about WKTRasterDataset::RasterIO method).

And what should be the criterion to choose between in-memory or
on-disk storage? If the memory buffer is bigger enough to store the
fetched data... it should be enough.

>
> The merge step could be based on RasterIO calls following this approach
>
> http://trac.osgeo.org/gdal/browser/trunk/gdal/swig/python/scripts/gdal_merge.py
>
> Also, the step 3 should be constrained by max limit of data pushed into
> memory, and fail if let's say a user queries 12000 tiles :-)
> and there is no overview level available that could be returned
> instead of such a huge amount of data.
> Perhaps it could be configurable.

Thanks! I've used gdal_merge, but I didn't study the code. I'm going
to study it just now :-)

>
>> Other part of my implementation is overriding
>> GDALRasterBand::IRasterIO method. My method will simply call the
>> WKTRasterDataset::IRasterIO method with only one band to read. Is it
>> correct?
>
> Looks OK.

Thanks for confirmation.


Best regards,
Jorge


>
> Best regards,
> --
> Mateusz Loskot, http://mateusz.loskot.net
> Charter Member of OSGeo, http://osgeo.org
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL WKT Raster cache: Problem, options to solve and doubts

[Jorge Arévalo]

Hello,

2009/8/14 Frank Warmerdam :
> 2009/8/14 Jorge Arévalo :
>> Problem: In basic GDAL WKT Raster driver, each row of a raster table
>> (one block, in regularly blocked rasters) means one server round. This
>> is slow, and "sub-optimal".
>
> Jorge,
>
> I would note that one round trip per block is not necessarily so terrible
> if the block size is reasonably large (128x128 or larger for instance).
>

Yes, but don't you think it could be a good idea trying to avoid extra
server rounds, in general?


>> How to solve it?: IReadBlock executes an spatial query to get all the
>> rows of a table that fits in a natural block, defined by nXOff, nYOff
>> and the values fetched by GetBlockSize. In many raster formats, one
>> natural block is a scanline, of size (nXRasterSize, 1). In WKT Raster
>> format, if we have a regularly blocked raster,  "natural" block size
>> will be equal to RASTER_COLUMNS-defined block size, and the query will
>> return one block. To avoid one spatial query for each block petition,
>> we should "force" the driver to get all the blocks covering the area
>> requested in a IRasterIO call.
>>
>> How to implement it? My approach is based on implementing
>> WKTRasterDataset::IRasterIO method (overriding GDALDataset::IRasterIO
>> method). This method executes a spatial query that returns all the
>> raster rows covering the area requested. Now, I have all the data of
>> an image region. If the requested region dimensions match the buffer
>> dimensions, I can copy all the pixels fetched in this way:
>>
>> pImage = 
>> {b1b1b1b1b1b1b1b1b1b1...|b2b2b2b2b2b2b2b2b2...|...|bnbnbnbnbnbnbnbnbn}
>>
>> Where bi are the bytes of the band i. A WKT Raster image has a
>> non-interleaved format, all the band are consecutive. Is it correct? I
>> mean, copy the data in pImage with this format.
>
> Some of the arguments to IRasterIO() are the values indicating
> how the imagery should be interleaved into the target buffer.
> You might want to check them and only implemented the direct
> copy if the interleaving matches what is convenient for you.

Ok, nPixelSpace and nLineSpace, right. Stupid me. -1


>
>> And if the region dimensions don't match the buffer dimensiones,
>> should I raise an error and finish or delegate in base
>> GDALDataset::IRasterIO implementation?
>
> Definately do not raise an error.  If for any reason it is not
> convenient to process a request in an optimized fashion then
> just call the underlying IRasterIO() method (on whatever your
> base class is, possible GDALDataset).

Ok, my code works in that way just now. Thanks


>
>> Another question: After copying the data from fetched rows in pImage
>> buffer, should I do anything more?
>
> You need to ensure it is in local machine byte order if it is not
> GDT_Byte.
>

Yes, I check this. I forgot to mention.

>> Other part of my implementation is overriding
>> GDALRasterBand::IRasterIO method. My method will simply call the
>> WKTRasterDataset::IRasterIO method with only one band to read. Is it
>> correct?
>
> This is acceptable I think.  My only concern is that there may be
> situations in which WKTRasterDataset::IRasterIO() will not
> process the request in an optimized fashion and will fallback to
> GDALDataset::IRasterIO() which in turn might call IRasterIO
> on the band again.  This may become obvious in use. :-)
>

Mmmm... I was studying the WCSDataset.cpp code and I thought that this
might happen:

- WCSDataset::IRasterIO calls GDALRasterBand::RasterIO
- GDALRasterBand::RasterIO calls WCSRasterBand::IRasterIO
- WCSRasterBand::IRasterIO calls WCSDataset::IRasterIO


But I'm sure I'm making a mistake, a misunderstanding, because WCS
driver is a solid, well tested driver. Anyway, my idea is only to call
IReadBlock as the "last resource". For this reason, I'd like to take
advantage of the GDAL Data Model with this simple algorithm:

WKTRasterDataset::IRasterIO(...)
{
   // Fetch all raster rows covered by the area requested

   If I have rows:
// Get the data of these rows and copy it into the pImage
buffer (byte swapping if needed)
// Anything more? return CE_None? add blocks to rasterband
cache? delegate in GDALDataset::RasterIO?
  else:
  // Delegate in GDALDataset::RasterIO? This will derive in
WKTRasterRasterBand::IReadBlock or not??
}


WKTRasterRasterBand::IRasterIO(...)
{
  poDS->IRasterIO(current band);
}


WKTRasterRasterBand::IReadBlock(...)
{
 // Fetch all raster rows that the block covers (in regularly blocked
rasters, 1 row (= 1 tile))

If only 1 block:
// copy data in pImage buffer
// return
else:
   // non-regularly blocked rasters. Raise an "under development"
error, just now.
}

But I don't know if this approach works as I want. IReadBlock works
fine, but I have problems with the rest of the system (IRasterIO
implementations), and I don't know what is the best approach on this
driver :-(

What's the normal way in which a program calls GDAL IO system. Is
usual to call RasterIO directly? mayb

Re: [gdal-dev] GDAL WKT Raster cache: Problem, options to solve and doubts

[Mateusz Loskot]

Jorge Arévalo wrote:
> Hello,
> 
> I've asked several concepts related with RasterIO-related methods in
> Dataset and RasterBand. Thanks to your responses, I've a better
> understanding of the GDAL drivers' I/O method. But I've a couple of
> doubts I need to solve to finish the GSoC, although I'd like to
> continue developing the driver after it.
> 
> Problem: In basic GDAL WKT Raster driver, each row of a raster table
> (one block, in regularly blocked rasters) means one server round. This
> is slow, and "sub-optimal".

Jorge,

What you mean as one server round? Is it 1 SQL query per tile?

> How to solve it?: IReadBlock executes an spatial query to get all the
> rows of a table that fits in a natural block, defined by nXOff, nYOff
> and the values fetched by GetBlockSize. In many raster formats, one
> natural block is a scanline, of size (nXRasterSize, 1).

Yes, I found the natural block (=scalnline for many datasets)
as inconvenient for querying regular blocking tables.
Instead, I'd recommend to use tile (block) size as size of
natural block, so you always query and process
whole tile or number of tiles.

> In WKT Raster 
> format, if we have a regularly blocked raster,  "natural" block size
> will be equal to RASTER_COLUMNS-defined block size, and the query will
> return one block.

A! Perfect. This is what I've mentioned above.
By the way, my thinking of WKT Raster in general may be a bit biased
because I'm personally focused on regular blocking only.

> To avoid one spatial query for each block petition,
> we should "force" the driver to get all the blocks covering the area
> requested in a IRasterIO call.

Yes, this is the crux.

> How to implement it? My approach is based on implementing
> WKTRasterDataset::IRasterIO method (overriding GDALDataset::IRasterIO
> method). This method executes a spatial query that returns all the
> raster rows covering the area requested. Now, I have all the data of
> an image region. If the requested region dimensions match the buffer
> dimensions, I can copy all the pixels fetched in this way:
> 
> pImage = 
> {b1b1b1b1b1b1b1b1b1b1...|b2b2b2b2b2b2b2b2b2...|...|bnbnbnbnbnbnbnbnbn}
> 
> Where bi are the bytes of the band i. A WKT Raster image has a
> non-interleaved format, all the band are consecutive. Is it correct? I
> mean, copy the data in pImage with this format.
> 
> And if the region dimensions don't match the buffer dimensiones,
> should I raise an error and finish or delegate in base
> GDALDataset::IRasterIO implementation?

What it means "region dimensions don't match the buffer"?

I'd imagine something like this:

1. Query tiles that match the requested region (window) of raster
coverage (table).
2. Fetch
3. Merge tiles into in-memory or on-disk file.

The merge step could be based on RasterIO calls following this approach

http://trac.osgeo.org/gdal/browser/trunk/gdal/swig/python/scripts/gdal_merge.py

Also, the step 3 should be constrained by max limit of data pushed into
memory, and fail if let's say a user queries 12000 tiles :-)
and there is no overview level available that could be returned
instead of such a huge amount of data.
Perhaps it could be configurable.

> Other part of my implementation is overriding
> GDALRasterBand::IRasterIO method. My method will simply call the
> WKTRasterDataset::IRasterIO method with only one band to read. Is it
> correct?

Looks OK.

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

RE: [gdal-dev] GeoEye-1 NITF - Warning 1: Image with 11 bits per sample will not be interpreted properly

[Gong, Shawn (Contractor)]

Hi Frank and list,

I rebuilt Gdal on a 64-bit Linux server, adding libecwj2, and it
ingested GeoEye-1 NITF just fine (a bit slow though).
So you are right that this warning should not really apply. 

My question is then why it failed on Windows XP, at libecwj2's
ncsjpcbuffer.cpp CNCSJPCBuffer(). The Jpeg2000 file is 463MB, after
decompression it is 2.375 GB. Do I need to specify "large file" when
building Gdal on Windows?


thanks,
Shawn



-Original Message-
From: gdal-dev-boun...@lists.osgeo.org
[mailto:gdal-dev-boun...@lists.osgeo.org] On Behalf Of Frank Warmerdam
Sent: Wednesday, August 12, 2009 10:12 AM
To: Gong, Shawn (Contractor)
Cc: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] GeoEye-1 NITF - Warning 1: Image with 11 bits
per sample will not be interpreted properly

On Tue, Aug 11, 2009 at 12:17 PM, Gong, Shawn
(Contractor) wrote:
> hi list,
>
> I found help clues to build libecwj2 on Windows.

Shawn,

Congrats on working though this issue!  Perhaps you would like to add
some notes to:

  http://trac.osgeo.org/gdal/wiki/ECW

> However when I tried to open GeoEye-1 NITF, I got "Warning 1: Image
with 11
> bits per sample will not be interpreted properly".

I thought there were changes in trunk to avoid this warning when the
image is jpeg2000 and being handled by a suitable codec.  However,
I think I only tested this specific issue when using JP2KAK for the
jpeg2000 support.  I'm not sure if there are problems with the ecw sdk
and such files or not.  However, the note about being badly corrupted
should not really apply.

> There are two sub-datasets in NITF (please see the gdalinfo at the
end). If
> I open the 2nd sub-dataset, the data show only 73 pixels and 220
lines;
> digital numbers are either 0 or 255.

Hmm, I don't know why it would be particularly small in size.
I'm also surprised that the latest windows fwtools has problems.
Perhaps you could file a ticket on the issue and I could look into
it a bit next week (especially since problems can be reproduced
with FWTools which I think uses JP2KAK for this purpose).

Hmm, on second thought, I believe I removed JP2KAK from
FWTools for licensing reasons - so perhaps it is using the
ecw jpeg2000 driver too.

Best regards,
-- 
---+
--
I set the clouds in motion - turn up   | Frank Warmerdam,
warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Expanding a palette image to 3 bands

[Frank Warmerdam]

On Fri, Aug 14, 2009 at 9:21 AM, Henneke, Amanda
M wrote:
> Hello all-
>
> I have a need to expand a 1 band palette tif file into a 3 band (rgb) tif 
> file.  I have found that gdal_translate -expand rgb will do this from the 
> command line, but I was wondering if there was some way to do this within C# 
> using the GDAL bindings?  I notice there is a ColorTable object and 
> PaletteInterpretation object, but I haven't figured out yet (or seen anything 
> online) about how to use these to expand the image.

Amanda,

You could skim the gdal_translate code to see how -expand rgb
works, but unfortunately it depends on access to VRT classes
that are not directly callable from C# or other swig bindings.  However,
you can compose the corresponding VRT XML directly in C# and
then "open" that VRT XML as a dataset that will have the expanded
bands.  This VRT dataset can then be passed to a CreateCopy() to
write out the expanded result.

I have attached the results of "gdal_translate -of VRT -expand rgb" on
a dataset as an example of the corresponding XML.

Best regards,
-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent


out.vrt
Description: Binary data
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL WKT Raster cache: Problem, options to solve and doubts

[Frank Warmerdam]

2009/8/14 Jorge Arévalo :
> Problem: In basic GDAL WKT Raster driver, each row of a raster table
> (one block, in regularly blocked rasters) means one server round. This
> is slow, and "sub-optimal".

Jorge,

I would note that one round trip per block is not necessarily so terrible
if the block size is reasonably large (128x128 or larger for instance).

> How to solve it?: IReadBlock executes an spatial query to get all the
> rows of a table that fits in a natural block, defined by nXOff, nYOff
> and the values fetched by GetBlockSize. In many raster formats, one
> natural block is a scanline, of size (nXRasterSize, 1). In WKT Raster
> format, if we have a regularly blocked raster,  "natural" block size
> will be equal to RASTER_COLUMNS-defined block size, and the query will
> return one block. To avoid one spatial query for each block petition,
> we should "force" the driver to get all the blocks covering the area
> requested in a IRasterIO call.
>
> How to implement it? My approach is based on implementing
> WKTRasterDataset::IRasterIO method (overriding GDALDataset::IRasterIO
> method). This method executes a spatial query that returns all the
> raster rows covering the area requested. Now, I have all the data of
> an image region. If the requested region dimensions match the buffer
> dimensions, I can copy all the pixels fetched in this way:
>
> pImage = 
> {b1b1b1b1b1b1b1b1b1b1...|b2b2b2b2b2b2b2b2b2...|...|bnbnbnbnbnbnbnbnbn}
>
> Where bi are the bytes of the band i. A WKT Raster image has a
> non-interleaved format, all the band are consecutive. Is it correct? I
> mean, copy the data in pImage with this format.

Some of the arguments to IRasterIO() are the values indicating
how the imagery should be interleaved into the target buffer.
You might want to check them and only implemented the direct
copy if the interleaving matches what is convenient for you.

> And if the region dimensions don't match the buffer dimensiones,
> should I raise an error and finish or delegate in base
> GDALDataset::IRasterIO implementation?

Definately do not raise an error.  If for any reason it is not
convenient to process a request in an optimized fashion then
just call the underlying IRasterIO() method (on whatever your
base class is, possible GDALDataset).

> Another question: After copying the data from fetched rows in pImage
> buffer, should I do anything more?

You need to ensure it is in local machine byte order if it is not
GDT_Byte.

> Other part of my implementation is overriding
> GDALRasterBand::IRasterIO method. My method will simply call the
> WKTRasterDataset::IRasterIO method with only one band to read. Is it
> correct?

This is acceptable I think.  My only concern is that there may be
situations in which WKTRasterDataset::IRasterIO() will not
process the request in an optimized fashion and will fallback to
GDALDataset::IRasterIO() which in turn might call IRasterIO
on the band again.  This may become obvious in use. :-)

I would note that the optimized IRasterIO() implementation is
not really necessary for a successful project though it would
certainly be a crowning achievement for the summer.

Best regards,
-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Expanding a palette image to 3 bands

[Henneke, Amanda M]

Hello all-

I have a need to expand a 1 band palette tif file into a 3 band (rgb) tif file. 
 I have found that gdal_translate -expand rgb will do this from the command 
line, but I was wondering if there was some way to do this within C# using the 
GDAL bindings?  I notice there is a ColorTable object and PaletteInterpretation 
object, but I haven't figured out yet (or seen anything online) about how to 
use these to expand the image.

Can anyone help?

Thanks,
Amanda

Amanda M. Henneke


 
When One tugs at a single thing in Nature, he finds it attached to the rest of 
the world. -John Muir
  Please consider the environment before printing this e-mail.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL WKT Raster cache: Problem, options to solve and doubts

[Jorge Arévalo]

Hello,

I've asked several concepts related with RasterIO-related methods in
Dataset and RasterBand. Thanks to your responses, I've a better
understanding of the GDAL drivers' I/O method. But I've a couple of
doubts I need to solve to finish the GSoC, although I'd like to
continue developing the driver after it.

Problem: In basic GDAL WKT Raster driver, each row of a raster table
(one block, in regularly blocked rasters) means one server round. This
is slow, and "sub-optimal".

How to solve it?: IReadBlock executes an spatial query to get all the
rows of a table that fits in a natural block, defined by nXOff, nYOff
and the values fetched by GetBlockSize. In many raster formats, one
natural block is a scanline, of size (nXRasterSize, 1). In WKT Raster
format, if we have a regularly blocked raster,  "natural" block size
will be equal to RASTER_COLUMNS-defined block size, and the query will
return one block. To avoid one spatial query for each block petition,
we should "force" the driver to get all the blocks covering the area
requested in a IRasterIO call.

How to implement it? My approach is based on implementing
WKTRasterDataset::IRasterIO method (overriding GDALDataset::IRasterIO
method). This method executes a spatial query that returns all the
raster rows covering the area requested. Now, I have all the data of
an image region. If the requested region dimensions match the buffer
dimensions, I can copy all the pixels fetched in this way:

pImage = {b1b1b1b1b1b1b1b1b1b1...|b2b2b2b2b2b2b2b2b2...|...|bnbnbnbnbnbnbnbnbn}

Where bi are the bytes of the band i. A WKT Raster image has a
non-interleaved format, all the band are consecutive. Is it correct? I
mean, copy the data in pImage with this format.

And if the region dimensions don't match the buffer dimensiones,
should I raise an error and finish or delegate in base
GDALDataset::IRasterIO implementation?

Another question: After copying the data from fetched rows in pImage
buffer, should I do anything more?

Other part of my implementation is overriding
GDALRasterBand::IRasterIO method. My method will simply call the
WKTRasterDataset::IRasterIO method with only one band to read. Is it
correct?

Thanks in advance.

Best regards,
Jorge
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] regarding GDALDriver::Create

[Frank Warmerdam]

On Fri, Aug 14, 2009 at 2:59 AM, Srikanth wrote:
> Dear All,
>
> I am reading and writing raster images using GDAL. Below is the code which I
> am using for writing a raster.
> At times the code is working fine. But many a time , it is getting crashed
> in the GetRasterBand. I have debugged the code several times and the
> variables passed seems to be perfect. Any help in this is regard would be
> appreciated.
...
>    poDataset=NULL;
>    poDataset = poDriver->Create(filename , width , height , numOfbands
> ,(GDALDataType)datatype,NULL);
>
>    for(int i=0 ; i < numOfbands ; i++)
>      {
>              poBand[i] = poDataset->GetRasterBand(i+1);
>       }

Hi,

I didn't see any showstopper problems in your code.  But
the above code is not safe if the dataset creation fails.
If the dataset creation fails, a null pointer will be returned
and GetRasterBand() will fail - possibly crashing.

Best regards,
-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev