Re: [gdal-dev] Motions: Promote 1.11.5 RC 1, 2.0.3 RC2 and and 2.1.1 RC2for release

[Tamas Szekeres]

+1 for all

Tamas



Feladó: Even Rouault___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL raster processing library?

[Even Rouault]

Le mardi 12 juillet 2016 16:39:34, James Ramm a écrit :
> Hi Even,
> 
> The difference I see with pixel functions is that, as far as I understand,
> the pixel function is applied per pixel, so there is no possibility of e.g.
> the pixel buffer when have the function apply to 'blocks'.
> I may be way off, but many of the algorithms we deal with require some kind
> of neighbourhood search - a polygonise algorithm or flow direction
> algorithm being good examples.
> I dont think VRT pixel functions allow this?

True, they don't. Although there's the KernelFilteredSource that is close

> 
> So in that sense I'd see a VRT being 'just' another potential input data
> source.
> 
> Perhaps VRT pixel functions could be expanded to also allow 'window'
> functions?

Would make sense

> 
> A downside is it requires creating a VRT even when you only want to apply a
> such a function to a single dataset. Small effort, but still a bit more
> than throwing in any GDALDataset to be processed.

You could have an helper function to create the VRT dataset from the input 
dataset, the processing function and its parameters. A bit similarly to what 
Julien is doing in 
https://github.com/OSGeo/gdal/pull/142/files#diff-0c9bf560508edf2453cd2f776d72f905R121

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL raster processing library?

[Peter Halls]

James,

  in reality, are you not requesting an implementation of Tomlin's
'Grid Algebra' in GDAL?  That defines the whole range of functions from
whole raster to pixel and has the distinct advantage of being both
published and extremely well known because of other implementations ...
which also provide ready-made reference bases for the GDAL implementors ...

Best wishes,
Peter

On 12 July 2016 at 15:39, James Ramm  wrote:

> Hi Even,
>
> The difference I see with pixel functions is that, as far as I understand,
> the pixel function is applied per pixel, so there is no possibility of e.g.
> the pixel buffer when have the function apply to 'blocks'.
> I may be way off, but many of the algorithms we deal with require some
> kind of neighbourhood search - a polygonise algorithm or flow direction
> algorithm being good examples.
> I dont think VRT pixel functions allow this?
>
> So in that sense I'd see a VRT being 'just' another potential input data
> source.
>
> Perhaps VRT pixel functions could be expanded to also allow 'window'
> functions?
>
> A downside is it requires creating a VRT even when you only want to apply
> a such a function to a single dataset. Small effort, but still a bit more
> than throwing in any GDALDataset to be processed.
>
> I see the overlap with raster algebra, although yes technically very
> different.
>
>
>
> On 12 July 2016 at 14:55, Even Rouault  wrote:
>
>> James,
>>
>> There's some intersection with Ari's proposal :
>> https://trac.osgeo.org/gdal/wiki/rfc62_raster_algebra . At least
>> regarding the
>> overall purposes, since technically this is quite different.
>>
>> Actually what you propose is very close to the existing VRT pixel
>> functions of
>> derived bands. See "Using Derived Bands" in
>> http://www.gdal.org/gdal_vrttut.html . In the last days, we've merged
>> Antonio's work regarding a predefined set of pixel functions.
>> Perhaps some extension to allow passing user parameters to the pixel func
>> could be useful. It is possible to use pixel functions from Python as
>> shown in
>>
>> https://github.com/OSGeo/gdal/blob/trunk/autotest/gcore/testnonboundtoswig.py#L303
>> although this is a bit ugly as it uses ctypes and not SWIG. But should be
>> possible through SWIG by introducing proper types similarly to what is
>> done
>> for the progress functions or error handler functions.
>>
>> Even
>>
>> Le mardi 12 juillet 2016 14:40:27, jramm a écrit :
>> > I wonder if there is a use case/interest in a small library within GDAL
>> to
>> > facilitate generic raster processing.
>> >
>> > My idea would be to have a user-extensible framework to run processing
>> > functions on whole rasters, with a growing library of common-such
>> functions
>> > within GDAL.
>> >
>> > Something along the lines of this:
>> >
>> > /***/
>> > typedef CPLErr (*GDALRasterProcessFn)(double *padfInArray, double
>> > *padfOutArray,
>> > int nWindowXSize, int nWindowYSize, void *pData, double
>> > *pdfNoDataValue);
>> > /**
>> > * \brief Definition of a raster processing function.
>> > *
>> > * A GDALRasterProcessFn accepts an array of data as input, applies
>> custom
>> > logic and writes the output to padfOutArray.
>> > * Such a function can be passed to GDALRunRasterProcess to apply custom
>> > processing to a GDALDataset in chunks and create
>> > * a new GDALDataset.
>> > *
>> > * @param padfInArray The input array of data.
>> > *
>> > * @param padfOutArray The output array of data. On first call (via
>> > GDALRunRasterProcess) this will be an empty, initialised array,
>> > *which should be populated with the result of calculations on
>> > padfInArray. In subsequent calls it will contain the result of the
>> > *previous window.
>> > *
>> > * @param nWindowXSize the actual x size (width) of the read window.
>> > *
>> > * @param nWindowYSize the actual y size (height) of the read window. The
>> > length of padfInArray == padfOutArray == nWindowXSize * nWindowYSize
>> > *
>> > * @param pData Process-specific data. This data is passed straight
>> through
>> > to the GDALRasterProcessFn and may contain e.g user defined parameters.
>> > * The GDALRasterProcessFn definition would define the
>> structure/type of
>> > such data.
>> > *
>> > * @param pdfNoDataValue The no data value of the dataset
>> > */
>> >
>> > CPLErr GDALRunRasterProcess(GDALRasterProcessFn processFn, GDALDataset
>> > *poSrcDataset,
>> > GDALDataset *poDstDataset, void *pData, int
>> > *pnWindowXSize,
>> > int *pnWindowYSize, int *pnPixelBuffer);
>> > /**
>> > * \brief Apply a raster processing function to each sub-window of a
>> raster.
>> > *
>> > * The input raster dataset is read in chunks of nWindowXSize *
>> nWindowYSize
>> > and each chunk is passed to the processing
>> > * function. The output array from the function is written to the

Re: [gdal-dev] GDAL raster processing library?

[James Ramm]

Hi Even,

The difference I see with pixel functions is that, as far as I understand,
the pixel function is applied per pixel, so there is no possibility of e.g.
the pixel buffer when have the function apply to 'blocks'.
I may be way off, but many of the algorithms we deal with require some kind
of neighbourhood search - a polygonise algorithm or flow direction
algorithm being good examples.
I dont think VRT pixel functions allow this?

So in that sense I'd see a VRT being 'just' another potential input data
source.

Perhaps VRT pixel functions could be expanded to also allow 'window'
functions?

A downside is it requires creating a VRT even when you only want to apply a
such a function to a single dataset. Small effort, but still a bit more
than throwing in any GDALDataset to be processed.

I see the overlap with raster algebra, although yes technically very
different.



On 12 July 2016 at 14:55, Even Rouault  wrote:

> James,
>
> There's some intersection with Ari's proposal :
> https://trac.osgeo.org/gdal/wiki/rfc62_raster_algebra . At least
> regarding the
> overall purposes, since technically this is quite different.
>
> Actually what you propose is very close to the existing VRT pixel
> functions of
> derived bands. See "Using Derived Bands" in
> http://www.gdal.org/gdal_vrttut.html . In the last days, we've merged
> Antonio's work regarding a predefined set of pixel functions.
> Perhaps some extension to allow passing user parameters to the pixel func
> could be useful. It is possible to use pixel functions from Python as
> shown in
>
> https://github.com/OSGeo/gdal/blob/trunk/autotest/gcore/testnonboundtoswig.py#L303
> although this is a bit ugly as it uses ctypes and not SWIG. But should be
> possible through SWIG by introducing proper types similarly to what is done
> for the progress functions or error handler functions.
>
> Even
>
> Le mardi 12 juillet 2016 14:40:27, jramm a écrit :
> > I wonder if there is a use case/interest in a small library within GDAL
> to
> > facilitate generic raster processing.
> >
> > My idea would be to have a user-extensible framework to run processing
> > functions on whole rasters, with a growing library of common-such
> functions
> > within GDAL.
> >
> > Something along the lines of this:
> >
> > /***/
> > typedef CPLErr (*GDALRasterProcessFn)(double *padfInArray, double
> > *padfOutArray,
> > int nWindowXSize, int nWindowYSize, void *pData, double
> > *pdfNoDataValue);
> > /**
> > * \brief Definition of a raster processing function.
> > *
> > * A GDALRasterProcessFn accepts an array of data as input, applies custom
> > logic and writes the output to padfOutArray.
> > * Such a function can be passed to GDALRunRasterProcess to apply custom
> > processing to a GDALDataset in chunks and create
> > * a new GDALDataset.
> > *
> > * @param padfInArray The input array of data.
> > *
> > * @param padfOutArray The output array of data. On first call (via
> > GDALRunRasterProcess) this will be an empty, initialised array,
> > *which should be populated with the result of calculations on
> > padfInArray. In subsequent calls it will contain the result of the
> > *previous window.
> > *
> > * @param nWindowXSize the actual x size (width) of the read window.
> > *
> > * @param nWindowYSize the actual y size (height) of the read window. The
> > length of padfInArray == padfOutArray == nWindowXSize * nWindowYSize
> > *
> > * @param pData Process-specific data. This data is passed straight
> through
> > to the GDALRasterProcessFn and may contain e.g user defined parameters.
> > * The GDALRasterProcessFn definition would define the structure/type
> of
> > such data.
> > *
> > * @param pdfNoDataValue The no data value of the dataset
> > */
> >
> > CPLErr GDALRunRasterProcess(GDALRasterProcessFn processFn, GDALDataset
> > *poSrcDataset,
> > GDALDataset *poDstDataset, void *pData, int
> > *pnWindowXSize,
> > int *pnWindowYSize, int *pnPixelBuffer);
> > /**
> > * \brief Apply a raster processing function to each sub-window of a
> raster.
> > *
> > * The input raster dataset is read in chunks of nWindowXSize *
> nWindowYSize
> > and each chunk is passed to the processing
> > * function. The output array from the function is written to the
> > destination dataset.
> > * An optional 'pixel buffer' can be specified to allow overlaps between
> > successive windows. This is useful for
> > * some algorithms, e.g. blob extraction, watershed/stream flow analysis,
> > convolution etc.
> > * Process specific data can be passed (e.g. configuration parameters).
> This
> > data is simply passed straight through to the processing
> > * function on each call.
> > *
> > * @param processFn A GDALRasterProcessFn to apply to each sub window of
> the
> > raster.
> > *
> > * @param poSrcDataset The source raster dataset from which pixel 

Re: [gdal-dev] Motions: Promote 1.11.5 RC 1, 2.0.3 RC2 and and 2.1.1 RC2 for release

[Daniel Morissette]

+0 on all three. I didn't have a chance to fully test, but I'm very 
supportive.


Daniel


On 2016-07-08 6:02 AM, Even Rouault wrote:

Hi,

Motions under vote:
Motion 1: GDAL/OGR 1.11.5 RC1 is promoted to be the official 1.11.5 final
release.

Motion 2: GDAL/OGR 2.0.3 RC2 is promoted to be the official 2.0.3 final release.

To replace the retracted motion 3:
Motion 3bis: GDAL/OGR 2.1.1 RC2 is promoted to be the official 2.1.1 final
release.

--

My votes :
Motion 1: + 1
Motion 2: + 1
Motion 3bis: + 1

Best regards,

Even




--
Daniel Morissette
http://www.mapgears.com/
T: +1 418-696-5056 #201

http://evouala.com/ - Location Intelligence Made Easy
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL raster processing library?

[Even Rouault]

James,

There's some intersection with Ari's proposal :
https://trac.osgeo.org/gdal/wiki/rfc62_raster_algebra . At least regarding the 
overall purposes, since technically this is quite different.

Actually what you propose is very close to the existing VRT pixel functions of 
derived bands. See "Using Derived Bands" in 
http://www.gdal.org/gdal_vrttut.html . In the last days, we've merged 
Antonio's work regarding a predefined set of pixel functions.
Perhaps some extension to allow passing user parameters to the pixel func 
could be useful. It is possible to use pixel functions from Python as shown in 
https://github.com/OSGeo/gdal/blob/trunk/autotest/gcore/testnonboundtoswig.py#L303
 
although this is a bit ugly as it uses ctypes and not SWIG. But should be 
possible through SWIG by introducing proper types similarly to what is done 
for the progress functions or error handler functions.

Even

Le mardi 12 juillet 2016 14:40:27, jramm a écrit :
> I wonder if there is a use case/interest in a small library within GDAL to
> facilitate generic raster processing.
> 
> My idea would be to have a user-extensible framework to run processing
> functions on whole rasters, with a growing library of common-such functions
> within GDAL.
> 
> Something along the lines of this:
> 
> /***/
> typedef CPLErr (*GDALRasterProcessFn)(double *padfInArray, double
> *padfOutArray,
> int nWindowXSize, int nWindowYSize, void *pData, double
> *pdfNoDataValue);
> /**
> * \brief Definition of a raster processing function.
> *
> * A GDALRasterProcessFn accepts an array of data as input, applies custom
> logic and writes the output to padfOutArray.
> * Such a function can be passed to GDALRunRasterProcess to apply custom
> processing to a GDALDataset in chunks and create
> * a new GDALDataset.
> *
> * @param padfInArray The input array of data.
> *
> * @param padfOutArray The output array of data. On first call (via
> GDALRunRasterProcess) this will be an empty, initialised array,
> *which should be populated with the result of calculations on
> padfInArray. In subsequent calls it will contain the result of the
> *previous window.
> *
> * @param nWindowXSize the actual x size (width) of the read window.
> *
> * @param nWindowYSize the actual y size (height) of the read window. The
> length of padfInArray == padfOutArray == nWindowXSize * nWindowYSize
> *
> * @param pData Process-specific data. This data is passed straight through
> to the GDALRasterProcessFn and may contain e.g user defined parameters.
> * The GDALRasterProcessFn definition would define the structure/type of
> such data.
> *
> * @param pdfNoDataValue The no data value of the dataset
> */
> 
> CPLErr GDALRunRasterProcess(GDALRasterProcessFn processFn, GDALDataset
> *poSrcDataset,
> GDALDataset *poDstDataset, void *pData, int
> *pnWindowXSize,
> int *pnWindowYSize, int *pnPixelBuffer);
> /**
> * \brief Apply a raster processing function to each sub-window of a raster.
> *
> * The input raster dataset is read in chunks of nWindowXSize * nWindowYSize
> and each chunk is passed to the processing
> * function. The output array from the function is written to the
> destination dataset.
> * An optional 'pixel buffer' can be specified to allow overlaps between
> successive windows. This is useful for
> * some algorithms, e.g. blob extraction, watershed/stream flow analysis,
> convolution etc.
> * Process specific data can be passed (e.g. configuration parameters). This
> data is simply passed straight through to the processing
> * function on each call.
> *
> * @param processFn A GDALRasterProcessFn to apply to each sub window of the
> raster.
> *
> * @param poSrcDataset The source raster dataset from which pixel values are
> read
> *
> * @param poDstDataset The destination raster dataset to which pixel values
> are written. Must support RasterIO in write mode.
> *
> * @param pData Process-specific data. This is passed straight through to
> the GDALRasterProcessFn on each call.
> *
> * @param pnWindowXSize The desired width of each read window. If NULL it
> defaults to the 'natural' block size of the raster
> *
> * @param pnWindowYSize The desired height of each read window. If NULL it
> defaults to the 'natural' block size.
> *
> * @param pnPixelBuffer A pixel buffer to apply to the read window. The read
> window is expanded by pnPixelBuffer pixels in all directions such that
> *each window overlaps by pnPixelBuffer pixels. Ignored if NULL or 0
> *
> * @return a CPLErr enum indicating whether the function succesfully ran.
> */
> 
> 
> CPLErr GDALRunRasterProcesses(GDALRasterProcessFn *paProcessFn, int
> nProcesses, GDALDataset *poSrcDataset,
> GDALDataset *poDstDataset, void **paData, int
> *pnWindowXSize,
> int *pnWindowYSize, int *pnPixelBuffer);
> 
> /**
> * \brief Apply multiple 

Re: [gdal-dev] Motions: Promote 1.11.5 RC 1, 2.0.3 RC2 and and 2.1.1 RC2 for release

[Howard Butler]

> On Fri, Jul 8, 2016 at 3:02 AM, Even Rouault  
> wrote:
> Hi,
> 
> Motions under vote:
> Motion 1: GDAL/OGR 1.11.5 RC1 is promoted to be the official 1.11.5 final
> release.

+1

> Motion 2: GDAL/OGR 2.0.3 RC2 is promoted to be the official 2.0.3 final 
> release.

+1

> To replace the retracted motion 3:
> Motion 3bis: GDAL/OGR 2.1.1 RC2 is promoted to be the official 2.1.1 final
> release.

+1

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motions: Promote 1.11.5 RC 1, 2.0.3 RC2 and and 2.1.1 RC2 for release

[Rahkonen Jukka (MML)]

+1 for all three.

-Jukka Rahkonen-


Lähettäjä: Even Rouault
Lähetetty: ‎12.‎7.‎2016 12:15
Vastaanottaja: Rahkonen Jukka (MML)
Aihe: Fwd: [gdal-dev] Motions: Promote 1.11.5 RC 1, 2.0.3 RC2 and and 2.1.1 RC2 
for release


--  Message transmis  --

Sujet : Fwd: [gdal-dev] Motions: Promote 1.11.5 RC 1, 2.0.3 RC2 and and 2.1.1
RC2 for release
Date : mardi 12 juillet 2016, 11:18:07
De : Even Rouault 
À : Frank Warmerdam , Daniel Morissette
, Tamas Szekeres , Howard
Butler , Andrey Kiselev , Jukka Rahkonen


Dear PSC members,

I'd appreciate a few more votes on those motions, especially regarding
2.0.3RC2 that hasn't yet reached the minimum +2 threshold.

Best regards,

Even

--  Message transmis  --

Sujet : [gdal-dev] Motions: Promote 1.11.5 RC 1, 2.0.3 RC2 and and 2.1.1 RC2
for release
Date : vendredi 08 juillet 2016, 12:02:10
De : Even Rouault 
À : gdal-dev@lists.osgeo.org

Hi,

Motions under vote:
Motion 1: GDAL/OGR 1.11.5 RC1 is promoted to be the official 1.11.5 final
release.

Motion 2: GDAL/OGR 2.0.3 RC2 is promoted to be the official 2.0.3 final release.

To replace the retracted motion 3:
Motion 3bis: GDAL/OGR 2.1.1 RC2 is promoted to be the official 2.1.1 final
release.

--

My votes :
Motion 1: + 1
Motion 2: + 1
Motion 3bis: + 1

Best regards,

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
---
--
Spatialys - Geospatial professional services
http://www.spatialys.com

---
--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] unable to create jpeg200 files with self compiled gdal and openjpeg

[Even Rouault]

André,

> 
> This code works for geotiff, but not for jpeg2000. There I'm getting the
> error: ERROR 6: GDALDriver::Create() ... no create method implemented
> for this format.

You cannot directly use Create() for any of the JPEG2000 drivers as Create() 
assumes random writing capabilities which the JPEG2000 format doesn't offer. So 
the solution is to create a temporary dataset (for example with the MEM driver 
if it fits into RAM, or with an intermediate GeoTIFF otherwise) and then 
CreateCopy() it to JPEG2000

Something like:

tmp_ds = 
gdal.GetDriverByName('MEM').Create('',img.shape[0],img.shape[1],1,gdal.GDT_UInt16)
 
# here I 
rb = tmp_ds.GetRasterBand(1)
rb.WriteArray(img)
dr.CreateCopy(fn, tmp_ds)
tmp_ds = None

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] unable to create jpeg200 files with self compiled gdal and openjpeg

[Andre Hollstein]

Dear gdal experts,

I'm compiling gdal and a whole list of other libraries myself. Currently 
I'm using gdal-2.1.0 and openjepeg v2.1.1 and use the libs from python:


import gdal
import numpy as np
gdal.AllRegister()
#driver = 'JP2KAK'
driver = "JP2OpenJPEG"
#driver = "GTiff"
#driver = "JPEG2000" # jasper

fn = "./img_buf.%s" % 
{"GTiff":"tiff",'JP2KAK':"jp2","JP2OpenJPEG":"jp2","JPEG2000":"jp2"}[driver]


sh = (1000,1000)
img = np.ones(sh,dtype=np.int16)
dr = gdal.GetDriverByName(driver)
ds = dr.Create(fn,img.shape[0],img.shape[1],1,gdal.GDT_UInt16) # here I 
get: ERROR 6: GDALDriver::Create() ... no create method implemented for 
this format.

if ds is None: raise ValueError(fn)
rb = ds.GetRasterBand(1)
rb.WriteArray(img)
ds.FlushCache()
ds = None

This code works for geotiff, but not for jpeg2000. There I'm getting the 
error: ERROR 6: GDALDriver::Create() ... no create method implemented 
for this format.


I tested various combinations of various data types without any change 
in behavior.


I guess it should work, what can I do to solve this issue?

I compile openjpeg like this:

cmake . -DCMAKE_INSTALL_PREFIX=$prefix_opt -DCMAKE_C_FLAGS="-O3 -fPIC" 
-DBUILD_SHARED_LIBS:bool=on -DBUILD_CODEC:bool=on


and gdal like that:

kdu_opts=--with-kakadu=$prefix_opt
conda install --yes numpy
./configure \
--prefix $prefix_opt \
--with-openjpeg=$prefix_opt \
--with-jasper=$prefix_opt \
--with-libtiff=$prefix_opt \
--without-libtool \
--with-hdf4=$prefix_opt \
--with-hdf5=no \
--with-netcdf=no \
--with-geotiff=internal \
--with-grass=no \
--with-sqlite3=no \
$kdu_opts \
--with-pic \
--with-python=$prefix_python/bin/python \

make install
make clean

Am I missing any needed flags or options?

Kind regards, André

--
Dr. André Hollstein
Sektion 1.4 Fernerkundung
Tel.: +49 (0)331/288-28969
Fax: +49 (0)331/288-28969
E-Mail: andre.hollst...@gfz-potsdam.de
___

Helmholtz-Zentrum Potsdam
Deutsches GeoForschungsZentrum GFZ
Stiftung des öff. Rechts Land Brandenburg
Telegrafenberg, 14473 Potsdam
Haus 17, Raum 20.23

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev