[gdal-dev] Extracting (arbitrary?) overviews from ECW file
Dear list, I am new to programming with GDAL (and to the ECW format). Let me say thank you first, because my first experiences have been rather pleasant, even if my setup is relatively involved (thanks everybody for providing good documentation): - Ubuntu 12.04 (AMD64) - GDAL 1.10.1 (build from source) - ECW SDK 5.0 - GDAL Java bindings I successfully created a toy Java class for extracting regions from an ECW file. Speed is impressive. My observations/questions: - If I understand correctly, ECW files internally do not have such a thing as fixed overviews. However, ECW supports extraction of regions with specified resolution on-the-fly. Is this correct? - GDAL API reports overviews for ECW datasets. Having a first look at the code, it seems to generate overview dimensions by subsequently dividing height and width by two (until a minium side length of 128 has been reached). Final question: Is it possible to efficiently retrieve (regions of) overviews at self-defined scales using the GDAL API? Or do I have to use the ECW SDK for that? Keep up the good work. Thanks for any pointers. Best regards, Markus -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Extracting (arbitrary?) overviews from ECW file
Frank, thanks for your reply. Could you (or somebody else) give me a few more details how to get hold of a raster region with a resolution that is not defined by an overview? Currently, I am able to extract regions from an overview using the following steps (in Java): ... Dataset dataset = gdal.Open( "dataset.ecw", GA_ReadOnly ); Band band = dataset.GetRasterBand( ... ); band = band.GetOverview(...); band.ReadRaster(x,y, ...) ... Afterwards, I have the raw data values in my array. It works fine for any overview. My problem is that I cannot figure out how to get hold of a Band for a non-predefined overview. How would I do this? Best regards, Markus Am 04.09.2013 18:15, schrieb Frank Warmerdam: > Markus, > > I believe the GDAL ECW driver will pass RasterIO() requests through to > the ECW SDK to evaluate meaning that there is no constraint based on the > advertized power-of-two overviews exposed from GDAL and no loss of quality. > > Best regards, > Frank > > > On Wed, Sep 4, 2013 at 6:52 AM, Markus Schneider <mailto:schnei...@occamlabs.de>> wrote: > > Dear list, > > I am new to programming with GDAL (and to the ECW format). Let me say > thank you first, because my first experiences have been rather pleasant, > even if my setup is relatively involved (thanks everybody for providing > good documentation): > > - Ubuntu 12.04 (AMD64) > - GDAL 1.10.1 (build from source) > - ECW SDK 5.0 > - GDAL Java bindings > > I successfully created a toy Java class for extracting regions from an > ECW file. Speed is impressive. > > My observations/questions: > > - If I understand correctly, ECW files internally do not have such a > thing as fixed overviews. However, ECW supports extraction of regions > with specified resolution on-the-fly. Is this correct? > - GDAL API reports overviews for ECW datasets. Having a first look at > the code, it seems to generate overview dimensions by subsequently > dividing height and width by two (until a minium side length of 128 has > been reached). > > Final question: Is it possible to efficiently retrieve (regions of) > overviews at self-defined scales using the GDAL API? Or do I have to use > the ECW SDK for that? > > Keep up the good work. Thanks for any pointers. > > Best regards, > Markus > > -- > Markus Schneider > CEO > > Occam Labs UG (haftungsbeschränkt) > Godesberger Allee 139 > 53175 Bonn, Germany > > +49 228 93798874 > > http://www.occamlabs.de > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org <mailto:gdal-dev@lists.osgeo.org> > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > > -- > ---+-- > I set the clouds in motion - turn up | Frank Warmerdam, > warmer...@pobox.com <mailto:warmer...@pobox.com> > light and sound - activate the windows | http://pobox.com/~warmerdam > and watch the world go round - Rush| Geospatial Software Developer -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] GDAL Warp API Java bindings
Hi again, I would like to try using the Warp API from Java. It seems that there are currently no Java bindings available. - Did anbody work on Java bindings for the Warp API? - How hard would it be to define them? I may be able to invest some time here. Best regards, Markus -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Warp API Java bindings
kss, thanks. gdal.ReprojectImage(..) looks like what I will be using then. :-) My use case: - My huge (ECW) dataset is in EPSG:28992 - I want to extract a defined region in EPSG:4326 as a (Java) BufferedImage How can I get hold of a suitable destination dataset so I can use ReprojectImage(...)? Thanks, Markus Am 05.09.2013 18:19, schrieb Kyle Shannon: > On Thu, Sep 5, 2013 at 10:05 AM, Markus Schneider > wrote: >> Hi again, >> >> I would like to try using the Warp API from Java. It seems that there >> are currently no Java bindings available. >> >> - Did anbody work on Java bindings for the Warp API? >> - How hard would it be to define them? I may be able to invest some time >> here. >> >> Best regards, >> Markus >> >> -- >> Markus Schneider >> CEO >> >> Occam Labs UG (haftungsbeschränkt) >> Godesberger Allee 139 >> 53175 Bonn, Germany >> >> +49 228 93798874 >> >> http://www.occamlabs.de >> >> >> ___ >> gdal-dev mailing list >> gdal-dev@lists.osgeo.org >> http://lists.osgeo.org/mailman/listinfo/gdal-dev > > Markus, > > What are you trying to do exactly? I see that ReprojectImage() and > AutoCreateWarpedVRT are exposed in org.gdal.gdal.gdal: > > http://gdal.org/java/ > > kss > -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Extracting (arbitrary?) overviews from ECW file
I believe I learned how to do this, so to answer my own question: Some of the Band.ReadRaster(...) methods allow to explictly specify the buffer width and height, e.g. [1]. This allows to extract resolutions that differs from the original image (and the overviews). Best regards, Markus [1] http://gdal.org/java/org/gdal/gdal/Band.html#ReadRaster(int, int, int, int, int, int, int, byte[]) Am 05.09.2013 18:01, schrieb Markus Schneider: > Frank, > > thanks for your reply. Could you (or somebody else) give me a few more > details how to get hold of a raster region with a resolution that is not > defined by an overview? > > Currently, I am able to extract regions from an overview using the > following steps (in Java): > > ... > Dataset dataset = gdal.Open( "dataset.ecw", GA_ReadOnly ); > Band band = dataset.GetRasterBand( ... ); > band = band.GetOverview(...); > band.ReadRaster(x,y, ...) > ... > > Afterwards, I have the raw data values in my array. It works fine for > any overview. > > My problem is that I cannot figure out how to get hold of a Band for a > non-predefined overview. How would I do this? > > Best regards, > Markus > > Am 04.09.2013 18:15, schrieb Frank Warmerdam: >> Markus, >> >> I believe the GDAL ECW driver will pass RasterIO() requests through to >> the ECW SDK to evaluate meaning that there is no constraint based on the >> advertized power-of-two overviews exposed from GDAL and no loss of quality. >> >> Best regards, >> Frank >> >> >> On Wed, Sep 4, 2013 at 6:52 AM, Markus Schneider > <mailto:schnei...@occamlabs.de>> wrote: >> >> Dear list, >> >> I am new to programming with GDAL (and to the ECW format). Let me say >> thank you first, because my first experiences have been rather pleasant, >> even if my setup is relatively involved (thanks everybody for providing >> good documentation): >> >> - Ubuntu 12.04 (AMD64) >> - GDAL 1.10.1 (build from source) >> - ECW SDK 5.0 >> - GDAL Java bindings >> >> I successfully created a toy Java class for extracting regions from an >> ECW file. Speed is impressive. >> >> My observations/questions: >> >> - If I understand correctly, ECW files internally do not have such a >> thing as fixed overviews. However, ECW supports extraction of regions >> with specified resolution on-the-fly. Is this correct? >> - GDAL API reports overviews for ECW datasets. Having a first look at >> the code, it seems to generate overview dimensions by subsequently >> dividing height and width by two (until a minium side length of 128 has >> been reached). >> >> Final question: Is it possible to efficiently retrieve (regions of) >> overviews at self-defined scales using the GDAL API? Or do I have to use >> the ECW SDK for that? >> >> Keep up the good work. Thanks for any pointers. >> >> Best regards, >> Markus >> >> -- >> Markus Schneider >> CEO >> >> Occam Labs UG (haftungsbeschränkt) >> Godesberger Allee 139 >> 53175 Bonn, Germany >> >> +49 228 93798874 >> >> http://www.occamlabs.de >> >> >> ___ >> gdal-dev mailing list >> gdal-dev@lists.osgeo.org <mailto:gdal-dev@lists.osgeo.org> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> >> >> >> -- >> ---+-- >> I set the clouds in motion - turn up | Frank Warmerdam, >> warmer...@pobox.com <mailto:warmer...@pobox.com> >> light and sound - activate the windows | http://pobox.com/~warmerdam >> and watch the world go round - Rush| Geospatial Software Developer > > > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Extracting (arbitrary?) overviews from ECW file
There's one question remaining: Will the mentioned methods automatically select the best-matching overview, or do I have to do this myselft in order to achieve the best performance? This probably doesn't matter to ECW, but to other image formats with "real" overviews (such as GeoTIFF), right? Best regards, Markus Am 10.09.2013 11:13, schrieb Markus Schneider: > I believe I learned how to do this, so to answer my own question: > > Some of the Band.ReadRaster(...) methods allow to explictly specify the > buffer width and height, e.g. [1]. This allows to extract resolutions > that differs from the original image (and the overviews). > > Best regards, > Markus > > [1] http://gdal.org/java/org/gdal/gdal/Band.html#ReadRaster(int, int, > int, int, int, int, int, byte[]) > > Am 05.09.2013 18:01, schrieb Markus Schneider: >> Frank, >> >> thanks for your reply. Could you (or somebody else) give me a few more >> details how to get hold of a raster region with a resolution that is not >> defined by an overview? >> >> Currently, I am able to extract regions from an overview using the >> following steps (in Java): >> >> ... >> Dataset dataset = gdal.Open( "dataset.ecw", GA_ReadOnly ); >> Band band = dataset.GetRasterBand( ... ); >> band = band.GetOverview(...); >> band.ReadRaster(x,y, ...) >> ... >> >> Afterwards, I have the raw data values in my array. It works fine for >> any overview. >> >> My problem is that I cannot figure out how to get hold of a Band for a >> non-predefined overview. How would I do this? >> >> Best regards, >> Markus >> >> Am 04.09.2013 18:15, schrieb Frank Warmerdam: >>> Markus, >>> >>> I believe the GDAL ECW driver will pass RasterIO() requests through to >>> the ECW SDK to evaluate meaning that there is no constraint based on the >>> advertized power-of-two overviews exposed from GDAL and no loss of quality. >>> >>> Best regards, >>> Frank >>> >>> >>> On Wed, Sep 4, 2013 at 6:52 AM, Markus Schneider >> <mailto:schnei...@occamlabs.de>> wrote: >>> >>> Dear list, >>> >>> I am new to programming with GDAL (and to the ECW format). Let me say >>> thank you first, because my first experiences have been rather pleasant, >>> even if my setup is relatively involved (thanks everybody for providing >>> good documentation): >>> >>> - Ubuntu 12.04 (AMD64) >>> - GDAL 1.10.1 (build from source) >>> - ECW SDK 5.0 >>> - GDAL Java bindings >>> >>> I successfully created a toy Java class for extracting regions from an >>> ECW file. Speed is impressive. >>> >>> My observations/questions: >>> >>> - If I understand correctly, ECW files internally do not have such a >>> thing as fixed overviews. However, ECW supports extraction of regions >>> with specified resolution on-the-fly. Is this correct? >>> - GDAL API reports overviews for ECW datasets. Having a first look at >>> the code, it seems to generate overview dimensions by subsequently >>> dividing height and width by two (until a minium side length of 128 has >>> been reached). >>> >>> Final question: Is it possible to efficiently retrieve (regions of) >>> overviews at self-defined scales using the GDAL API? Or do I have to use >>> the ECW SDK for that? >>> >>> Keep up the good work. Thanks for any pointers. >>> >>> Best regards, >>> Markus >>> >>> -- >>> Markus Schneider >>> CEO >>> >>> Occam Labs UG (haftungsbeschränkt) >>> Godesberger Allee 139 >>> 53175 Bonn, Germany >>> >>> +49 228 93798874 >>> >>> http://www.occamlabs.de >>> >>> >>> ___ >>> gdal-dev mailing list >>> gdal-dev@lists.osgeo.org <mailto:gdal-dev@lists.osgeo.org> >>> http://lists.osgeo.org/mailman/listinfo/gdal-dev >>> >>> >>> >>> >>> -- >>> ---+-- >>> I set the clouds in motion - turn up | Frank Warmerdam, >>> warmer...@pobox.com <mailto:warmer...@pobox.com> >>> light and sound - activate the windows | http://pobox.com/~warmerdam >>> and watch the world go round - Rush| Geospatial Software Developer >> >> >> >> >> ___ >> gdal-dev mailing list >> gdal-dev@lists.osgeo.org >> http://lists.osgeo.org/mailman/listinfo/gdal-dev >> > > > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Extracting (arbitrary?) overviews from ECW file
Ok, I just tried it out with an example GeoTIFF, and the mentioned methods seem to benefit from overviews in the file, so it seems to automatically do the right thing. Best regards, Markus Am 10.09.2013 11:20, schrieb Markus Schneider: > There's one question remaining: > > Will the mentioned methods automatically select the best-matching > overview, or do I have to do this myselft in order to achieve the best > performance? > > This probably doesn't matter to ECW, but to other image formats with > "real" overviews (such as GeoTIFF), right? > > Best regards, > Markus > > Am 10.09.2013 11:13, schrieb Markus Schneider: >> I believe I learned how to do this, so to answer my own question: >> >> Some of the Band.ReadRaster(...) methods allow to explictly specify the >> buffer width and height, e.g. [1]. This allows to extract resolutions >> that differs from the original image (and the overviews). >> >> Best regards, >> Markus >> >> [1] http://gdal.org/java/org/gdal/gdal/Band.html#ReadRaster(int, int, >> int, int, int, int, int, byte[]) >> >> Am 05.09.2013 18:01, schrieb Markus Schneider: >>> Frank, >>> >>> thanks for your reply. Could you (or somebody else) give me a few more >>> details how to get hold of a raster region with a resolution that is not >>> defined by an overview? >>> >>> Currently, I am able to extract regions from an overview using the >>> following steps (in Java): >>> >>> ... >>> Dataset dataset = gdal.Open( "dataset.ecw", GA_ReadOnly ); >>> Band band = dataset.GetRasterBand( ... ); >>> band = band.GetOverview(...); >>> band.ReadRaster(x,y, ...) >>> ... >>> >>> Afterwards, I have the raw data values in my array. It works fine for >>> any overview. >>> >>> My problem is that I cannot figure out how to get hold of a Band for a >>> non-predefined overview. How would I do this? >>> >>> Best regards, >>> Markus >>> >>> Am 04.09.2013 18:15, schrieb Frank Warmerdam: >>>> Markus, >>>> >>>> I believe the GDAL ECW driver will pass RasterIO() requests through to >>>> the ECW SDK to evaluate meaning that there is no constraint based on the >>>> advertized power-of-two overviews exposed from GDAL and no loss of quality. >>>> >>>> Best regards, >>>> Frank >>>> >>>> >>>> On Wed, Sep 4, 2013 at 6:52 AM, Markus Schneider >>> <mailto:schnei...@occamlabs.de>> wrote: >>>> >>>> Dear list, >>>> >>>> I am new to programming with GDAL (and to the ECW format). Let me say >>>> thank you first, because my first experiences have been rather >>>> pleasant, >>>> even if my setup is relatively involved (thanks everybody for providing >>>> good documentation): >>>> >>>> - Ubuntu 12.04 (AMD64) >>>> - GDAL 1.10.1 (build from source) >>>> - ECW SDK 5.0 >>>> - GDAL Java bindings >>>> >>>> I successfully created a toy Java class for extracting regions from an >>>> ECW file. Speed is impressive. >>>> >>>> My observations/questions: >>>> >>>> - If I understand correctly, ECW files internally do not have such a >>>> thing as fixed overviews. However, ECW supports extraction of regions >>>> with specified resolution on-the-fly. Is this correct? >>>> - GDAL API reports overviews for ECW datasets. Having a first look at >>>> the code, it seems to generate overview dimensions by subsequently >>>> dividing height and width by two (until a minium side length of 128 has >>>> been reached). >>>> >>>> Final question: Is it possible to efficiently retrieve (regions of) >>>> overviews at self-defined scales using the GDAL API? Or do I have to >>>> use >>>> the ECW SDK for that? >>>> >>>> Keep up the good work. Thanks for any pointers. >>>> >>>> Best regards, >>>> Markus >>>> >>>> -- >>>> Markus Schneider >>>> CEO >>>> >>>> Occam Labs UG (haftungsbeschränkt) >>>> Godesberger Allee 139 >>>> 53175 Bonn, Germany >>>> >>>> +49 228 93798874 &g
Re: [gdal-dev] GDAL Warp API Java bindings
Hi Florent, yes, I am aware of that and I managed to create BufferedImages from GDAL Datasets. However, I find the Java Image API (and the whole subject) relatively complex, so I am not sure if my code is efficient. Can anybody have a quick look at my code and tell me if there are any quick performance gains? Code is here: http://cdv.lt/Qw1? Thanks. Best regards, Markus P.S.: I am aware that the code is limited to RGB without transparency and byte-encoded color values. Am 05.09.2013 19:23, schrieb Florent JITIAUX: > Hi, > > in fact with the Java binding there's no method to get a BufferedImage > from a Dataset. > > You can warp your image and get a Dataset > > You have to create a (Java)Raster and fill it with the raster or a part > of the raster of the warped Dataset. > But be carefull because when you create a Raster you must create a > correct SampleModel and the right DataBuffer generally Byte or Int, it > depends of the type of the raster in the Dataset. > > Florent > > > 2013/9/5 Markus Schneider <mailto:schnei...@occamlabs.de>> > > Hi again, > > I would like to try using the Warp API from Java. It seems that there > are currently no Java bindings available. > > - Did anbody work on Java bindings for the Warp API? > - How hard would it be to define them? I may be able to invest some time > here. > > Best regards, > Markus > > -- > Markus Schneider > CEO > > Occam Labs UG (haftungsbeschränkt) > Godesberger Allee 139 > 53175 Bonn, Germany > > +49 228 93798874 > > http://www.occamlabs.de > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org <mailto:gdal-dev@lists.osgeo.org> > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Sharing GDAL Dataset objects between threads (JNI)
Hi, I am using GDAL (1.10.1) in a Java Servlet environment via JNI. I observed that Dataset objects cannot be shared between threads, even if they are not used in parallel, but strictly consecutively. For me, the VM crashes if a thread uses a Dataset that it didn't create. So somehow Dataset objects (on the Java or on the C side) seem to be using ThreadLocal information. Is this correct? Would it be hard to eliminate this behaviour (as it would make pooling Dataset objects much easier and more efficient)? Thanks for any pointers or remarks, Markus -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Sharing GDAL Dataset objects between threads (JNI)
Hi Even, Am 14.10.2013 13:10, schrieb Even Rouault: > This is surprising. Are you seeing that with a particular driver or all > drivers > ? I am experiencing this with ECW driver. > I don't see any use of TLS on the Java side. There might be some use of TLS > objects on the C side, for functions in cpl_path.cpp for example, but their > use > should be limited to the lifetime of the execution of a particular function, > so > that shouldn't cause problems in the scenario you describe. I will try to prepare a minimal example to reproduce the issue. I will also check if it occurs with a GeoTIFF. >> Would it be hard to eliminate this behaviour (as it >> would make pooling Dataset objects much easier and more efficient)? > > Difficult to tell until we have understood what really happens... > > Do you manage to reproduce that by running under Valgrind ? If so, that might > help understanding what is going wrong. I never used Valgrind, but with some help, I may be able to check behaviour and report. Best regards, Markus -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Sharing GDAL Dataset objects between threads (JNI)
Hi Even, thanks for you help so far. I now observed that handing around a (ECW) Dataset between threads works fine, as long as I open it using: - gdal.Open(String name) However, when I open it using: - gdal.OpenShared(String name) I see strange effects and VM crashes. Does this make sense? What do you believe is preferrable with regard to performance and memory consumption: 1. Using a ThreadPool and obtain a Dataset using gdal.OpenShared for each thread 2. Using gdal.Open and simply pool the obtained Dataset objects (and hand them around between threads) Best regards, Markus Am 14.10.2013 13:26, schrieb Even Rouault: > Selon Markus Schneider : > >> Hi Even, >> >> Am 14.10.2013 13:10, schrieb Even Rouault: >>> This is surprising. Are you seeing that with a particular driver or all >> drivers >>> ? >> >> I am experiencing this with ECW driver. > > Ah, well, anything can happen within the beast then... Which SDK version : 3.3 > or 5 ? > >> >>> I don't see any use of TLS on the Java side. There might be some use of TLS >>> objects on the C side, for functions in cpl_path.cpp for example, but their >> use >>> should be limited to the lifetime of the execution of a particular >> function, so >>> that shouldn't cause problems in the scenario you describe. >> >> I will try to prepare a minimal example to reproduce the issue. I will >> also check if it occurs with a GeoTIFF. > > Yes that's definnitely something to check. > >> >>>> Would it be hard to eliminate this behaviour (as it >>>> would make pooling Dataset objects much easier and more efficient)? >>> >>> Difficult to tell until we have understood what really happens... >>> >>> Do you manage to reproduce that by running under Valgrind ? If so, that >> might >>> help understanding what is going wrong. >> >> I never used Valgrind, but with some help, I may be able to check >> behaviour and report. > > $ apt-get/yum install valgrind > $ valgrind [valgrind_options] the_binary [the_binary_options] > 2>valgrind_log.txt > > where the_binary will be your servlet server. Not sure how well it run within > Valgrind. You need to use a recent enough version of Valgrind to run the JVM > with it (at least for 64 bit builds). And things will run at the minimum 10x > slower > > Running Java with Valgrind tends to generate a lot of warnings. You'll > probably > have to check the Valgrind option that increases the maximum number of warning > messages. > >> >> Best regards, >> Markus >> >> -- >> Markus Schneider >> CEO >> >> Occam Labs UG (haftungsbeschränkt) >> Godesberger Allee 139 >> 53175 Bonn, Germany >> >> +49 228 93798874 >> >> http://www.occamlabs.de >> >> -- Markus Schneider CEO Occam Labs UG (haftungsbeschränkt) Godesberger Allee 139 53175 Bonn, Germany +49 228 93798874 http://www.occamlabs.de signature.asc Description: OpenPGP digital signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Sharing GDAL Dataset objects between threads (JNI)
Thanks again.
I found the comments in gdaldataset.cpp, and understand now that there's
no gain in using GDALOpenShared over GDALOpen in a multi-threaded
environment, as GDALOpenShared will create multiple GDALDataset objects
anyway in such a scenario.
Best regards,
Markus
Am 17.10.2013 13:27, schrieb Even Rouault:
> Selon Markus Schneider :
>
>> Hi Even,
>>
>> thanks for you help so far.
>>
>> I now observed that handing around a (ECW) Dataset between threads works
>> fine, as long as I open it using:
>>
>> - gdal.Open(String name)
>>
>> However, when I open it using:
>>
>> - gdal.OpenShared(String name)
>>
>> I see strange effects and VM crashes.
>>
>> Does this make sense?
>
> Yes, GDALOpenShared() ( see the doc of the C function ) must be used with
> great
> caution in a multi-threaded scenario. As long as the thread that has called
> GDALOpenShared() continues to use it until it closes it, it is OK. But if that
> thread handles the dataset to another one, then chaos might happen.
>
> For example in the following scenario :
>
> Thread 1 Thread 2
> ds1 = OpenShared("foo")
>-- handle it to -->
> ds2 = OpenShared("foo") ( == ds1)
> do_things(ds2)do_things(ds1)
> ds2.Close()
> ds1.Close()
>
> At a point, you can call methods on the same dataset object from different
> threads...
>
> By the way, the issue you've observed with ECW would also be true with other
> drivers. It is just by chance that you didn't notice it.
>
>> What do you believe is preferrable with regard to
>> performance and memory consumption:
>>
>> 1. Using a ThreadPool and obtain a Dataset using gdal.OpenShared for
>> each thread
>> 2. Using gdal.Open and simply pool the obtained Dataset objects (and
>> hand them around between threads)
>
> I'm not sure to completely understand what you've in mind, but I would rather
> recommand option 2.
>
> Best regards,
>
> Even
>
--
Markus Schneider
CEO
Occam Labs UG (haftungsbeschränkt)
Godesberger Allee 139
53175 Bonn, Germany
+49 228 93798874
http://www.occamlabs.de
signature.asc
Description: OpenPGP digital signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
