Re: [gdal-dev] GDAL /vsistdin/ 1MB limit

[Romeo Alvaraz]

Thanks Even for the response and clarifications on /vsistdin/.  Our reasons
for attempting to use /vsistdin/ is to avoid having to read a file from the
local file system (which we would have to write to first by other means) on
our remote server. We've been using this in conjunction with /vsistdout/. 

/vsicurl/  was another option we considered for the input but it's not ideal
to our situation.

I'm just wondering  if there is another handler/method you could suggest
that could feed the input directly to gdal_translate in order to bypass use
of file system?

Thanks. 




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] About gdal RasterizeLayer

[jratike80]

Hi,

I tried if someone who don't know Python could do about the same with
gdal_rasterize https://gdal.org/programs/gdal_rasterize.html and I had some
success.


gdal_rasterize -dialect sqlite -sql "select st_buffer(geometry,0.01) from
states" -burn 100 -of GTIFF -ot byte -tr 0.1 0.1 -add states.shp
testburn.tif

I used byte type and burn value 100 for making quick visible result. The
default type is Float64 and it has room for much more overlapping geometries
than two.

-Jukka Rahkonen-


Frank Warmerdam wrote
> Miguel,
> 
> I can't seem to find this in the docs, but I think what you want is
> "MERGE_ALG" set to ADD instead of the default REPLACE.
> 
> https://github.com/OSGeo/gdal/blob/master/gdal/alg/gdalrasterize.cpp#L1165
> 
> This doesn't address the buffering (I guess you might still want to buffer
> the whole set), but it should take care of the additive step.
> 
> I actually implemented this as part of a programming test for my current
> job - to build heatmaps of satellite coverage.
> 
> Best regards,
> Frank
> 
> 
> On Tue, May 4, 2021 at 1:30 PM Miguel A. Manso <

> m.manso@

> > wrote:
> 
>> Dear all
>>
>> I have a multiline layer type (tracking) and I want to make a heat map
>> (densities) on a buffer of them, so that in those places where several
>> buffers overlap the density is added as many times as lines overlap.
>>
>> Now I'm doing it with a python script that reads from a shp the lines,
>> generates a buffer and rasterize the polygon filling it with a given
>> value with gda.RasterizeLayer(). I read the image as an array and with
>> numpy I add it to another array (accumulating). When I finish I convert
>> it to an image and save it in a geotiff.
>>
>> This procedure is relatively slow. Especially if you have to do it for
>> many lines.
>>
>> My question is along the lines of whether the RasterizeLayer function
>> itself could do this process in a more efficient way, making the burning
>> of the objects not an OR operation but an arithmetic sum...
>>
>> Any suggestions would be welcome.
>>
>> Regards
>> Miguel A. Manso
>>
>>
>> ___
>> gdal-dev mailing list
>> 

> gdal-dev@.osgeo

>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
> 
> 
> -- 
> ---+--
> I set the clouds in motion - turn up   | Frank Warmerdam,

> warmerdam@

> light and sound - activate the windows | +1 650-701-7823
> and watch the world go round - Rush| Geospatial Software Developer
> 
> ___
> gdal-dev mailing list

> gdal-dev@.osgeo

> https://lists.osgeo.org/mailman/listinfo/gdal-dev





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] About gdal RasterizeLayer

[Frank Warmerdam]

Miguel,

I can't seem to find this in the docs, but I think what you want is
"MERGE_ALG" set to ADD instead of the default REPLACE.

https://github.com/OSGeo/gdal/blob/master/gdal/alg/gdalrasterize.cpp#L1165

This doesn't address the buffering (I guess you might still want to buffer
the whole set), but it should take care of the additive step.

I actually implemented this as part of a programming test for my current
job - to build heatmaps of satellite coverage.

Best regards,
Frank


On Tue, May 4, 2021 at 1:30 PM Miguel A. Manso  wrote:

> Dear all
>
> I have a multiline layer type (tracking) and I want to make a heat map
> (densities) on a buffer of them, so that in those places where several
> buffers overlap the density is added as many times as lines overlap.
>
> Now I'm doing it with a python script that reads from a shp the lines,
> generates a buffer and rasterize the polygon filling it with a given
> value with gda.RasterizeLayer(). I read the image as an array and with
> numpy I add it to another array (accumulating). When I finish I convert
> it to an image and save it in a geotiff.
>
> This procedure is relatively slow. Especially if you have to do it for
> many lines.
>
> My question is along the lines of whether the RasterizeLayer function
> itself could do this process in a more efficient way, making the burning
> of the objects not an OR operation but an arithmetic sum...
>
> Any suggestions would be welcome.
>
> Regards
> Miguel A. Manso
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam,
warmer...@pobox.com
light and sound - activate the windows | +1 650-701-7823
and watch the world go round - Rush| Geospatial Software Developer
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Gdal_translate batch processing

[Matt.Wilkie]

Hi Gulnihal,

Something like this should work:

pushd C:\Users\HP\Desktop\modis
for /r %F in (*.hdf) do @echo gdal_translate -sds %F %~dpnF.tif
popd


This is a dry run to show what it will attempt, it won't actually do anything. 
Remove the "@echo" part to make it active. Double the % to use in a batch file 
(so "for /r %%F ...").

For help on what is being done here see:

 * https://ss64.com/nt/for_r.html
 * https://ss64.com/nt/syntax-args.html - what the "%~dpnF" part is doing.
 * https://ss64.com/nt/syntax-percent.html


Cheers,

-Matt

-Original Message-
From: gdal-dev  On Behalf Of gnihalk
Sent: April 29, 2021 5:36 AM
To: gdal-dev@lists.osgeo.org
Subject: [gdal-dev] Gdal_translate batch processing

*** External email: Do not click on links or attachments except from 
trusted senders. *** 
**


Hi you all. I have a huge MODIS dataset and I need to translate them to tiff 
but each data is in a different sub-folder.
C:\Users\HP\Desktop\modis\500889913\500889913\88258783\MOD10A2.2016322095024.hdf
C:\Users\HP\Desktop\modis\500889913\500889913\96113659\MOD10A2.A2016321.h20v05.006.2016340133152.hdf

.
the dataset sub-folders go like this...

and the code that im trying to run is:
for /R C:\Users\HP\Desktop\modis\500889913\500889913\88258783 %f IN
(MOD10A2.2016322095024.hdf) do gdal_translate.exe -sds -of gtiff %f 
C:\Users\HP\Desktop\modis\500889913\500889913\88258783\MOD10A2.2016322095024.tiff

I tried to make a loop for all subfolders that I have but I got an error 
massage. I would appreciate it if I can take your opinions about it. I am also 
not sure that I post this question in true title/section.
Thank you,
Gulnihal




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] About gdal RasterizeLayer

[Miguel A. Manso]

Dear all

I have a multiline layer type (tracking) and I want to make a heat map 
(densities) on a buffer of them, so that in those places where several 
buffers overlap the density is added as many times as lines overlap.


Now I'm doing it with a python script that reads from a shp the lines, 
generates a buffer and rasterize the polygon filling it with a given 
value with gda.RasterizeLayer(). I read the image as an array and with 
numpy I add it to another array (accumulating). When I finish I convert 
it to an image and save it in a geotiff.


This procedure is relatively slow. Especially if you have to do it for 
many lines.


My question is along the lines of whether the RasterizeLayer function 
itself could do this process in a more efficient way, making the burning 
of the objects not an OR operation but an arithmetic sum...


Any suggestions would be welcome.

Regards
Miguel A. Manso


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.2.3 is released

[Even Rouault]

Hi,

On behalf of the GDAL/OGR development team, I am pleased to announce the
release of the GDAL/OGR 3.2.3 bug fix version, the last one in the 3.2 
series, and

just for those who can't yet update to the yesterday 3.3.0 release.

Consult the release notes for the list of issues addressed :
  https://github.com/OSGeo/gdal/blob/v3.2.3/gdal/NEWS

The sources are available at:

    https://download.osgeo.org/gdal/3.2.3/gdal-3.2.3.tar.xz
    https://download.osgeo.org/gdal/3.2.3/gdal-3.2.3.tar.gz
    https://download.osgeo.org/gdal/3.2.3/gdal323.zip
    https://download.osgeo.org/gdal/3.2.3/gdal-grass-3.2.3.tar.gz

Also note that GDAL 3.2.3 will be the last release to support Python 2.7.

It's been an exciting time in GDAL, as our sustainable GDAL initiative 
has attracted a number of major sponsors, which should bring more energy 
to the project in future releases.


Best regards,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: promote GDAL 3.2.3 RC1

[Even Rouault]

Motion:

Adopt GDAL 3.2.3 RC1 as final 3.2.3 release


Passed with +1 from JukkaR and myself

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[jratike80]

Hi,

gdal_translate -of gtiff -co compress=jpeg p4433h.tif -co
NUM_THREADS=ALL_CPUS comp.tif

Time: 4.7 seconds (with GDAL 3.3.0 gisinternals build)
Time: 11.5 seconds (with GDAL 3.1.4 OSGeo4W build)

-Jukka-



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Even Rouault]

gdal_translate -of gtiff -co compress=jpeg p4433h.tif comp.tif


for the gdal_translate to JPEG compressed TIFF, adding -co 
NUM_THREADS=ALL_CPUS should help to keep the comparison fair, and if you 
use gisinternal builds, I'm not completely sure they use libjpeg-turbo, 
which would bring another performance boost.


--

http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[jratike80]

Hi Even,

You are right, Kakadu 8.x supports HTJ2K. I just had a try with the
kdu_compress demo utility from KDU805 demo apps.  The speed was good,
actually amazing. As a test image I used a 12000x12000 RGB image. The
kdu_compress command I took from this high qualite white-paper
https://htj2k.com/wp-content/uploads/white-paper.pdf.

kdu_compress -i p4433h.ppm -o tmp3.jph -rate 2 -tolerance 0 Qstep=0.001
-quiet -no_weights

As a reference I used

gdal_translate -of gtiff -co compress=jpeg p4433h.tif comp.tif

Results (total time disk-to-disk on Windows, SSD disk, laptop with Intel
core i7)

kdu_compress - JPH: 1.4 seconds (file size 36 MB, fixed size defined by
option -rate 2)
kdu_compress - JPH: 1.1 seconds (file size 90 MB, fixed size defined by
option -rate 5)
gdal_translate - TIFF: 13.1 seconds (file size 92 MB)

The Kakadu demo applications that can be acquired on request from the Kakadu
site are compiled without libtiff and therefore the input must be in ppm
format. Those who have access to Kakadu source code can compile the
applications with "KDU_INCLUDE_TIFF".

-Jukka Rahkonen-




Even Rouault-2 wrote
> I'm not sure which support of HTJ2K the ECW and MrSID SDKs have. I 
> believe there's such support Kakadu in recent releases, but might be 
> subject to extra fees (or perhaps for  faster support)





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Aaron Boxer]

>
>
> did you bench timings using GDAL API & utilities (typically with
> gdal_translate) ? The ones indicated in the PR use the utilities that come
> which each library, but the integration within a GDAL driver might have
> some influence on the timings.
>

Yes, the timings will definitely be lower with the current driver, as it
doesn't take full advantage of some library features
such as

1. tile cache
2. multi-threading across tiles
3. combination of memory mapping and PLT markers, where parts of
the codestream that don't need to be decoded are not even read from disk at
all.

There's no reason why JP2Grok can't match the library utilities in
performance, once I support
the above features in the driver.


> What about single threaded usage ?
>

I can certainly benchmark this.

Aaron



> Le 03/05/2021 à 16:59, Aaron Boxer a écrit :
>
> Hi Folks,
>
> I would like to draw people's attention once again to the pending JPEG
> 2000 driver PR
> https://github.com/OSGeo/gdal/pull/3449
> which was opened 3 months ago.  Since I last posted, the driver now has an
> autotest script courtesy of Frank Warmerdam, and all tests are passing.
>
> In a nutshell, performance is 2-5 times faster than the only other viable
> open source driver, from OpenJPEG, in several common geospatial work flows.
> And performance rises by around 2X if the new HTJ2K entropy coder is used.
> The AGPL license will not work for everyone, which is why the driver is
> disabled by default.
>
> We've already discussed this at length in the previous thread and in the
> PR, but if you would like to see this new driver merged, please head over
> to the PR and give it a thumbs up. Or, if you don't want this driver
> merged, please share your feedback on the PR.
>
> Kind Regards,
> Aaron
>
>
>
>
>
>
>
>
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Python Binding Update?

[Even Rouault]

Hi Andrew,

I recently updated my local GDAL tree from github and it apparently 
forced the need to regenerate some Python bindings with SWIG. This 
modified the local files:


Yeah, I also noticed that a few times lately. This is a side effect of 
https://github.com/OSGeo/gdal/pull/3697 . I'm not sure why this happens 
as the timestamps of the generated .cpp files should be higher than the 
source .i files, and the issue is that when they are regenerated the 
'sed' tweaks done in the make generate target aren't applied. You may 
open a ticket about that




swig/python/extensions/gdal_array_wrap.cpp
swig/python/osgeo/gdal_array.py

Are these files supposed to be committed to the source repo?


Yes and no.

Yes, given how things are set up currently.

No, ideally: see https://github.com/OSGeo/gdal/issues/3562


Am I supposed to make a PR with the changes?


No

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Python Binding Update?

[Andrew Bell]

Hi,

I recently updated my local GDAL tree from github and it apparently forced
the need to regenerate some Python bindings with SWIG. This modified the
local files:

swig/python/extensions/gdal_array_wrap.cpp
swig/python/osgeo/gdal_array.py

Are these files supposed to be committed to the source repo? Am I supposed
to make a PR with the changes? I believe the SWIG I have is version 4 and
the files in question were previously generated with version 3.

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Even Rouault]

Aaron,

did you bench timings using GDAL API & utilities (typically with 
gdal_translate) ? The ones indicated in the PR use the utilities that 
come which each library, but the integration within a GDAL driver might 
have some influence on the timings.


What about single threaded usage ?

Even

Le 03/05/2021 à 16:59, Aaron Boxer a écrit :

Hi Folks,

I would like to draw people's attention once again to the pending JPEG 
2000 driver PR
https://github.com/OSGeo/gdal/pull/3449 

which was opened 3 months ago.  Since I last posted, the driver now 
has an autotest script courtesy of Frank Warmerdam, and all tests are 
passing.


In a nutshell, performance is 2-5 times faster than the only other 
viable open source driver, from OpenJPEG, in several common geospatial 
work flows. And performance rises by around 2X if the new HTJ2K 
entropy coder is used. The AGPL license will not work for everyone, 
which is why the driver is disabled by default.


We've already discussed this at length in the previous thread and in 
the PR, but if you would like to see this new driver merged, please 
head over to the PR and give it a thumbs up. Or, if you don't want 
this driver merged, please share your feedback on the PR.


Kind Regards,
Aaron









___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL /vsistdin/ 1MB limit

[Even Rouault]

/vsistdin/ is only meant for format whose reading is compatible of a 
streamed reading, that is you read bytes as they come from stdin. Which 
is only a subset of formats, and PDF is not in that subset (typically 
the cross reference table to objects is always put at the end of the 
file, hence the Seek(SEEK_END error you get). The 1 MB limit corresponds 
to the amount of data that is cached at the beginning of the file (this 
helps for drivers that do almost streamed reading, but do some backward 
seeks in their header part). Beyond that, /vsimem/ doesn't buffer 
anything, so if the driver  tries to seek back, that  fails. This 1MB 
limit could potentially be changed to be runtime configured, but that's 
not the case currently.


Le 04/05/2021 à 07:04, Romeo Alvaraz a écrit :

Hi there,

I've got a PDF geo-referencing implementation in place which makes use of
the gdal_translate utility.  Recently I've been testing out and
implementation to read in the input content from standard input stream using
the  /vsistdin/ handler.

According to the documentation for /vsistdin/ handler -  "Full seek in the
first MB of a file is possible" and I've encountered the following error(s)
when the content I've attempted to pipe in is greater than 1MB.

ERROR 6: Seek(SEEK_END) unsupported on /vsistdin when stdin > 1 MB
ERROR 6: backward Seek() unsupported on /vsistdin above first MB

Would this be a hard limit of 1MB placed the content size which can be piped
in via /vsistdin/(the source code and the error message suggests it migh me)
or  can it be configured? Has anyone encountered this before?

Any suggestions will be appreciated

Thanks.



Platform: Windows 64-bit
Version: GDAL 2.4.4, released 2020/01/08 sourced from
https://www.gisinternals.com/release.php




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Cannot compile 3.3.0 without geos

[thomas bonfort]

It's known regression, see https://github.com/OSGeo/gdal/issues/3782

--
thomas

On Tue, May 4, 2021 at 10:42 AM Javier Jimenez Shaw 
wrote:

> Hi
>
> I am trying to compile the new release 3.3.0 in Ubuntu 18.04 with the
> option "--without-geos".
>
> The message is this:
>
> GEOS support disabled
> configure: error: GEOS support explicitly enabled, but geos-config could
> not be found
>
> I tried also changing to "--with-geo=no", with the same result. (Both
> options are equivalent, right?)
>
> Looking at the news, I see one entry related:
> * configure: Ensure --with-geos/sfcgal fail if unavailable.
>
> Should I do something different to disable geos?
>
> Thanks
> .___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ...  ._ .__
> Entre dos pensamientos racionales
> hay infinitos pensamientos irracionales.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Cannot compile 3.3.0 without geos

[Javier Jimenez Shaw]

Hi

I am trying to compile the new release 3.3.0 in Ubuntu 18.04 with the
option "--without-geos".

The message is this:

GEOS support disabled
configure: error: GEOS support explicitly enabled, but geos-config could
not be found

I tried also changing to "--with-geo=no", with the same result. (Both
options are equivalent, right?)

Looking at the news, I see one entry related:
* configure: Ensure --with-geos/sfcgal fail if unavailable.

Should I do something different to disable geos?

Thanks
.___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ...  ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev