Re: [gdal-dev] Compression artifacts

[Dmitry Baryshnikov]

Hi Jan,

You can cut result raster with the geometry (shape file) using gdalwarp 
with cutline option (http://gdal.org/gdalwarp.html).


Best regards,
Dmitry

27.12.2012 16:38, Jan Hartmann ?:
Can anyone tell me wath is happening here? I have a large three-band 
raster (3.8G) looking like this:


http://mapserver.sara.nl/map2a.png

I compress it (from information from this list a few days ago) with:

gdal_translate -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR big.tif small.tif
gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW 
YCBCR -ro -r gauss small.tif 2 4 8 16 32 64


and get a file of 37M, which is nice. However, the map has all kinds 
of rubbish at the borders. It looks like this:


http://mapserver.sara.nl/map2b.png

The equivalent pictures of a small part of the map look like this:

http://mapserver.sara.nl/map1a.png
http://mapserver.sara.nl/map1b.png

Can anyone help me?

Jan


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Jan Hartmann]

That would be an option. Am I right that the datasource of the -cutline 
option is a closed polygon, and that everything outside the polygon will 
be clipped?


However, I would prefer not to add extra steps. These are large maps and 
there are lots of them (about 17.000 for the whole of the Netherlands). 
It takes a lot of organisation to georeference them and display them 
efficiently over the web. Every additional processing step is a 
potential source of problems. So, although I am happy there is at least 
a workable solution, do you know what is happening here, and whether 
these artifacts can be prevented?


On 12/27/2012 01:44 PM, Dmitry Baryshnikov wrote:

Hi Jan,

You can cut result raster with the geometry (shape file) using 
gdalwarp with cutline option (http://gdal.org/gdalwarp.html).


Best regards,
Dmitry

27.12.2012 16:38, Jan Hartmann ?:
Can anyone tell me wath is happening here? I have a large three-band 
raster (3.8G) looking like this:


http://mapserver.sara.nl/map2a.png

I compress it (from information from this list a few days ago) with:

gdal_translate -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR big.tif small.tif
gdaladdo --config COMPRESS_OVERVIEW JPEG --config 
PHOTOMETRIC_OVERVIEW YCBCR -ro -r gauss small.tif 2 4 8 16 32 64


and get a file of 37M, which is nice. However, the map has all kinds 
of rubbish at the borders. It looks like this:


http://mapserver.sara.nl/map2b.png

The equivalent pictures of a small part of the map look like this:

http://mapserver.sara.nl/map1a.png
http://mapserver.sara.nl/map1b.png

Can anyone help me?

Jan


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev




___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Even Rouault]

Le jeudi 27 décembre 2012 13:38:50, Jan Hartmann a écrit :
> Can anyone tell me wath is happening here? I have a large three-band
> raster (3.8G) looking like this:
> 
> http://mapserver.sara.nl/map2a.png
> 
> I compress it (from information from this list a few days ago) with:
> 
> gdal_translate -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR big.tif small.tif
> gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
> YCBCR -ro -r gauss small.tif 2 4 8 16 32 64
> 
> and get a file of 37M, which is nice. However, the map has all kinds of
> rubbish at the borders. It looks like this:
> 
> http://mapserver.sara.nl/map2b.png
> 
> The equivalent pictures of a small part of the map look like this:
> 
> http://mapserver.sara.nl/map1a.png
> http://mapserver.sara.nl/map1b.png
> 
> Can anyone help me?

The artifacts are due to the lossy characteristics of JPEG compression that 
are particularly visible on borders between nodata pixel and valid pixels. An 
option would be to use nearblack to create a .msk mask file :

nearblack -setmask -o big_with_mask.tif big.tif -of gtiff

(I'm not sure if your original image has black or white nodata areas. If it is 
white, then add -white to the previous command line)

And then do the gdal_translate with big_with_mask.tif as input (you should 
perhaps consider adding "-co TILED=YES" too), and then gdaladdo. I've just 
pushed a fix since building overviews with "--config COMPRESS_OVERVIEW JPEG --
config PHOTOMETRIC_OVERVIEW
YCBCR" when there's a .msk didn't work.

MapServer should be able to use the mask file as a transparency mask.

> 
> Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Jan Hartmann]

On 12/27/2012 02:27 PM, Even Rouault wrote:

The artifacts are due to the lossy characteristics of JPEG compression that
are particularly visible on borders between nodata pixel and valid pixels. An
option would be to use nearblack to create a .msk mask file :

nearblack -setmask -o big_with_mask.tif big.tif -of gtiff

(I'm not sure if your original image has black or white nodata areas. If it is
white, then add -white to the previous command line)

And then do the gdal_translate with big_with_mask.tif as input (you should
perhaps consider adding "-co TILED=YES" too), and then gdaladdo. I've just
pushed a fix since building overviews with "--config COMPRESS_OVERVIEW JPEG --
config PHOTOMETRIC_OVERVIEW
YCBCR" when there's a .msk didn't work.

MapServer should be able to use the mask file as a transparency mask.



OK Even, thanks. This means that I should use gdal-svn?
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Even Rouault]

Le jeudi 27 décembre 2012 14:37:49, Jan Hartmann a écrit :
> On 12/27/2012 02:27 PM, Even Rouault wrote:
> > The artifacts are due to the lossy characteristics of JPEG compression
> > that are particularly visible on borders between nodata pixel and valid
> > pixels. An option would be to use nearblack to create a .msk mask file :
> > 
> > nearblack -setmask -o big_with_mask.tif big.tif -of gtiff
> > 
> > (I'm not sure if your original image has black or white nodata areas. If
> > it is white, then add -white to the previous command line)
> > 
> > And then do the gdal_translate with big_with_mask.tif as input (you
> > should perhaps consider adding "-co TILED=YES" too), and then gdaladdo.
> > I've just pushed a fix since building overviews with "--config
> > COMPRESS_OVERVIEW JPEG -- config PHOTOMETRIC_OVERVIEW
> > YCBCR" when there's a .msk didn't work.
> > 
> > MapServer should be able to use the mask file as a transparency mask.
> 
> OK Even, thanks. This means that I should use gdal-svn?

Not necessarily. I think you can workaround the issue by manually building the 
overview on the .msk file as a post-processing step :

gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE small.tif.msk 2 4 8 16 32 64
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Jan Hartmann]

On 12/27/2012 02:41 PM, Even Rouault wrote:

I've just pushed a fix since building overviews with "--config
COMPRESS_OVERVIEW JPEG -- config PHOTOMETRIC_OVERVIEW
YCBCR" when there's a .msk didn't work.



OK Even, thanks. This means that I should use gdal-svn?

Not necessarily. I think you can workaround the issue by manually building the
overview on the .msk file as a post-processing step :

gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE small.tif.msk 2 4 8 16 32 64
Do you mean I should use DEFLATE instead of JPEG as compression for 
overviews for the time being? What is the difference in compression ratio?
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Even Rouault]

Le jeudi 27 décembre 2012 15:11:23, Jan Hartmann a écrit :
> On 12/27/2012 02:41 PM, Even Rouault wrote:
> > I've just pushed a fix since building overviews with "--config
> > COMPRESS_OVERVIEW JPEG -- config PHOTOMETRIC_OVERVIEW
> > YCBCR" when there's a .msk didn't work.
> > 
> >> OK Even, thanks. This means that I should use gdal-svn?
> > 
> > Not necessarily. I think you can workaround the issue by manually
> > building the overview on the .msk file as a post-processing step :
> > 
> > gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE small.tif.msk 2 4 8 16 32
> > 64
> 
> Do you mean I should use DEFLATE instead of JPEG as compression for
> overviews for the time being? What is the difference in compression ratio?

Use DEFLATE only for the mask file (see small.tif.msk in the above command 
line), as it is a lossless compression, which is needed for a mask. JPEG is 
fine for the imagery itself

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Jan Hartmann]

On 12/27/2012 03:11 PM, Even Rouault wrote:

Le jeudi 27 décembre 2012 15:11:23, Jan Hartmann a écrit :

On 12/27/2012 02:41 PM, Even Rouault wrote:

I've just pushed a fix since building overviews with "--config
COMPRESS_OVERVIEW JPEG -- config PHOTOMETRIC_OVERVIEW
YCBCR" when there's a .msk didn't work.


OK Even, thanks. This means that I should use gdal-svn?

Not necessarily. I think you can workaround the issue by manually
building the overview on the .msk file as a post-processing step :

gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE small.tif.msk 2 4 8 16 32
64

Do you mean I should use DEFLATE instead of JPEG as compression for
overviews for the time being? What is the difference in compression ratio?

Use DEFLATE only for the mask file (see small.tif.msk in the above command
line), as it is a lossless compression, which is needed for a mask. JPEG is
fine for the imagery itself
Sorry, I don't understand. Where does that small.tif.msk come from? 
Until now I had:

- the original raster (big.tif)
- the raster masked with nearblack (big_with_mask.tif)
- the compressed raster (gdal_translate -co...)
- the overviews built with gdaladdo

The mask is part of the raster file, isn't  it?

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Even Rouault]

Le jeudi 27 décembre 2012 15:35:25, Jan Hartmann a écrit :
> On 12/27/2012 03:11 PM, Even Rouault wrote:
> > Le jeudi 27 décembre 2012 15:11:23, Jan Hartmann a écrit :
> >> On 12/27/2012 02:41 PM, Even Rouault wrote:
> >>> I've just pushed a fix since building overviews with "--config
> >>> COMPRESS_OVERVIEW JPEG -- config PHOTOMETRIC_OVERVIEW
> >>> YCBCR" when there's a .msk didn't work.
> >>> 
>  OK Even, thanks. This means that I should use gdal-svn?
> >>> 
> >>> Not necessarily. I think you can workaround the issue by manually
> >>> building the overview on the .msk file as a post-processing step :
> >>> 
> >>> gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE small.tif.msk 2 4 8 16
> >>> 32 64
> >> 
> >> Do you mean I should use DEFLATE instead of JPEG as compression for
> >> overviews for the time being? What is the difference in compression
> >> ratio?
> > 
> > Use DEFLATE only for the mask file (see small.tif.msk in the above
> > command line), as it is a lossless compression, which is needed for a
> > mask. JPEG is fine for the imagery itself
> 
> Sorry, I don't understand. Where does that small.tif.msk come from?
> Until now I had:
> - the original raster (big.tif)
> - the raster masked with nearblack (big_with_mask.tif)

There should be a big_with_mask.tif.msk generated at that step, alongside 
big_with_mask.tif

> - the compressed raster (gdal_translate -co...)

And gdal_translate should also generate a small.tif.msk, as the input had a 
.msk file.

> - the overviews built with gdaladdo
> 
> The mask is part of the raster file, isn't  it?
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Hare, Trent]

Jan,
   Jpeg is a lossy compression and thus the collar will not be maintained
perfectly as black (or 0). You will not get as much compression, but you
can try a lossless method like LZW or Packbits. If you are getting close to
the 4GB limit you might need to add "-co bigtiff=if_safer".

Good luck,
Trent



On Thu, Dec 27, 2012 at 5:38 AM, Jan Hartmann  wrote:

>  Can anyone tell me wath is happening here? I have a large three-band
> raster (3.8G) looking like this:
>
> http://mapserver.sara.nl/map2a.png
>
> I compress it (from information from this list a few days ago) with:
>
> gdal_translate -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR big.tif small.tif
> gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
> YCBCR -ro -r gauss small.tif 2 4 8 16 32 64
>
> and get a file of 37M, which is nice. However, the map has all kinds of
> rubbish at the borders. It looks like this:
>
> http://mapserver.sara.nl/map2b.png
>
> The equivalent pictures of a small part of the map look like this:
>
> http://mapserver.sara.nl/map1a.png
> http://mapserver.sara.nl/map1b.png
>
> Can anyone help me?
>
> Jan
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression artifacts

[Jan Hartmann]

Thanks Trent, I tried it out:

original:  3.886.894.126b
compressed DEFLATE:266.046.034b
compressed LZW: 372.085.756b
compressed JPEG: 90.055.344b

Although DEFLATE and LZW give very good results, JPEG compression is so 
much better that I need to go for it. The base raster is the cadastral 
map of Amsterdam in 1832 with a resolution of 30 cm. This is only the 
low-resolution scan, the base scan has a resolution of 15 cm, slightly 
better than Google Satellite. We are trying to set things up for the 
whole of the Netherlands, and with this sort of precision even a small 
country generates an awful lot of raster scans, and needs unbelievable 
amounts of processing, even with the Cloud system we have available now.


That means, I'll have to clip the maps. It can be done as Dmitry 
suggested (gdalwarp -cutline), or alternatively using gdal-rasterize. I 
do have the bounding polygons. It can also be done with nearblack, as 
Even suggested. I guess the last method will take somewhat more time, as 
the area of the map has to be computed.


Thanks for all your suggestions, they made me see where the problem 
lies. And a happy New Year to all.


Jan

On 12/27/2012 05:18 PM, Hare, Trent wrote:

Jan,
   Jpeg is a lossy compression and thus the collar will not be 
maintained perfectly as black (or 0). You will not get as much 
compression, but you can try a lossless method like LZW or Packbits. 
If you are getting close to the 4GB limit you might need to add "-co 
bigtiff=if_safer".


Good luck,
Trent



On Thu, Dec 27, 2012 at 5:38 AM, Jan Hartmann > wrote:


Can anyone tell me wath is happening here? I have a large
three-band raster (3.8G) looking like this:

http://mapserver.sara.nl/map2a.png

I compress it (from information from this list a few days ago) with:

gdal_translate -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR big.tif
small.tif
gdaladdo --config COMPRESS_OVERVIEW JPEG --config
PHOTOMETRIC_OVERVIEW YCBCR -ro -r gauss small.tif 2 4 8 16 32 64

and get a file of 37M, which is nice. However, the map has all
kinds of rubbish at the borders. It looks like this:

http://mapserver.sara.nl/map2b.png

The equivalent pictures of a small part of the map look like this:

http://mapserver.sara.nl/map1a.png
http://mapserver.sara.nl/map1b.png

Can anyone help me?

Jan

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org 
http://lists.osgeo.org/mailman/listinfo/gdal-dev




___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression Artifacts After Merging Imagery

[Even Rouault]

On lundi 25 septembre 2017 16:16:26 CEST Jacob Adams wrote:
> Oh ye great minds of the GDAL world, I've run into a problem that I've not
> been able to solve.
> 
> I have aerial imagery comprised of about 1500 jpegs that I am trying to
> merge together into three large (~18gb) jpeg-compressed geotiffs to be
> served by Geoserver. The images are tiled such that they shift down about
> 100 pixels (out of ~7000) every so often. With a pure gdal_merge or
> gdalwarp mosaic, this leaves compression artifacts in the nodata spaces
> along the borders. The cause of these artifacts seems to be explained in
> Evan's comments on this ticket: https://trac.osgeo.org/gdal/ticket/5102).
> 
> I tried pre-processing a handful of the tilesinto geotiffs with the
> following three commands based on a similar discussion in February
> (http://lists.osgeo.org/pipermail/gdal-dev/2017-February/046027.html):

> nearblack -co TILED=YES -of Gtiff -nb 0 -near 0 -setmask -o temp1.tif  
> Tile1.jpeg

I think one issue is in the above nearblack invocation. Did you check temp1.tif 
is "clean" by 
displaying it ?
Basically by setting -near 0, you say that nearblack should expect a very clean 
frontier 
between nodata areas and valid areas. Whereas JPEG decompression artifacts in 
such a 
situation will result in intermediate values in a few pixels beyond the border.
I'd suggest trying with the default of nearblack, -near 15 and -nb 2, and tune 
them if needed.


> gdal_translate -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -b 1 
> -b 2 -b 
3 -mask 4 --config GDAL_TIFF_INTERNAL_MASK YES temp2.tif Tile1_out.tif 

You shouldn't JPEG compress intermediate products. This will decrease image 
quality. Only 
apply it to the final stage of your workflow.


Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression Artifacts After Merging Imagery

[Jacob Adams]

Evan,

Thanks for your help. Adjusting the nearblack parameters and using
-setalpha instead of -setmask got the mask working as expected. 

I'd normally agree with you on mid-process compression, and if I were
using the data for any kind of analysis I would do as little compression
as possible. However, with this project I'm pressed for space and I'm
only trying to create an aerial imagery basemap. I did a quick test
compressing a single tile 10 times with gdal_translate and the output
was adequate for my project. There was an identifiable softening to the
image at the pixel-level, but the imagery was gathered at 3" per pixel.


On a different topic altogether, how would I go about contributing to
the doxygen-created documentation? In playing around with gdal_rasterize
I discovered that it's not clear in the current docs what options
trigger creation of a new raster instead of overwriting an existing
raster. My C++ skills are very rusty, but if I'm reading the source code
properly, the options -of, -init, -a_nodata, -a_srs, -te, -a_ullr, -co,
-ot, -ts, -outsize, -tr, and -tap set the bCreateOutput flag True. I'd
like to add a note explaining this so that someone else doesn't make the
same beginner's mistake I did of trying to specify creation options on
an existing dataset ;) Is this something I open a ticket for, or is
there a more direct way?

Thanks,

Jacob Adams
GIS Specialist, Cache County
 


>>> Even Rouault  9/25/2017 5:01 PM >>>
On lundi 25 septembre 2017 16:16:26 CEST Jacob Adams wrote:
> Oh ye great minds of the GDAL world, I've run into a problem that
I've not
> been able to solve.
> 
> I have aerial imagery comprised of about 1500 jpegs that I am trying
to
> merge together into three large (~18gb) jpeg‑compressed geotiffs to
be
> served by Geoserver. The images are tiled such that they shift down
about
> 100 pixels (out of ~7000) every so often. With a pure gdal_merge or
> gdalwarp mosaic, this leaves compression artifacts in the nodata
spaces
> along the borders. The cause of these artifacts seems to be explained
in
> Evan's comments on this ticket:
https://trac.osgeo.org/gdal/ticket/5102).
> 
> I tried pre‑processing a handful of the tilesinto geotiffs with the
> following three commands based on a similar discussion in February
>
(http://lists.osgeo.org/pipermail/gdal‑dev/2017‑February/046027.html):


> nearblack ‑co TILED=YES ‑of Gtiff ‑nb 0 ‑near 0 ‑setmask ‑o temp1.tif
 Tile1.jpeg

I think one issue is in the above nearblack invocation. Did you check
temp1.tif is "clean" by 
displaying it ?
Basically by setting ‑near 0, you say that nearblack should expect a
very clean frontier 
between nodata areas and valid areas. Whereas JPEG decompression
artifacts in such a 
situation will result in intermediate values in a few pixels beyond the
border.
I'd suggest trying with the default of nearblack, ‑near 15 and ‑nb 2,
and tune them if needed.


> gdal_translate ‑co TILED=YES ‑co COMPRESS=JPEG ‑co
PHOTOMETRIC=YCBCR ‑b 1 ‑b 2 ‑b 
3 ‑mask 4 ‑‑config GDAL_TIFF_INTERNAL_MASK YES temp2.tif Tile1_out.tif


You shouldn't JPEG compress intermediate products. This will decrease
image quality. Only 
apply it to the final stage of your workflow.


Even

‑‑ 
Spatialys ‑ Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Compression Artifacts After Merging Imagery

[Even Rouault]

> On a different topic altogether, how would I go about contributing to
> the doxygen-created documentation? In playing around with gdal_rasterize
> I discovered that it's not clear in the current docs what options
> trigger creation of a new raster instead of overwriting an existing
> raster. My C++ skills are very rusty, but if I'm reading the source code
> properly, the options -of, -init, -a_nodata, -a_srs, -te, -a_ullr, -co,
> -ot, -ts, -outsize, -tr, and -tap set the bCreateOutput flag True. I'd
> like to add a note explaining this so that someone else doesn't make the
> same beginner's mistake I did of trying to specify creation options on
> an existing dataset ;) Is this something I open a ticket for, or is
> there a more direct way?

A ticket is fine. The changes are to be done in apps/gdal_utilities.dox

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev