Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Markus Neteler]

Hi,

just for the record (by Markus M):

  https://github.com/OSGeo/grass/pull/287

markusN
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Markus Metz]

On Thu, Jan 16, 2020 at 10:56 AM Moritz Lennert <
mlenn...@club.worldonline.be> wrote:
>
> Coming back to this to continue the discussion:
>
> On 15/12/19 21:41, Markus Metz wrote:
> >
> >
> > On Tue, Dec 10, 2019 at 6:47 AM Moritz Lennert
> > mailto:mlenn...@club.worldonline.be>>
wrote:
> >  >
> >  > On 9/12/19 22:50, Markus Metz wrote:
> >  > >   Hi Moritz,
> >  > >
> >  > > On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
> >  > > mailto:mlenn...@club.worldonline.be>
> >  > >> wrote:
> >  > >  >
> >  > >  > Hi Markus,
> >  > >  >
> >  > >  > In recent days, I've been confronted several times with the
issue of
> >  > >  > people trying to share data among themselves, but using
different
> >  > >  > versions of GRASS, and so raster data compressed in a more
recent
> >  > >  > version of GRASS was not usable in an older version of GRASS.
> >  > >  >
> >  > >  > Now, I agree that generally the solution is to tell people to
> > use the
> >  > >  > latest and greatest, but this is not always possible / it is not
> >  > >  > necessarily highest on the list of priorities of people to see
> > how they
> >  > >  > can install the latest version of GRASS within their particular
> >  > > environment.
> >  > >  >
> >  > >  > Obviously, those with the latest version of GRASS can simple
> > recompress
> >  > >  > using ZLIB. However, compression method is defined as an
environment
> >  > >  > variable. This is somewhat daunting to many MS Windows users out
> > there.
> >  > >  > Is there any specific reason that lead to the choice of not
using a
> >  > >  > parameter to allow the choice of compression method (possibly to
> >  > >  > override a default that is still defined by an environment
> > variable) ?
> >  > >
> >  > > Such a parameter would need to be added to every module creating
raster
> >  > > output.
> >  >
> >  > My request is more linked to use cases where one would like to share
> >  > data (e.g. with r.pack) with other GRASS GIS users who do not
> >  > necessarily have access to the same compression method, not
necessarily
> >  > to changing the default compression method. I was just wondering
whether
> >  > it might be easily possible to just implement r.compress method= as a
> >  > quick way to recompress a specific map with a chosen method,
overriding
> >  > the default method. Currently, to do that, you have to change the
> >  > default method by changing the env variable, run r.compress, then
change
> >  > the variable back to the value one wishes generally to use as
default.
> >
> > recompressing an existing raster map seems like a waste of time. Why not
> > compress it in the first place with the appropriate method?
>
> Because what you consider the appropriate method at the time creation of
> the data (and still the appropriate method for your actual work) might
> not be the appropriate method for sharing with someone using a different
> system.

True, for sharing the highest (smallest size) compression method is usually
preferred, while for actual work the fastest compression method is usually
preferred.
> >
> > About the use cases:
> > People want to share a mapset: in this case all GRASS raster maps should
> > be compressed with the most compatible compression method (ZLIB).
> > Currently, this can be done with an env var. Alternatively, a new
> > mechanism via g.gisenv could be established.
>
> This would already make life easier for Windows users who (for the
> majority) are not used to env variables (and this is becoming more and
> more true of GNU/Linux users as well...).
>
> > Another option could be to set the default compression method per
> > mapset, to be stored in in the mapset's VAR file, to be set with e.g.
> > r.compress method=.
>
> That would probably be a nice solution. One could then create an
> "Export" mapset which can then be easily shared.

You could do the same by creating some "Export" mapset, then setting the
compression method and creating raster maps to be exported in this "Export"
mapset.

Implementation of a compression method set with g.gisenv is relatively easy.
Implementation of a compression method defined in a mapset's VAR file is a
bit more work and requires a mechanism to set the compression method in a
mapset's VAR file, e.g. r.compress method=.

But, if e.g. the aim is to reduce file size, the best compression method
depends on the datatype and size of the individual raster map to be
compressed. There is no "one size fits all" method.
>
> >  > Obviously, you can always just export as tiff and share that, but
that
> >  > just feels less elegant. Anyway, this is probably somewhat of a
luxury
> >  > problem :-)
> >
> > This only a luxury problem if you don't need to care about processing
> > speed and disk usage. The different compression methods by now available
> > in GRASS allow for optimization for speed (LZ4) or for optimization for
> > disk usage (BZIP2 for larger CELL maps). An 

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Moritz Lennert]

Coming back to this to continue the discussion:

On 15/12/19 21:41, Markus Metz wrote:



On Tue, Dec 10, 2019 at 6:47 AM Moritz Lennert 
mailto:mlenn...@club.worldonline.be>> wrote:

 >
 > On 9/12/19 22:50, Markus Metz wrote:
 > >   Hi Moritz,
 > >
 > > On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
 > > mailto:mlenn...@club.worldonline.be> 
>> wrote:

 > >  >
 > >  > Hi Markus,
 > >  >
 > >  > In recent days, I've been confronted several times with the issue of
 > >  > people trying to share data among themselves, but using different
 > >  > versions of GRASS, and so raster data compressed in a more recent
 > >  > version of GRASS was not usable in an older version of GRASS.
 > >  >
 > >  > Now, I agree that generally the solution is to tell people to 
use the

 > >  > latest and greatest, but this is not always possible / it is not
 > >  > necessarily highest on the list of priorities of people to see 
how they

 > >  > can install the latest version of GRASS within their particular
 > > environment.
 > >  >
 > >  > Obviously, those with the latest version of GRASS can simple 
recompress

 > >  > using ZLIB. However, compression method is defined as an environment
 > >  > variable. This is somewhat daunting to many MS Windows users out 
there.

 > >  > Is there any specific reason that lead to the choice of not using a
 > >  > parameter to allow the choice of compression method (possibly to
 > >  > override a default that is still defined by an environment 
variable) ?

 > >
 > > Such a parameter would need to be added to every module creating raster
 > > output.
 >
 > My request is more linked to use cases where one would like to share
 > data (e.g. with r.pack) with other GRASS GIS users who do not
 > necessarily have access to the same compression method, not necessarily
 > to changing the default compression method. I was just wondering whether
 > it might be easily possible to just implement r.compress method= as a
 > quick way to recompress a specific map with a chosen method, overriding
 > the default method. Currently, to do that, you have to change the
 > default method by changing the env variable, run r.compress, then change
 > the variable back to the value one wishes generally to use as default.

recompressing an existing raster map seems like a waste of time. Why not 
compress it in the first place with the appropriate method?


Because what you consider the appropriate method at the time creation of 
the data (and still the appropriate method for your actual work) might 
not be the appropriate method for sharing with someone using a different 
system.




Recompressing an existing map with a chosen method via r.compress 
method=XY means that a new raster map should not be compressed with the 
default method but another supported method. This would require a new 
function in lib/raster like Rast_open_new_compression_method() that 
overrides the default compression method. Of course this is possible, 
but it requires substantial changes to lib/raster.


Ok, I understand that this is complicated.



About the use cases:
People want to share a mapset: in this case all GRASS raster maps should 
be compressed with the most compatible compression method (ZLIB). 
Currently, this can be done with an env var. Alternatively, a new 
mechanism via g.gisenv could be established.


This would already make life easier for Windows users who (for the 
majority) are not used to env variables (and this is becoming more and 
more true of GNU/Linux users as well...).


Another option could be to set the default compression method per 
mapset, to be stored in in the mapset's VAR file, to be set with e.g. 
r.compress method=.


That would probably be a nice solution. One could then create an 
"Export" mapset which can then be easily shared.



 > Obviously, you can always just export as tiff and share that, but that
 > just feels less elegant. Anyway, this is probably somewhat of a luxury
 > problem :-)

This only a luxury problem if you don't need to care about processing 
speed and disk usage. The different compression methods by now available 
in GRASS allow for optimization for speed (LZ4) or for optimization for 
disk usage (BZIP2 for larger CELL maps). An easy (easier) way to set the 
compression method seems reasonable.


I didn't mean that compression is a luxury problem, but rather that 
since the option exists of just exporting the map as a tiff, readable by 
most systems (and you can chose a sharable tiff compression method at 
export time), all the above is probably a bit less of a necessity, and 
I'm just making noise ;-)


Moritz


___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Markus Metz]

On Tue, Dec 10, 2019 at 6:47 AM Moritz Lennert 
wrote:
>
> On 9/12/19 22:50, Markus Metz wrote:
> >   Hi Moritz,
> >
> > On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
> > mailto:mlenn...@club.worldonline.be>>
wrote:
> >  >
> >  > Hi Markus,
> >  >
> >  > In recent days, I've been confronted several times with the issue of
> >  > people trying to share data among themselves, but using different
> >  > versions of GRASS, and so raster data compressed in a more recent
> >  > version of GRASS was not usable in an older version of GRASS.
> >  >
> >  > Now, I agree that generally the solution is to tell people to use the
> >  > latest and greatest, but this is not always possible / it is not
> >  > necessarily highest on the list of priorities of people to see how
they
> >  > can install the latest version of GRASS within their particular
> > environment.
> >  >
> >  > Obviously, those with the latest version of GRASS can simple
recompress
> >  > using ZLIB. However, compression method is defined as an environment
> >  > variable. This is somewhat daunting to many MS Windows users out
there.
> >  > Is there any specific reason that lead to the choice of not using a
> >  > parameter to allow the choice of compression method (possibly to
> >  > override a default that is still defined by an environment variable)
?
> >
> > Such a parameter would need to be added to every module creating raster
> > output.
>
> My request is more linked to use cases where one would like to share
> data (e.g. with r.pack) with other GRASS GIS users who do not
> necessarily have access to the same compression method, not necessarily
> to changing the default compression method. I was just wondering whether
> it might be easily possible to just implement r.compress method= as a
> quick way to recompress a specific map with a chosen method, overriding
> the default method. Currently, to do that, you have to change the
> default method by changing the env variable, run r.compress, then change
> the variable back to the value one wishes generally to use as default.

recompressing an existing raster map seems like a waste of time. Why not
compress it in the first place with the appropriate method?

Recompressing an existing map with a chosen method via r.compress method=XY
means that a new raster map should not be compressed with the default
method but another supported method. This would require a new function in
lib/raster like Rast_open_new_compression_method() that overrides the
default compression method. Of course this is possible, but it requires
substantial changes to lib/raster.

About the use cases:
People want to share a mapset: in this case all GRASS raster maps should be
compressed with the most compatible compression method (ZLIB). Currently,
this can be done with an env var. Alternatively, a new mechanism via
g.gisenv could be established.
Another option could be to set the default compression method per mapset,
to be stored in in the mapset's VAR file, to be set with e.g. r.compress
method=.

The point is that it is fairly easy to set a suitable compression method
before the raster lib is initialized, and there are different possible
mechanisms apart from the current method using an env var that are easy to
implement.

> Obviously, you can always just export as tiff and share that, but that
> just feels less elegant. Anyway, this is probably somewhat of a luxury
> problem :-)

This only a luxury problem if you don't need to care about processing speed
and disk usage. The different compression methods by now available in GRASS
allow for optimization for speed (LZ4) or for optimization for disk usage
(BZIP2 for larger CELL maps). An easy (easier) way to set the compression
method seems reasonable.

Markus M

>
> Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Huidae Cho]

One workaround may be GRASS_COMPRESSOR=method r.compress, but it won't
directly work on MS-Windows without a batch file, I think.

Huidae

On Tue, Dec 10, 2019 at 12:47 AM Moritz Lennert <
mlenn...@club.worldonline.be> wrote:

> On 9/12/19 22:50, Markus Metz wrote:
> >   Hi Moritz,
> >
> > On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
> > mailto:mlenn...@club.worldonline.be>>
> wrote:
> >  >
> >  > Hi Markus,
> >  >
> >  > In recent days, I've been confronted several times with the issue of
> >  > people trying to share data among themselves, but using different
> >  > versions of GRASS, and so raster data compressed in a more recent
> >  > version of GRASS was not usable in an older version of GRASS.
> >  >
> >  > Now, I agree that generally the solution is to tell people to use the
> >  > latest and greatest, but this is not always possible / it is not
> >  > necessarily highest on the list of priorities of people to see how
> they
> >  > can install the latest version of GRASS within their particular
> > environment.
> >  >
> >  > Obviously, those with the latest version of GRASS can simple
> recompress
> >  > using ZLIB. However, compression method is defined as an environment
> >  > variable. This is somewhat daunting to many MS Windows users out
> there.
> >  > Is there any specific reason that lead to the choice of not using a
> >  > parameter to allow the choice of compression method (possibly to
> >  > override a default that is still defined by an environment variable) ?
> >
> > Such a parameter would need to be added to every module creating raster
> > output.
>
> My request is more linked to use cases where one would like to share
> data (e.g. with r.pack) with other GRASS GIS users who do not
> necessarily have access to the same compression method, not necessarily
> to changing the default compression method. I was just wondering whether
> it might be easily possible to just implement r.compress method= as a
> quick way to recompress a specific map with a chosen method, overriding
> the default method. Currently, to do that, you have to change the
> default method by changing the env variable, run r.compress, then change
> the variable back to the value one wishes generally to use as default.
>
> Obviously, you can always just export as tiff and share that, but that
> just feels less elegant. Anyway, this is probably somewhat of a luxury
> problem :-)
>
> Moritz
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev



-- 
Huidae Cho, Ph.D., GISP, PE (MD), CFM, M.ASCE
Open Source GIS Developer, GRASS GIS Development Team
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Moritz Lennert]

On 9/12/19 22:50, Markus Metz wrote:

  Hi Moritz,

On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert 
mailto:mlenn...@club.worldonline.be>> wrote:

 >
 > Hi Markus,
 >
 > In recent days, I've been confronted several times with the issue of
 > people trying to share data among themselves, but using different
 > versions of GRASS, and so raster data compressed in a more recent
 > version of GRASS was not usable in an older version of GRASS.
 >
 > Now, I agree that generally the solution is to tell people to use the
 > latest and greatest, but this is not always possible / it is not
 > necessarily highest on the list of priorities of people to see how they
 > can install the latest version of GRASS within their particular 
environment.

 >
 > Obviously, those with the latest version of GRASS can simple recompress
 > using ZLIB. However, compression method is defined as an environment
 > variable. This is somewhat daunting to many MS Windows users out there.
 > Is there any specific reason that lead to the choice of not using a
 > parameter to allow the choice of compression method (possibly to
 > override a default that is still defined by an environment variable) ?

Such a parameter would need to be added to every module creating raster 
output.


My request is more linked to use cases where one would like to share 
data (e.g. with r.pack) with other GRASS GIS users who do not 
necessarily have access to the same compression method, not necessarily 
to changing the default compression method. I was just wondering whether 
it might be easily possible to just implement r.compress method= as a 
quick way to recompress a specific map with a chosen method, overriding 
the default method. Currently, to do that, you have to change the 
default method by changing the env variable, run r.compress, then change 
the variable back to the value one wishes generally to use as default.


Obviously, you can always just export as tiff and share that, but that 
just feels less elegant. Anyway, this is probably somewhat of a luxury 
problem :-)


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Markus Metz]

 Hi Moritz,

On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert 
wrote:
>
> Hi Markus,
>
> In recent days, I've been confronted several times with the issue of
> people trying to share data among themselves, but using different
> versions of GRASS, and so raster data compressed in a more recent
> version of GRASS was not usable in an older version of GRASS.
>
> Now, I agree that generally the solution is to tell people to use the
> latest and greatest, but this is not always possible / it is not
> necessarily highest on the list of priorities of people to see how they
> can install the latest version of GRASS within their particular
environment.
>
> Obviously, those with the latest version of GRASS can simple recompress
> using ZLIB. However, compression method is defined as an environment
> variable. This is somewhat daunting to many MS Windows users out there.
> Is there any specific reason that lead to the choice of not using a
> parameter to allow the choice of compression method (possibly to
> override a default that is still defined by an environment variable) ?

Such a parameter would need to be added to every module creating raster
output.

The reason for an environment variable is historical: previously, CELL maps
were by default compressed with RLE, this could be changed with the env var
GRASS_INT_ZLIB. This mechanism of using an env var for raster compression
has been extended.

Other solutions than an env var would be to use g.gisenv/$GISRC to set the
default compression method per user or a mapset-specific compression method
(to be set with a modified r.compress?) stored in the mapset's VAR file.

Regarding MS Windows users + raster compression: I think it is reasonable
to assume at least GRASS 7.6 which has support all compression methods
available in GRASS.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Moritz Lennert]

Le Thu, 5 Dec 2019 08:30:43 +0200,
Maris Nartiss  a écrit :

> What should be the solution? Moving to choice set by g.gisenv?

As mentioned in my mail, I suggest to add a parameter to the module
which allows to override the default setting when compressing a
specific raster map.

Using a GRASS environment variable instead of a shell variable might be
an idea to make setting the default easier.

Moritz


> 
> trešd., 2019. g. 4. dec., plkst. 11:28 — lietotājs Moritz Lennert
> () rakstīja:
> >
> > Hi Markus,
> >
> > In recent days, I've been confronted several times with the issue of
> > people trying to share data among themselves, but using different
> > versions of GRASS, and so raster data compressed in a more recent
> > version of GRASS was not usable in an older version of GRASS.
> >
> > Now, I agree that generally the solution is to tell people to use
> > the latest and greatest, but this is not always possible / it is not
> > necessarily highest on the list of priorities of people to see how
> > they can install the latest version of GRASS within their
> > particular environment.
> >
> > Obviously, those with the latest version of GRASS can simple
> > recompress using ZLIB. However, compression method is defined as an
> > environment variable. This is somewhat daunting to many MS Windows
> > users out there. Is there any specific reason that lead to the
> > choice of not using a parameter to allow the choice of compression
> > method (possibly to override a default that is still defined by an
> > environment variable) ?
> >
> > Moritz
> >
> >
> > ___
> > grass-dev mailing list
> > grass-dev@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/grass-dev  



-- 
Département Géosciences, Environnement et Société Université Libre de
Bruxelles Bureau: S.DB.6.138
CP 130/03
Av. F.D. Roosevelt 50
1050 Bruxelles
Belgique

tél. + 32 2 650.68.12 / 68.11 (secr.)
fax  + 32 2 650.68.30 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Maris Nartiss]

What should be the solution? Moving to choice set by g.gisenv?

Māris.

trešd., 2019. g. 4. dec., plkst. 11:28 — lietotājs Moritz Lennert
() rakstīja:
>
> Hi Markus,
>
> In recent days, I've been confronted several times with the issue of
> people trying to share data among themselves, but using different
> versions of GRASS, and so raster data compressed in a more recent
> version of GRASS was not usable in an older version of GRASS.
>
> Now, I agree that generally the solution is to tell people to use the
> latest and greatest, but this is not always possible / it is not
> necessarily highest on the list of priorities of people to see how they
> can install the latest version of GRASS within their particular environment.
>
> Obviously, those with the latest version of GRASS can simple recompress
> using ZLIB. However, compression method is defined as an environment
> variable. This is somewhat daunting to many MS Windows users out there.
> Is there any specific reason that lead to the choice of not using a
> parameter to allow the choice of compression method (possibly to
> override a default that is still defined by an environment variable) ?
>
> Moritz
>
>
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Moritz Lennert]

Hi Markus,

In recent days, I've been confronted several times with the issue of 
people trying to share data among themselves, but using different 
versions of GRASS, and so raster data compressed in a more recent 
version of GRASS was not usable in an older version of GRASS.


Now, I agree that generally the solution is to tell people to use the 
latest and greatest, but this is not always possible / it is not 
necessarily highest on the list of priorities of people to see how they 
can install the latest version of GRASS within their particular environment.


Obviously, those with the latest version of GRASS can simple recompress 
using ZLIB. However, compression method is defined as an environment 
variable. This is somewhat daunting to many MS Windows users out there. 
Is there any specific reason that lead to the choice of not using a 
parameter to allow the choice of compression method (possibly to 
override a default that is still defined by an environment variable) ?


Moritz


___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev