Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
Vaclav Petras wenzesl...@gmail.com writes: On Thu, Jun 4, 2015 at 3:25 PM, Rainer M Krug rai...@krugs.de wrote: [1, 2] The code now allows to introduce such interface. The challenge is where to store the information about active Location and Mapset. Currently, .gisrc (or .grassrc or .grassgisrc) file stored in the current working directory seems as the best option to me. Another option is to use environment variables [3, 4] but I don't consider it a nice solution. I agree - unless it is in a scrip with shebang #!grass71exec or similar. Then it would be easy to use environmental variables. Let me clarify here, GRASS modules read Mapset from a gisrc file. The path to this file is stored in `GISRC` environment variable. The standard interface now creates this file (its content) based on what was provided in the command line. OK - got this one. In the subcommand interface, only the first command knows the Mapset and it must pass it to commands called later. The way how to do this easily is to create a file with a given name in the current directory (e.g., .grassrc). When grass command is called with subcommand (other than setmapset), it looks for that file, if it is there, GISRC is set to this file, if it is not there, it fails. OK. So a command without the mapset specified shoud fail - but it doesn't, if a .grass7/ exists with an rc file in it which holds the mapset to be used. I think that interaction between grass session usage of mapset and grass --exec should be separated and --exec should have it's own way of storing the mapset to be used to avoid interfere with an interactive grass70 session. By doing this, one could also, if I am not mistaken, use , | grass71 --exec ... ` for a different mapset inside of a running interactive grass71 session. The environment variables I was mentioning are different thing. They wouldn't be (cannot) be used inside the GRASS session (so no usage for them in shebang mode). If the grass program would respect them, you can get poor man's subcommand-like interface without implementing it: export GRASS_FULL_MAPSET=/path/to/mapset grass [--mapset-from-environment] --exec r.univar map=elevation I don't think this is how the API should look like and the subcommand interface seems more natural (or just modern?) and more extensible (but do we need it?). No - this does not look optimal - agreed. I think one should aim for a syntax which does not limit the further developments - and the sub command syntax really is widely used successfully. I see many possibilities for future developments like e.g. defining particular default values for a session. What about a user specified name? This would make switching mapsets in a script much more symbolic, as one does not has to specify the path of the mapset and just an easy to remember name? This would make it possible to setup a whole bunch of mapsets associated with names in a text file and then just load the .grassEXEC file? Mapset is the data. Should the data be specified in a config file or some general file and then passed to the program? I don't think so. I think one should pass the Mapset as a parameter. I was somehow never to happy about using parameter in scripts, as this somehow makes it less reproducible. When one uses a file as input / config for the session, it can be put under VC, the same for the grass script, and I can easily go back in time. With parameters one needs an additional layer of a script which calls the R script with the parameter and put that one under VC. So I think that the file approach is more reproducible and, in the longer run, more extensible. But it should be possible to specify the data (including mapset) in the script as well by using the , | grass71 set mapset /PATH/TO/THE/MAPSET ` For the file one could use a syntax like , | grass71 setFromFile /PATH/TO/THE/FILE/WHATEVER.conf ` If a file with a default name exists in the working directory, that one is loaded when doing , | grass71 start ` or to start with a specific file , | grass71 start /PATH/TO/THE/FILE/WHATEVER.conf ` or to just specify a mapset , | grass71 start /PATH/TO/MAPSET ` could be used to start the session. obviously , | grass71 start $1 ` as well where $1 could be the config file or the mapset. Initially, the file would contain only the mapset, but further on all settings for which set can be used (default output format for maps comes to mind, interpreter possibly as well?). To your suggestion of symbolic names. It might be easier to switch but it would complicate the API and implementation a lot. How the file would be managed? I don't think it is worth the trouble. Also I don't want to implement interfaces which don't have precedence. An existing example of something like that could change my opinion about a nickname file but I'm unsure how this would fit with the rest. Was just an idea - not
Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
Hi Vaclav, first of all, thank you for your great work on this I really like it. Do you think that would be possible to avoid to parse manually the sys.argv and use the argparse library? I know that the argparse is not present in python2.6, but people that are still using this old version of the python interpreter can install this library: https://pypi.python.org/pypi/argparse I think this change could help to further simplify the code and make it easier to expand/extend in the future, splitting the command line interface from the logic. Then we could define a function that return the default GRASS parser instance and the main function to apply the logic on what to do with the parsed command line. What do you think? All the best. Pietro ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
On Thu, Jun 4, 2015 at 3:25 PM, Rainer M Krug rai...@krugs.de wrote: [1, 2] The code now allows to introduce such interface. The challenge is where to store the information about active Location and Mapset. Currently, .gisrc (or .grassrc or .grassgisrc) file stored in the current working directory seems as the best option to me. Another option is to use environment variables [3, 4] but I don't consider it a nice solution. I agree - unless it is in a scrip with shebang #!grass71exec or similar. Then it would be easy to use environmental variables. Let me clarify here, GRASS modules read Mapset from a gisrc file. The path to this file is stored in `GISRC` environment variable. The standard interface now creates this file (its content) based on what was provided in the command line. In the subcommand interface, only the first command knows the Mapset and it must pass it to commands called later. The way how to do this easily is to create a file with a given name in the current directory (e.g., .grassrc). When grass command is called with subcommand (other than setmapset), it looks for that file, if it is there, GISRC is set to this file, if it is not there, it fails. The environment variables I was mentioning are different thing. They wouldn't be (cannot) be used inside the GRASS session (so no usage for them in shebang mode). If the grass program would respect them, you can get poor man's subcommand-like interface without implementing it: export GRASS_FULL_MAPSET=/path/to/mapset grass [--mapset-from-environment] --exec r.univar map=elevation I don't think this is how the API should look like and the subcommand interface seems more natural (or just modern?) and more extensible (but do we need it?). What about a user specified name? This would make switching mapsets in a script much more symbolic, as one does not has to specify the path of the mapset and just an easy to remember name? This would make it possible to setup a whole bunch of mapsets associated with names in a text file and then just load the .grassEXEC file? Mapset is the data. Should the data be specified in a config file or some general file and then passed to the program? I don't think so. I think one should pass the Mapset as a parameter. To your suggestion of symbolic names. It might be easier to switch but it would complicate the API and implementation a lot. How the file would be managed? I don't think it is worth the trouble. Also I don't want to implement interfaces which don't have precedence. An existing example of something like that could change my opinion about a nickname file but I'm unsure how this would fit with the rest. [1] https://trac.osgeo.org/grass/ticket/2579#Additionalideas [2] https://trac.osgeo.org/grass/ticket/2579#comment:10 [3] https://trac.osgeo.org/grass/ticket/2679 [4] https://trac.osgeo.org/grass/ticket/2681 , | grass71 --setmapset .../grassdata/test1/PERMANENT/ | grass71 --exec r.external input=basins.tiff output=basins | grass71 --exec r.external input=elevation.tiff output=elevation | grass71 --unsetmapset ` Rather than this, I prefer subcommand interface as known from git, docker or apt-get: , | grass71 setmapset .../grassdata/test1/PERMANENT/ | grass71 run r.external input=basins.tiff output=basins | grass71 run r.external input=elevation.tiff output=elevation | grass71 unsetmapset ` Looks much nicer - I agree. Which subcommands and how to name them is of course a separate question. I'm quite comfortable with `run` but not so much with `setmapset` and `unsetmapset`. They were just descriptive names. And if you want to stick with the subcommand options, something like , | grass71 set mapset .../grassdata/test1/PERMANENT/ | grass71 run r.external input=basins.tiff output=basins | grass71 run r.external input=elevation.tiff output=elevation | grass71 unset mapset ` The decision if to nest subcommands or not is one of the main disadvantages of this kind of interface. I'm quite unsure about the syntax you proposed. Perhaps something simpler like: , | grass71 start .../grassdata/test1/PERMANENT/ | grass71 run r.external input=basins.tiff output=basins | grass71 run r.external input=elevation.tiff output=elevation | grass71 stop ` would be nicer and simpler. Then one could also use , | grass71 set mapset ./input ` where input would be a file which contains something like , | mapset = .../grassdata/test1/PERMANENT/ ` This would make it possible to easily run the same script for different mapsets. I don't see any advantage in that. You should pass the the Mapset to your script as a parameter instead of hardcoding path to file in the script and then changing the file. If you script has a lot of parameters then some kind of config file is appropriate but this should be solved by the script itself. In script (naive bash): , | grass71 start $1 ` Using
Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
Thank you all for the comments. Here are some more from me. On Thu, Jun 4, 2015 at 3:53 AM, Rainer M Krug rai...@krugs.de wrote: Vaclav Petras wenzesl...@gmail.com writes: Hi all, I'm happy to announce new command line interface of the `grass` program [1, 2] available in trunk. This sounds very exciting - and will make life much easier. Thanks a lot. It might be to early to ask, but are there any penalties is speed when using this construct compared with the traditional way? It is not too early. The time you have to currently pay is approximately 0.05 s. However, you pay for having the full session with all checks, cleanups and Mapset locking. Although the execution time is much longer than when just setting the variables manually (when you don't execute any cleanups, for example), the time is relatively short when running some computation. Unfortunately, the price has to be paid every time you call a module. Then the price can grow especially if you call something many times (e.g. in a loop). This can be dealt with by not executing individual modules directly and creating a script (or module) for GRASS instead. Although this is less convenient then direct execution of modules, it brings advantages of writing Python script with GRASS Python API, script which can become true GRASS module and finally, and finally it separates GRASS-related functionality from your script (separation is good for maintenance and reuse). In any case, the goal is to keep the time as low as possible to support as many use cases as possible. And here is my simple benchmark: time grass71 ~/grassdata/nc_spm_08_grass7/PERMANENT/ --exec echo from GRASS real0m0.069s user0m0.048s sys 0m0.008s Certainly you already know that you can start GRASS GIS session in the specified Location and Mapset in the following way: grass71 .../grassdata/location/mapset/ The new interface adds `--exec` flag for specifying a command which will be executed inside that GRASS session: grass71 .../grassdata/location/mapset/ --exec r.univar map=elevation Am I right in assuming, that the --exec effectively splits the command into the setup part (mapset) before and the command part after and whatever follows will be executed? Exactly, it splits the command line: standard `grass` program parameters are before --exec, command to execute is after. What happens when one specifies a shell command - I assume it will it also be executed? Right, anything is executed, e.g. env RScript... python -c import grass.script as gscript; gscript.run_command('g.region', flags='g') r.external --ui g.gui -f Just note that the shell syntax is interpreted in the current shell (e.g. $VARIABLE), `grass` program gets just plain parameters to execute. I guess that if this is working, the next step would be to introduce the single command version so that one does not have to specify the mapset each time. E.g: Yes! [1, 2] The code now allows to introduce such interface. The challenge is where to store the information about active Location and Mapset. Currently, .gisrc (or .grassrc or .grassgisrc) file stored in the current working directory seems as the best option to me. Another option is to use environment variables [3, 4] but I don't consider it a nice solution. [1] https://trac.osgeo.org/grass/ticket/2579#Additionalideas [2] https://trac.osgeo.org/grass/ticket/2579#comment:10 [3] https://trac.osgeo.org/grass/ticket/2679 [4] https://trac.osgeo.org/grass/ticket/2681 , | grass71 --setmapset .../grassdata/test1/PERMANENT/ | grass71 --exec r.external input=basins.tiff output=basins | grass71 --exec r.external input=elevation.tiff output=elevation | grass71 --unsetmapset ` Rather than this, I prefer subcommand interface as known from git, docker or apt-get: , | grass71 setmapset .../grassdata/test1/PERMANENT/ | grass71 run r.external input=basins.tiff output=basins | grass71 run r.external input=elevation.tiff output=elevation | grass71 unsetmapset ` Which subcommands and how to name them is of course a separate question. I'm quite comfortable with `run` but not so much with `setmapset` and `unsetmapset`. and finally introduce a mapper around this (similar to Rscript ore littler) so that one could say , | grass71exec setmapset .../grassdata/test1/PERMANENT/ | grass71exec r.external input=basins.tiff output=basins | grass71exec r.external input=elevation.tiff output=elevation | grass71exec unsetmapset ` I don't see much advantage in this comparing to grass71 --exec for the usages other than shebang which requires special approach anyway (see below). and could use grass71exec as a shebang - that would be the ultimate convenience. This would make sense. GRASS looks like an interpreter by requiring things to run inside, so it should behave as an interpreter. Shebang may require to have a special wrapper to set some parameters (since shebang allows just one short parameter and no
Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
Hi Vaclav, That sounds very, very useful and will definitely simplify some of the data-maintenance scripts we run regularly! Thanks for your work. I will test asap. Cheers Stefan From: grass-user-boun...@lists.osgeo.org [mailto:grass-user-boun...@lists.osgeo.org] On Behalf Of Vaclav Petras Sent: 4. juni 2015 02:44 To: GRASS user list Subject: [GRASS-user] Announcing new command line interface of `grass` program in trunk Hi all, I'm happy to announce new command line interface of the `grass` program [1, 2] available in trunk. Certainly you already know that you can start GRASS GIS session in the specified Location and Mapset in the following way: grass71 .../grassdata/location/mapset/ The new interface adds `--exec` flag for specifying a command which will be executed inside that GRASS session: grass71 .../grassdata/location/mapset/ --exec r.univar map=elevation This starts GRASS session, executes whatever command is specified after `--exec` flag and then ends showing the (text) output of the module as if the module would be executed manually in the GRASS session. And example showing this in combination with r.external to get the data into the Mapset is in the documentation [3]. This interface is meant to remove the need for lengthy manual setup of environmental variables followed by execution of GRASS modules in the mock-up GRASS session [4]. The idea is to use correct GRASS session maintained by `grass` program which means that any GRASS-related code must be passed to the `grass` program as opposed to direct execution in the former case. The interface can run any module or custom script with or without parameters. GUI tools including g.gui [5] can be called as well which opens new possibilities for application developers. The interface is the first implementation and currently is fully operational only on Linux and similar systems but it benefits from a lot of stable code which was already in place. Try it out and feel free to comment here, on grass-dev or in the related ticket [6]. There is definitively many potential improvements and it would be great to know what the community demands. Vaclav [1] https://trac.osgeo.org/grass/changeset/65252 [2] https://trac.osgeo.org/grass/changeset/65294 [3] http://grass.osgeo.org/grass71/manuals/grass7.html#exec-interface-example [4] http://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly [5] https://trac.osgeo.org/grass/changeset/65306 [6] https://trac.osgeo.org/grass/ticket/2579 ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
Great news ! thank you Vaclav for this new step in gis system integration : Shell one-liners surely will enjoy... Vincent Le mercredi 03 juin 2015 à 20:44 -0400, Vaclav Petras a écrit : Hi all, I'm happy to announce new command line interface of the `grass` program [1, 2] available in trunk. Certainly you already know that you can start GRASS GIS session in the specified Location and Mapset in the following way: grass71 .../grassdata/location/mapset/ The new interface adds `--exec` flag for specifying a command which will be executed inside that GRASS session: grass71 .../grassdata/location/mapset/ --exec r.univar map=elevation This starts GRASS session, executes whatever command is specified after `--exec` flag and then ends showing the (text) output of the module as if the module would be executed manually in the GRASS session. And example showing this in combination with r.external to get the data into the Mapset is in the documentation [3]. This interface is meant to remove the need for lengthy manual setup of environmental variables followed by execution of GRASS modules in the mock-up GRASS session [4]. The idea is to use correct GRASS session maintained by `grass` program which means that any GRASS-related code must be passed to the `grass` program as opposed to direct execution in the former case. The interface can run any module or custom script with or without parameters. GUI tools including g.gui [5] can be called as well which opens new possibilities for application developers. The interface is the first implementation and currently is fully operational only on Linux and similar systems but it benefits from a lot of stable code which was already in place. Try it out and feel free to comment here, on grass-dev or in the related ticket [6]. There is definitively many potential improvements and it would be great to know what the community demands. Vaclav [1] https://trac.osgeo.org/grass/changeset/65252 [2] https://trac.osgeo.org/grass/changeset/65294 [3] http://grass.osgeo.org/grass71/manuals/grass7.html#exec-interface-example [4] http://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly [5] https://trac.osgeo.org/grass/changeset/65306 [6] https://trac.osgeo.org/grass/ticket/2579 ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
Vaclav Petras wenzesl...@gmail.com writes: Hi all, I'm happy to announce new command line interface of the `grass` program [1, 2] available in trunk. This sounds very exciting - and will make life much easier. Thanks a lot. It might be to early to ask, but are there any penalties is speed when using this construct compared with the traditional way? Certainly you already know that you can start GRASS GIS session in the specified Location and Mapset in the following way: grass71 .../grassdata/location/mapset/ The new interface adds `--exec` flag for specifying a command which will be executed inside that GRASS session: grass71 .../grassdata/location/mapset/ --exec r.univar map=elevation Am I right in assuming, that the --exec effectively splits the command into the setup part (mapset) before and the command part after and whatever follows will be executed? What happens when one specifies a shell command - I assume it will it also be executed? I guess that if this is working, the next step would be to introduce the single command version so that one does not have to specify the mapset each time. E.g: , | grass71 --setmapset .../grassdata/test1/PERMANENT/ | grass71 --exec r.external input=basins.tiff output=basins | grass71 --exec r.external input=elevation.tiff output=elevation | grass71 --unsetmapset ` , | grass71 --setmapset .../grassdata/test1/PERMANENT/ | grass71 --exec r.external input=basins.tiff output=basins | grass71 --exec r.external input=elevation.tiff output=elevation | grass71 --unsetmapset ` and finally introduce a mapper around this (similar to Rscript ore littler) so that one could say , | grass71exec setmapset .../grassdata/test1/PERMANENT/ | grass71exec r.external input=basins.tiff output=basins | grass71exec r.external input=elevation.tiff output=elevation | grass71exec unsetmapset ` and could use grass71exec as a shebang - that would be the ultimate convenience. Thanks a lot, Rainer This starts GRASS session, executes whatever command is specified after `--exec` flag and then ends showing the (text) output of the module as if the module would be executed manually in the GRASS session. And example showing this in combination with r.external to get the data into the Mapset is in the documentation [3]. This interface is meant to remove the need for lengthy manual setup of environmental variables followed by execution of GRASS modules in the mock-up GRASS session [4]. The idea is to use correct GRASS session maintained by `grass` program which means that any GRASS-related code must be passed to the `grass` program as opposed to direct execution in the former case. The interface can run any module or custom script with or without parameters. GUI tools including g.gui [5] can be called as well which opens new possibilities for application developers. The interface is the first implementation and currently is fully operational only on Linux and similar systems but it benefits from a lot of stable code which was already in place. Try it out and feel free to comment here, on grass-dev or in the related ticket [6]. There is definitively many potential improvements and it would be great to know what the community demands. Vaclav [1] https://trac.osgeo.org/grass/changeset/65252 [2] https://trac.osgeo.org/grass/changeset/65294 [3] http://grass.osgeo.org/grass71/manuals/grass7.html#exec-interface-example [4] http://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly [5] https://trac.osgeo.org/grass/changeset/65306 [6] https://trac.osgeo.org/grass/ticket/2579 ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user -- Rainer M. Krug email: Raineratkrugsdotde PGP: 0x0F52F982 signature.asc Description: PGP signature ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Announcing new command line interface of `grass` program in trunk
Vaclav Petras wenzesl...@gmail.com writes: Thank you all for the comments. Here are some more from me. On Thu, Jun 4, 2015 at 3:53 AM, Rainer M Krug rai...@krugs.de wrote: Vaclav Petras wenzesl...@gmail.com writes: Hi all, I'm happy to announce new command line interface of the `grass` program [1, 2] available in trunk. This sounds very exciting - and will make life much easier. Thanks a lot. It might be to early to ask, but are there any penalties is speed when using this construct compared with the traditional way? It is not too early. The time you have to currently pay is approximately 0.05 s. However, you pay for having the full session with all checks, cleanups and Mapset locking. Although the execution time is much longer than when just setting the variables manually (when you don't execute any cleanups, for example), the time is relatively short when running some computation. 0.05s is not to bad ion most cases, as the computation of the command time will usually be much longer. Unfortunately, the price has to be paid every time you call a module. Then the price can grow especially if you call something many times (e.g. in a loop). This can be dealt with by not executing individual modules directly and creating a script (or module) for GRASS instead. Although this is less convenient then direct execution of modules, it brings advantages of writing Python script with GRASS Python API, script which can become true GRASS module and finally, and finally it separates GRASS-related functionality from your script (separation is good for maintenance and reuse). In any case, the goal is to keep the time as low as possible to support as many use cases as possible. That leads to the shebang point below - I will come back to it later. And here is my simple benchmark: time grass71 ~/grassdata/nc_spm_08_grass7/PERMANENT/ --exec echo from GRASS real0m0.069s user0m0.048s sys 0m0.008s Thanks. Certainly you already know that you can start GRASS GIS session in the specified Location and Mapset in the following way: grass71 .../grassdata/location/mapset/ The new interface adds `--exec` flag for specifying a command which will be executed inside that GRASS session: grass71 .../grassdata/location/mapset/ --exec r.univar map=elevation Am I right in assuming, that the --exec effectively splits the command into the setup part (mapset) before and the command part after and whatever follows will be executed? Exactly, it splits the command line: standard `grass` program parameters are before --exec, command to execute is after. Perfect. What happens when one specifies a shell command - I assume it will it also be executed? Right, anything is executed, e.g. env RScript... python -c import grass.script as gscript; gscript.run_command('g.region', flags='g') r.external --ui g.gui -f Just note that the shell syntax is interpreted in the current shell (e.g. $VARIABLE), `grass` program gets just plain parameters to execute. Good to know and might be useful (I just don't know for what, at the moment - but I am sure there is something...). I guess that if this is working, the next step would be to introduce the single command version so that one does not have to specify the mapset each time. E.g: Yes! Great. [1, 2] The code now allows to introduce such interface. The challenge is where to store the information about active Location and Mapset. Currently, .gisrc (or .grassrc or .grassgisrc) file stored in the current working directory seems as the best option to me. Another option is to use environment variables [3, 4] but I don't consider it a nice solution. I agree - unless it is in a scrip with shebang #!grass71exec or similar. Then it would be easy to use environmental variables. What about a user specified name? This would make switching mapsets in a script much more symbolic, as one does not has to specify the path of the mapset and just an easy to remember name? This would make it possible to setup a whole bunch of mapsets associated with names in a text file and then just load the .grassEXEC file? [1] https://trac.osgeo.org/grass/ticket/2579#Additionalideas [2] https://trac.osgeo.org/grass/ticket/2579#comment:10 [3] https://trac.osgeo.org/grass/ticket/2679 [4] https://trac.osgeo.org/grass/ticket/2681 , | grass71 --setmapset .../grassdata/test1/PERMANENT/ | grass71 --exec r.external input=basins.tiff output=basins | grass71 --exec r.external input=elevation.tiff output=elevation | grass71 --unsetmapset ` Rather than this, I prefer subcommand interface as known from git, docker or apt-get: , | grass71 setmapset .../grassdata/test1/PERMANENT/ | grass71 run r.external input=basins.tiff output=basins | grass71 run r.external input=elevation.tiff output=elevation | grass71 unsetmapset ` Looks much nicer - I agree. Which subcommands and how to name them is of