Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
Hi, I've had some success prototyping the below idea on a few utilities. See https://github.com/OSGeo/gdal/pull/9445 for details Even Le 08/03/2024 à 16:40, Even Rouault via gdal-dev a écrit : Hi, Our command line C++ utilities use ad-hoc manual parsing, which means that: - the usage message must be manually composed, - you must take care to check that there are enough remaining arguments for the ones that take value to avoid out-of-bounds accesses (tests like argc + 1 < argn) - detection for duplicated arguments when only a single occurrence is allowed must be manually done, nd thus is often not done, confusing users, cf https://github.com/OSGeo/gdal/issues/9415 - etc. I've come across https://github.com/p-ranav/argparse which fit all my requirements at first sight: compatible with our C++ requirements (C++17), MIT license, easily usable (single header), well documented, and enough feature-full. From a quick testing, it seems to work well. It looks also as it has taken some inspiration from the Python argparse module. I'd be tempted to give that a try to retrofit our existing utilities (probably starting with the ones with the less options :-)). Opinions? I guess there must be a plethora of similar projects, due to the absence of a std::argparse module... At least I see it is in the list of (9) alternatives mentioned at https://en.cppreference.com/w/cpp/links/libs?source=post_page---#Configuration:Command_Line CLI11 looked like a candidate too, but reading https://github.com/CLIUtils/CLI11?tab=readme-ov-file#features-not-supported-by-this-library "There are some other possible "features" that are intentionally not supported by this library:... Non-standard variations on syntax, like |-long| options. This is non-standard and should be avoided, so that is enforced by this library." . Fair enough, but we use that extensively in GDAL. Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
Le 08/03/2024 à 18:34, Javier Jimenez Shaw a écrit : I don't like that the behaviour of a command line depends on the configuration of the user (that is usually not aware of). So a command that works for me doesn't work for you. That is bad. I was perhaps not clear. Let me clarify, but at least for the current state of utilities, the current locale shouldn't matter because we use locale-independent methods. And that should be an aim for the new parser too. I don't know in GDAL, but in proj there is the option --bbox, that has comma separated coordinates. That can be -10.5,5.1,-9.5,6.3 in GDAL I can only think of cases where the separator is space. I believe that we could use CPLAtofM() (accepting dot or comma) in situations where there's no ambiguity, and CPLAtof() (accepting dot only) if there are situations where comma is used as a separator between numbers themselves (to be confirmed we have such occurences) -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
> I don't like that the behaviour of a command line depends on the > configuration of the user (that is usually not aware of). So a command that > works for me doesn't work for you. That is bad. > +1 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
I don't like that the behaviour of a command line depends on the configuration of the user (that is usually not aware of). So a command that works for me doesn't work for you. That is bad. I don't know in GDAL, but in proj there is the option --bbox, that has comma separated coordinates. That can be -10.5,5.1,-9.5,6.3 On Fri, 8 Mar 2024 at 18:27, Even Rouault wrote: > > > > > In principle the idea sounds good. > > > > How is it parsing the numbers? is it locale agnostic? I think it is > > not, because it is using "strtod". That means that if I have my locale > > in Spanish, French, German, ... it will expect "," as the decimal > > separator, right? > if running under a non-C locale, and without modification, yes > > ... well, how is GDAL expecting floating values? > > Only if the utilities enables setlocale(LC_ALL, ""), which they don't by > default. That said as the command line utility parsing is also used for > the utility-as-C-function, we may run under a non-C locale (that is a > Spanish/French/whatever locale) in some contexts. > > > Is GDAL locale agnostic? > Grep'ping I see that we use both CPLAtof() which is locale-agnostic and > assume dot as decimal separator or CPLAtofM() which accept dot or comma > as decimal separator. I don't think there is a particular reason in > using any of them. Inconsistency likely due to doing the work manually > and depending on the mood of the developer. Not totally sure of the > behavior we want: should we always require dot as the decimal separator, > or be lenient and accept both dot and comma (wondering if there wouldn't > be situations where we would take a "a,b,c" string, with a, b, c being > floating point numbers, in which case obviously we would have to require > dot. But I'm not sure if that can happen) > > > > Do you want to add it as a dependency, or just copy-paste the header > > file into gdal repo? > > yes, copy-paste and potentially with a few changes for our needs, like > using CPLAtof or CPLAtofM. > > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
In principle the idea sounds good. How is it parsing the numbers? is it locale agnostic? I think it is not, because it is using "strtod". That means that if I have my locale in Spanish, French, German, ... it will expect "," as the decimal separator, right? if running under a non-C locale, and without modification, yes ... well, how is GDAL expecting floating values? Only if the utilities enables setlocale(LC_ALL, ""), which they don't by default. That said as the command line utility parsing is also used for the utility-as-C-function, we may run under a non-C locale (that is a Spanish/French/whatever locale) in some contexts. Is GDAL locale agnostic? Grep'ping I see that we use both CPLAtof() which is locale-agnostic and assume dot as decimal separator or CPLAtofM() which accept dot or comma as decimal separator. I don't think there is a particular reason in using any of them. Inconsistency likely due to doing the work manually and depending on the mood of the developer. Not totally sure of the behavior we want: should we always require dot as the decimal separator, or be lenient and accept both dot and comma (wondering if there wouldn't be situations where we would take a "a,b,c" string, with a, b, c being floating point numbers, in which case obviously we would have to require dot. But I'm not sure if that can happen) Do you want to add it as a dependency, or just copy-paste the header file into gdal repo? yes, copy-paste and potentially with a few changes for our needs, like using CPLAtof or CPLAtofM. -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
Some options have related environment variables, eg --debug /CPL_DEBUG. Options are also passed through to several drivers. I assume that the implications for these are not significant ? This wouldn't change the general logic of GDAL and so the existing (possibly confusing) set of various type of options, configuration options/env variables, dataset-creation, layer-creation, etc. would remain :-) Hopefully the change would be mostly unnoticed by users. -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
Hi Even. In principle the idea sounds good. How is it parsing the numbers? is it locale agnostic? I think it is not, because it is using "strtod". That means that if I have my locale in Spanish, French, German, ... it will expect "," as the decimal separator, right? ... well, how is GDAL expecting floating values? Is GDAL locale agnostic? Do you want to add it as a dependency, or just copy-paste the header file into gdal repo? On Fri, 8 Mar 2024 at 16:40, Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > Our command line C++ utilities use ad-hoc manual parsing, which means that: > > - the usage message must be manually composed, > > - you must take care to check that there are enough remaining arguments > for the ones that take value to avoid out-of-bounds accesses (tests like > argc + 1 < argn) > > - detection for duplicated arguments when only a single occurrence is > allowed must be manually done, nd thus is often not done, confusing users, > cf https://github.com/OSGeo/gdal/issues/9415 > > - etc. > > I've come across https://github.com/p-ranav/argparse which fit all my > requirements at first sight: compatible with our C++ requirements (C++17), > MIT license, easily usable (single header), well documented, and enough > feature-full. From a quick testing, it seems to work well. It looks also as > it has taken some inspiration from the Python argparse module. > > I'd be tempted to give that a try to retrofit our existing utilities > (probably starting with the ones with the less options :-)). Opinions? I > guess there must be a plethora of similar projects, due to the absence of a > std::argparse module... At least I see it is in the list of (9) > alternatives mentioned at > https://en.cppreference.com/w/cpp/links/libs?source=post_page---#Configuration:Command_Line > > CLI11 looked like a candidate too, but reading > https://github.com/CLIUtils/CLI11?tab=readme-ov-file#features-not-supported-by-this-library > "There are some other possible "features" that are intentionally not > supported by this library:... Non-standard variations on syntax, like > -long options. This is non-standard and should be avoided, so that is > enforced by this library." . Fair enough, but we use that extensively in > GDAL. > > Even > > -- http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
Le ven. 8 mars 2024, 17:42, Even Rouault a écrit : > Thomas, > > > > my number 1 requirement would be that the rewrite not cause any > > backwards compatibility issues compared to today's argument handling. > > I suspect many users are calling the gdal utilities through scripts > > and it would be a pain to have to update those when upgrading gdal. > > Yes, backward compatibility is an obvious requirement. That said, it > would not be fully backwards compatible in the situations where users > currently misuse the official synopsis, and typically when they repeat > an argument that is supposed to not be repeatable. Currently depending > on the argument and utility, they might potentially get an error, or the > first occurrence will be used, or the last occurrence will be used (most > often this last case). With a standard argument parser, this will be > rejected with an explicit error message. This is actually what triggered > me to consider using a dedicated mechanism for argument parsing. > fully agree. my worry concerned e.g. the "-b " case where an arg parser handling the int conversion could have been useful, but would break our "-b mask" current usage. > Backward compatibility for nominal cases will be tricky in some > situations. For example I see that ogrinfo supports a weird syntax like > "-geom=NO" that p-ranav/argparse cannot support out of the box (it > supports ["-geom", "NO"], and if a dash dash alias is also declared, > ["--geom", "NO"] or ["--geom=NO"]). So for those hopefully rare cases, > one will have to pre-process argv[] before passing to p-ranav/argparse > so that is sees 2 separate argv[] values ["-geom", "NO"] instead to work > properly. I hope those are not to common, so the rewrite does not become more difficult to maintain than today! > > > > a nice to have enhancement that could be added during this upgrade > > (which actually could be incompatible with my previous point, although > > I don't see any on the top of my head) would be to support "--config > > key=value" alongside "--config key value" in order to align with e.g. > > creation options. > > That's kind of orthogonal. We can support both with the logic: if the > argv[i] value after "--config" has an equal sign in it, assume it is a > key=value pair (none of our valid config options should have an equal > sign in the key!). Otherwise assume argv[i] is the key and argv[i+1] is > the value. > fully agree it is orthogonal, I was just throwing in a pain point I have today. hopefully we don't have an occurence where the config key contains an equal sign 爛 regards, tb > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
On Fri, 8 Mar 2024, Even Rouault via gdal-dev wrote: Hi, Our command line C++ utilities use ad-hoc manual parsing, which means that: - the usage message must be manually composed, - you must take care to check that there are enough remaining arguments for the ones that take value to avoid out-of-bounds accesses (tests like argc + 1 < argn) - detection for duplicated arguments when only a single occurrence is allowed must be manually done, nd thus is often not done, confusing users, cf https://github.com/OSGeo/gdal/issues/9415 - etc. I've come across https://github.com/p-ranav/argparse which fit all my requirements at first sight: compatible with our C++ requirements (C++17), MIT license, easily usable (single header), well documented, and enough feature-full. From a quick testing, it seems to work well. It looks also as it has taken some inspiration from the Python argparse module. I'd be tempted to give that a try to retrofit our existing utilities (probably starting with the ones with the less options :-)). Opinions? I guess there must be a plethora of similar projects, due to the absence of a std::argparse module... At least I see it is in the list of (9) alternatives mentioned at https://en.cppreference.com/w/cpp/links/libs?source=post_page---#Configuration:Command_Line Some options have related environment variables, eg --debug /CPL_DEBUG. Options are also passed through to several drivers. I assume that the implications for these are not significant ? -- Andrew C. Aitchison Kendal, UK and...@aitchison.me.uk___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
Thomas, my number 1 requirement would be that the rewrite not cause any backwards compatibility issues compared to today's argument handling. I suspect many users are calling the gdal utilities through scripts and it would be a pain to have to update those when upgrading gdal. Yes, backward compatibility is an obvious requirement. That said, it would not be fully backwards compatible in the situations where users currently misuse the official synopsis, and typically when they repeat an argument that is supposed to not be repeatable. Currently depending on the argument and utility, they might potentially get an error, or the first occurrence will be used, or the last occurrence will be used (most often this last case). With a standard argument parser, this will be rejected with an explicit error message. This is actually what triggered me to consider using a dedicated mechanism for argument parsing. Backward compatibility for nominal cases will be tricky in some situations. For example I see that ogrinfo supports a weird syntax like "-geom=NO" that p-ranav/argparse cannot support out of the box (it supports ["-geom", "NO"], and if a dash dash alias is also declared, ["--geom", "NO"] or ["--geom=NO"]). So for those hopefully rare cases, one will have to pre-process argv[] before passing to p-ranav/argparse so that is sees 2 separate argv[] values ["-geom", "NO"] instead to work properly. a nice to have enhancement that could be added during this upgrade (which actually could be incompatible with my previous point, although I don't see any on the top of my head) would be to support "--config key=value" alongside "--config key value" in order to align with e.g. creation options. That's kind of orthogonal. We can support both with the logic: if the argv[i] value after "--config" has an equal sign in it, assume it is a key=value pair (none of our valid config options should have an equal sign in the key!). Otherwise assume argv[i] is the key and argv[i+1] is the value. -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a "standard" argument parser for command line utilities?
Hi Even, my number 1 requirement would be that the rewrite not cause any backwards compatibility issues compared to today's argument handling. I suspect many users are calling the gdal utilities through scripts and it would be a pain to have to update those when upgrading gdal. a nice to have enhancement that could be added during this upgrade (which actually could be incompatible with my previous point, although I don't see any on the top of my head) would be to support "--config key=value" alongside "--config key value" in order to align with e.g. creation options. best regards, TB Le ven. 8 mars 2024, 16:41, Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> a écrit : > Hi, > > Our command line C++ utilities use ad-hoc manual parsing, which means that: > > - the usage message must be manually composed, > > - you must take care to check that there are enough remaining arguments > for the ones that take value to avoid out-of-bounds accesses (tests like > argc + 1 < argn) > > - detection for duplicated arguments when only a single occurrence is > allowed must be manually done, nd thus is often not done, confusing users, > cf https://github.com/OSGeo/gdal/issues/9415 > > - etc. > > I've come across https://github.com/p-ranav/argparse which fit all my > requirements at first sight: compatible with our C++ requirements (C++17), > MIT license, easily usable (single header), well documented, and enough > feature-full. From a quick testing, it seems to work well. It looks also as > it has taken some inspiration from the Python argparse module. > > I'd be tempted to give that a try to retrofit our existing utilities > (probably starting with the ones with the less options :-)). Opinions? I > guess there must be a plethora of similar projects, due to the absence of a > std::argparse module... At least I see it is in the list of (9) > alternatives mentioned at > https://en.cppreference.com/w/cpp/links/libs?source=post_page---#Configuration:Command_Line > > CLI11 looked like a candidate too, but reading > https://github.com/CLIUtils/CLI11?tab=readme-ov-file#features-not-supported-by-this-library > "There are some other possible "features" that are intentionally not > supported by this library:... Non-standard variations on syntax, like > -long options. This is non-standard and should be avoided, so that is > enforced by this library." . Fair enough, but we use that extensively in > GDAL. > > Even > > -- http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Using a "standard" argument parser for command line utilities?
Hi, Our command line C++ utilities use ad-hoc manual parsing, which means that: - the usage message must be manually composed, - you must take care to check that there are enough remaining arguments for the ones that take value to avoid out-of-bounds accesses (tests like argc + 1 < argn) - detection for duplicated arguments when only a single occurrence is allowed must be manually done, nd thus is often not done, confusing users, cf https://github.com/OSGeo/gdal/issues/9415 - etc. I've come across https://github.com/p-ranav/argparse which fit all my requirements at first sight: compatible with our C++ requirements (C++17), MIT license, easily usable (single header), well documented, and enough feature-full. From a quick testing, it seems to work well. It looks also as it has taken some inspiration from the Python argparse module. I'd be tempted to give that a try to retrofit our existing utilities (probably starting with the ones with the less options :-)). Opinions? I guess there must be a plethora of similar projects, due to the absence of a std::argparse module... At least I see it is in the list of (9) alternatives mentioned at https://en.cppreference.com/w/cpp/links/libs?source=post_page---#Configuration:Command_Line CLI11 looked like a candidate too, but reading https://github.com/CLIUtils/CLI11?tab=readme-ov-file#features-not-supported-by-this-library "There are some other possible "features" that are intentionally not supported by this library:... Non-standard variations on syntax, like |-long| options. This is non-standard and should be avoided, so that is enforced by this library." . Fair enough, but we use that extensively in GDAL. Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
