[issue35005] argparse should accept json and yaml argument types

[Raymond Hettinger]

Change by Raymond Hettinger :


--
pull_requests: +22713
pull_request: https://github.com/python/cpython/pull/23849

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I am going to decline this feature request on the principle of keeping our 
modules loosely coupled and orthogonal to one another.  

As paul.j3 pointed out, the "type" option seems to work best with simple 
conversions like "int".  As pointed-out by another respondent, FileType hasn't 
aged well and it may have been a mistake to include it all.  Likewise other 
converters like DateTime have been previously discussed and rejected.

There is a gray area between where argument parsing stops and application logic 
begins.   Since this is a standard library module, we should draw the line at 
simple converters/validators like "int", leaving anything more complex for 
downstream logic.  That will afford users greater flexibility and control than 
supported by the add_argument() API which intentionally only handles common 
cases.

[Braden Groom]
> I agree with Paul. This is probably simple enough for applications
> to implement. It also doesn't make sense to add either of these
> if the precedent was set by rejecting DateTime previously.

I concur as well.

[Mauricio Villegas]
> FYI there is a new python package that extends argparse 
> with the enhancements proposed here and more.

Thank you for the link. It will likely prove to be a valuable resource for 
people finding this issue in the future.  The referenced project shows the 
value of external projects being able to go in directions that are well beyond 
the scope of our standard library module: "Not exclusively intended for parsing 
command line arguments. The main focus is parsing yaml or jsonnet configuration 
files and not necessarily from a command line tool."

--
assignee: bob.ippolito -> rhettinger
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Mauricio Villegas]

Mauricio Villegas  added the comment:

FYI there is a new python package that extends argparse with the enhancements 
proposed here and more.

https://pypi.org/project/jsonargparse/

--
nosy: +mauvilsa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Braden Groom]

Braden Groom  added the comment:

I agree with Paul. This is probably simple enough for applications to 
implement. It also doesn't make sense to add either of these if the precedent 
was set by rejecting DateTime previously.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[paul j3]

paul j3  added the comment:

This kind of file read can be done just as easily after parsing.  For example 
using the function in my previous post:

In [127]: parser = argparse.ArgumentParser()
In [128]: parser.add_argument('-d','--data');
In [129]: args = parser.parse_args(['-d','@foo.json'])
In [130]: args
Out[130]: Namespace(data='@foo.json')
In [131]: args.data = readdata(args.data)
In [132]: args
Out[132]: Namespace(data={'foo': 12, 'bar': 'twelve'})

I've pointed out in various SO answers that using the 'type' parameter has just 
a few benefits.

- If you have many arguments that require this conversion, using type is a 
little more streamlined.  But it's not hard to iterate of a list of 'dest'.

- Using type routes the errors through the standard argparse mechanism, 
including the display of usage and system exit.  But the type function has to 
raise TypeError, ValueError, or ArgumentTypeError.  But you can also use 
'parser.error(...)' in the post parsing processing.

Error handling is the make-or-break-it issue.  What kinds of errors do we want 
to handle?  What if the file name is bad or not accessible?  What if the file 
is poorly formatted JSON?  How do other APIs handle these errors?

For example the function that I defined can raise a FileNotFoundError if the 
file isn't found, or a JSONDecodeError if the file is badly formed.  
JSONDecodeError is a subclass of ValueError, but IO errors are not.

If such a type function is added to argparse, the unittest file, 
test_argparse.py will have to have a number of test cases.  It will have to 
create a valid json test file, and possibly an invalid one.  It will have test 
working cases, and various error cases.  The amount of testing code is likely 
to be many times larger than the function code itself.  And we can't overlook 
the documentation additions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Bob Ippolito]

Bob Ippolito  added the comment:

I don't think that this has anything in particular to do with the json module, 
at least it certainly shouldn't need any additional functionality from there.

YAML parsing isn't available in the stdlib last I checked, so that is probably 
not really up for consideration for direct integration.

In any case, I think the best approach would be to first do some research 
(StackOverflow, GitHub, etc.) to see how other folks are doing this in the 
wild, to see if there's a common pattern that should be made available in the 
stdlib.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[paul j3]

paul j3  added the comment:

Adding a new 'type' function or factor is certainly possible, and probably will 
be free of backward compatibility issues.  But so far no other custom type has 
been added to argparse.

https://bugs.python.org/issue23884 - rejects adding a DateTime class

https://bugs.python.org/issue22884 - there are several outstanding issue 
dealing with the FileType class.  That hasn't aged well, since file IO 
standards have changed over the years.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Braden Groom]

Braden Groom  added the comment:

I could try adding JSON and YAML argument types if that's what we want to do.

--
nosy: +bradengroom

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Bob, do you prefer the current arrange where the user needs to write their own 
"type" function to handle JSON and YAML, or would you like to add this as a 
built-in option?

--
assignee:  -> bob.ippolito
nosy: +bob.ippolito, rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[paul j3]

paul j3  added the comment:

If I define a 'type' function like:

def readdata(astr):
if astr.startswith('@'):
with open(astr[1:], 'r') as f:
d = json.load(f)
return d
else:
return astr

I can use it to load a json file, or use the string directly:

In [59]: parser = argparse.ArgumentParser()
In [60]: parser.add_argument('-d','--data', type=readdata);
In [61]: parser.parse_args(['-d','@foo.json'])
Out[61]: Namespace(data={'foo': 12, 'bar': 'twelve'})
In [62]: parser.parse_args(['-d','xxx'])
Out[62]: Namespace(data='xxx')

This seems to behave as the 'curl' and 'ansible' examples you give, where the 
interpretation of the '@' is option specific.

A fully functional version of this type function needs to catch possible errors 
(not a file, not proper json, etc) and raise a ValueError or 
argparse.ArgumentTypeError.

The fact that 'curl -d' uses the '@', and 'fromfile_prefix_args' uses '@' in 
the documentation, should be seen as purely coincidental.  argparse would be 
just as happy using '#' or '%' as fromfile-prefix characters, just so long as 
the shell passes them unchanged to 'sys.argv'.  Conversely a type function can 
pay attention to special characters without needing to define them in the 
parser definition.

argparse doesn't define many custom 'type' functions.  Mostly it depends on 
people using stock Python functions like 'int' and 'float', or writing their 
own functions.  This keeps things simple for the common uses, while giving more 
advanced users a lot of flexibility.

This '@file' sensitivity could also be built into a custom Action subclass.  
There too, argparse has defined a set of common cases, but lets the users 
customize to their heart's content.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Robert Benson]

Robert Benson  added the comment:

What I'm talking about is reading a single arg (of a dictionary or collection 
type) that can be split across multiple lines, rather than a single line 
containing multiple args

My motivation was that reading args from a file should behave in a manner 
similar to other command-line utilities, such as the `-d` option for `curl` and 
the `-e` option for `ansible`. These take the entire file you give it and store 
it as one dictionary or object, not by merging it with the rest of the 
namespace but by taking the dictionary as the value of just that arg. So:

argument_parser.add_argument("-d", "--data", type=argparse.JsonType)  # just 
for example

if I call the program with `--data @foo.json`
I want argument_parser.parse_args().data to be the dict that is in foo.json, 
whether foo.json is pretty-printed or not.

I haven't done an exhaustive search of StackOverflow, but seeing a couple top 
answers indicated that this was not readily available without the user at least 
having to call `json.loads` on a string argument themselves, when it seems 
logical that it would be built into the library to parse the json into a 
dictionary

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[paul j3]

paul j3  added the comment:

The results of reading one of these @-prefix files are just spliced into the 
`argv` list before it is parsed.  This is done early in the parsing method, in 
the 

_read_args_from_files

method.  The documentation illustrates how this file reading can be modified to 
take several strings from each line:

https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.convert_arg_line_to_args

That shouldn't be taken as the only possible modification.

That said, I don't see how reading from a JSON or YAML file fits with this 
mechanism.  Presumably those would define some dictionary like key:value pairs. 
 I assume you'd want to enter those directly into the Namespace, not the argv 
list that will be parsed.  Or otherwise merged with a dictionary produced by 
parsing the other commandline strings.

So what you want to do with a JSON file, and how that relates to argparse is 
not clear.  You need to elaborate before we can discuss this issue further.

You might want to search on Stackoverflow with the tags '[argparse] [json]' to 
see how others have tried to use the two together.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Zachary Ware]

Change by Zachary Ware :


--
nosy: +bethard, paul.j3
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Robert Benson]

New submission from Robert Benson :

Using `argparse`, I wanted to create an argument that was a JSON dictionary. I 
found that using this in combination with the `fromfile_prefix_args` keyword 
argument, that the parser assumes that each argument provided in the file must 
be on a single line. I want the module to be able to support json files that 
may be pretty-printed. If it is to accept JSON in this manner, it would be not 
much more effort to implement YAML parsing as well

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Robert Benson]

Change by Robert Benson :


--
versions:  -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Robert Benson]

Change by Robert Benson :


--
components: Library (Lib)
nosy: derelbenkoenig
priority: normal
severity: normal
status: open
title: argparse should accept json and yaml argument types
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35005] argparse should accept json and yaml argument types

[Robert Benson]

Change by Robert Benson :


--
versions: +Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com