Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Serhiy Storchaka]

On 28.03.17 02:35, Greg Ewing wrote:

Paul Moore wrote:

Is this a well-defined idea? ... There's nothing describing how
multiple values would be stored in the same file/transmitted in the
same stream.


I think this is something that's outside the scope of the spec.

But since the grammar makes it clear when you've reached the end
of a value, it seems entirely reasonable for a parser to just
stop reading from the stream at that point, and leave whatever
remains for the application to deal with as it sees fit. The
application can then choose to immediately read another value
from the same stream if it wants.


You can determine the end of integer literal only after reading a 
character past the end of the integer literal. This there is not a way 
to put back a character, it will be lost for following readers.


And currently json.load() is implemented by reading all file content at 
once and passing it to json.loads(). Different implementation would be 
much more complex (if we don't want to loss the performance).



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] What about regexp string litterals : re".*" ?

[Markus Meskanen]

On Tue, Mar 28, 2017 at 8:37 AM, Chris Angelico  wrote:
>
> Yes, but if the "in" operator is used, it would still work, because
> r"..." is a str, and "str" in "string" is meaningful.
>
> But I think a better solution will be for regex literals to be
> syntax-highlighted differently. If they're a truly-supported syntactic
> feature, they can be made visually different in your editor, making
> the distinction blatantly obvious.
>
> That said, though, I'm -1 on this. Currently, every prefix letter has
> its own meaning, and broadly speaking, combining them combines their
> meanings. An re"..." literal should be a raw "e-string", whatever that
> is, so I would expect that e"..." is the same kind of thing but with
> different backslash handling.
> 
>

Fair enough, I haven't followed this thread too closely and didn't consider
the "in" operator being used. Even then I find it unlikely that confusing
re'...' with r'...' and not noticing would turn out to be an issue.

That being said, I'm also -1 on this, especially now after your point on
"e-string". Adding these re-strings would straight out prevent e-string
from ever being implemented.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] What about regexp string litterals : re".*" ?

[Chris Angelico]

On Tue, Mar 28, 2017 at 2:24 PM, Markus Meskanen
 wrote:
> While I agree with most of your arguments, surely you must be the one joking
> here? "Ugly" is obviously a matter of opinion, I personally find the
> proposed syntax more beautiful than the // used in many other languages. But
> claiming it's bad because people would mix it up with raw strings and people
> not realizing is nonsense. Not only does it look very different, but
> attempting to call match() or any other regex method on it would surely give
> out a reasonable error:
>
>   AttributeError: 'str' object has no attribute 'match'
>
> Which _in the worst case scenario_ results into googling where the top rated
> StackOverflow question clearly explains the difference between r'' and re''

Yes, but if the "in" operator is used, it would still work, because
r"..." is a str, and "str" in "string" is meaningful.

But I think a better solution will be for regex literals to be
syntax-highlighted differently. If they're a truly-supported syntactic
feature, they can be made visually different in your editor, making
the distinction blatantly obvious.

That said, though, I'm -1 on this. Currently, every prefix letter has
its own meaning, and broadly speaking, combining them combines their
meanings. An re"..." literal should be a raw "e-string", whatever that
is, so I would expect that e"..." is the same kind of thing but with
different backslash handling.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Proposal: Query language extension to Python (PythonQL)

[Pavol Lisy]

On 3/27/17, Chris Angelico  wrote:
> On Mon, Mar 27, 2017 at 12:26 PM, Terry Reedy  wrote:
>> It might be possible (or not!) to make the clause-heading words like
>> 'where'
>> or 'groupby' (this would have to be one word) recognized as special only
>> in
>> the context of starting a new comprehension clause. The precedents for
>> 'keyword in context' are 'as', 'async', and 'await'.  But these were
>> temporary and a nuisance (both to code and for syntax highlighting) and I
>> would not be in favor of repeating for this case.
>
> Apologies if it's already been mentioned, but is MacroPy able to do
> this without introducing actual language keywords?
>
> ChrisA

In very first mail (section "Current Status") in this thread was
mentioned that they use "encoding hack" now.

PL
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] What about regexp string litterals : re".*" ?

[Markus Meskanen]

On Mar 28, 2017 06:08, "Steven D'Aprano"  wrote:

On Mon, Mar 27, 2017 at 05:17:40PM +0200, Simon D. wrote:

> The regexp string litteral could be represented by : re""
>
> It would ease the use of regexps in Python, allowing to have some regexp
> litterals, like in Perl or JavaScript.
>
> We may end up with an integration like :
>
> >>> import re
> >>> if re".k" in 'ok':
> ... print "ok"
> ok

I dislike the suggested syntax re".k". It looks ugly and not different
enough from a raw string. I can easily see people accidentally writing:

if r".k" in 'ok':
...

and wondering why their regex isn't working.


While I agree with most of your arguments, surely you must be the one
joking here? "Ugly" is obviously a matter of opinion, I personally find the
proposed syntax more beautiful than the // used in many other languages.
But claiming it's bad because people would mix it up with raw strings and
people not realizing is nonsense. Not only does it look very different, but
attempting to call match() or any other regex method on it would surely
give out a reasonable error:

  AttributeError: 'str' object has no attribute 'match'

Which _in the worst case scenario_ results into googling where the top
rated StackOverflow question clearly explains the difference between r''
and re''
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] What about regexp string litterals : re".*" ?

[Steven D'Aprano]

On Mon, Mar 27, 2017 at 05:17:40PM +0200, Simon D. wrote:

> The regexp string litteral could be represented by : re""
> 
> It would ease the use of regexps in Python, allowing to have some regexp
> litterals, like in Perl or JavaScript.
> 
> We may end up with an integration like :
> 
> >>> import re
> >>> if re".k" in 'ok':
> ... print "ok"
> ok

I dislike the suggested syntax re".k". It looks ugly and not different 
enough from a raw string. I can easily see people accidentally writing:

if r".k" in 'ok':
...

and wondering why their regex isn't working.


Javascript uses /regex/ as a literal syntax for creating RegExp objects. 
That's the closest equivalent to the way Python would have to operate, 
although I don't think we can use the /.../ syntax without breaking the 
rule that Python's parser will not be more complex than LL(1). So I 
think /.../ is definitely out.

Perl 6 uses m/regex/ and a number of other variations:

https://docs.perl6.org/language/regexes


I doubt that this will actually be useful. It *seems* useful if you just 
write trivial regexes like your example, but without Perl's rich set of 
terse (cryptic?) operators, I don't know that literal regexes 
makes enough difference to be worth the trouble. There's not very 
much difference between (say) these:

mo = re.search(r'.k', mystring)
if mo:
print(mo.group())

mo = re.'.k'.search(mystring)
if mo:
print(mo.group())


You effectively save two parentheses, that's all. That doesn't seem like 
much of a win for introducing new syntax. Can you show some example code 
where a regex literal will have a worthwhile advantage?


> Regexps are part of the language in Perl, and the rather complicated
> integration of regexp in other languages, especially in Python, is
> something that comes up easily in language comparing discussion.

Surely you are joking?

Regex integration in Python is simple. Regular expression objects are 
ordinary objects, like lists and dicts and floats. The only difference 
is that you don't call the Regex object constructor directly, you either 
pass a string to a module level function

re.match(r'my regex', mystring)

or you create a regex object:

regex = re.compile(r'my regex')
regex.match(mystring)


That's very neat, Pythonic and simple. The regex itself is very close to 
the same syntax uses by Perl, Javascript or other variations, the only 
complication is that due to Python's escaping rules you should use a raw 
string r'' instead of doubling up all backslashes. I wouldn't call that 
"rather complicated" -- it is a lot less complicated than Perl:

- m// can be abbreviated //
- when do you use // directly and when do you use qr// ?
- s/// operator implicitly defines a regex

In Perl 6, I *think* they use rx// instead of qr//, or are they 
different things? Both m// and the s/// operator can use arbitrary 
delimiters, e.g. ! or , (but not : or parentheses) instead of the 
slashes, and m// regexes will implicitly match against $_ if you don't 
explicitly match against something else.

Compared to Perl, I don't think Python's regexes are complicated.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding an 'errors' argument to print

[eryk sun]

On Mon, Mar 27, 2017 at 8:52 PM, Barry  wrote:
> I took to using
>
>  chcp 65001
>
> This puts cmd.exe into unicode mode.

conhost.exe hosts the console, and chcp.com is a console app that
calls GetConsoleCP, SetConsoleCP and SetConsoleOutputCP to show or
modify the console's input and output codepages. It doesn't support
changing them separately.

cmd.exe is just another console client, no different from python.exe
or powershell.exe in this regard. Also, it's unrelated to how Python
uses the console, but for the record, cmd has used the console's
wide-character API since it was ported from OS/2 in the early 90s.

Back then the console was hosted using threads in the csrss.exe system
process, which made sense because the windowing system was hosted
there. When they moved most of the window manager to kernel mode in NT
4 (1996), the console was mostly left behind in csrss.exe. It wasn't
until Windows 7 that it found a new home in conhost.exe. In Windows 8
it got a real device driver instead of using fake file handles. In
Windows 10 it was updated to be less of a franken-window -- e.g. now
it has line-wrapped selection and text reflowing.

Using codepage 65001 (UTF-8) in a console app has a couple of annoying
bugs in the console itself, and another due to flushing of C FILE
streams. For example, reading text that has even a single non-ASCII
character will fail because conhost's encoding buffer is too small. It
handles the error by returning a read of 0 bytes. That's EOF, so
Python's REPL quits; input() raises EOFError; and stdin.read() returns
an empty string. Microsoft should fix this in Windows 10, and probably
will eventually. The Linux subsystem needs UTF-8, and it's silly that
the console doesn't allow entering non-ASCII text in Linux programs.

As was already recommended, I suggest using the wide-character API via
win_unicode_console in 2.7 and 3.5. In 3.6 we get the wide-character
API automatically thanks to Steve Dower's io._WindowsConsoleIO class.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Victor Stinner]

2017-03-27 17:04 GMT+02:00 Steven D'Aprano :
> Of course pathlib can already read JSON, or for that matter ReST text
> or JPG binary files. It can read anything as text or bytes, including
> JSON:
>
> some_path.write_text(json.dumps(obj))
> json.loads(some_path.read_text())

Note: You should specify the encoding:

some_path.write_text(json.dumps(obj), encoding='utf8')
json.loads(some_path.read_text(encoding='utf8'))


> I don't think it should be pathlib's responsibility to deal with the
> file format (besides text).

Right.

Victor
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Greg Ewing]

Paul Moore wrote:

Is this a well-defined idea? ... There's nothing describing how
multiple values would be stored in the same file/transmitted in the
same stream.


I think this is something that's outside the scope of the spec.

But since the grammar makes it clear when you've reached the end
of a value, it seems entirely reasonable for a parser to just
stop reading from the stream at that point, and leave whatever
remains for the application to deal with as it sees fit. The
application can then choose to immediately read another value
from the same stream if it wants.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Philipp A.]

Ram Rachum  schrieb am Mo., 27. März 2017 um 16:42 Uhr:

> Another idea: Maybe make json.load and json.dump support Path objects?
>

yes, all string-path expecting stdlib APIs should support PEP 519

https://www.python.org/dev/peps/pep-0519/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Wes Turner]

FWIW, pyline could produce streaming JSON w/ json.dumps(indent=0),
but because indent>0, there are newlines.

pydoc json | pyline '{"a":l} if "json" in l.lower() else None' -O json
pydoc json | pyline -r '.*JSON.*' 'rgx and line' -O json

It's a similar issue:
what are good default JSON encoding/decoding settings?

 # loads/JSONDecoder
 file.encoding # UTF-8
 object_pairs_hook
 object_hook

 # dumps/JSONEncoder
 file.encoding # UTF-8
 cls
 separators
 indent

- [ ] ENH: pyline: add 'jsonlines' as an {output,} format


>From https://twitter.com/raymondh/status/842777864193769472 :

#python tip:  Set separators=(',', ':') to dump JSON more compactly.
> >>> json.dumps({'a':1, 'b':2}, separators=(',',':'))
> '{"a":1,"b":2}'


On Mon, Mar 27, 2017 at 3:46 PM, David Mertz  wrote:

> This is a better link: https://en.m.wikipedia.org/wiki/JSON_Streaming
>
> On Mar 27, 2017 3:45 PM, "David Mertz"  wrote:
>
>> The format JSON lines (http://jsonlines.org/) is pretty widely used, but
>> is an extension of JSON itself. Basically, it's the idea that you can put
>> one object per physical line to allow incremental reading or spending of
>> objects.
>>
>> It's a good idea, and I think the `json` module should support it. But it
>> definitely doesn't belong in `pathlib`.
>>
>> On Mar 27, 2017 3:36 PM, "Paul Moore"  wrote:
>>
>>> On 27 March 2017 at 17:43, Bruce Leban  wrote:
>>> > the ability to read one json object from the input rather than reading
>>> the
>>> > entire input
>>>
>>> Is this a well-defined idea? From a quick read of the JSON spec (which
>>> is remarkably short on details of how JSON is stored in files, etc)
>>> the only reference I can see is to a "JSON text" which is a JSON
>>> representation of a single value. There's nothing describing how
>>> multiple values would be stored in the same file/transmitted in the
>>> same stream. It's not unreasonable to assume "read one object, then
>>> read another" but without an analysis of the grammar, it's not 100%
>>> clear if the grammar supports that (you sort of have to assume that
>>> when you hit "the end of the object" you skip some whitespace then
>>> start on the next - but the spec doesn't say anything like that.
>>> Alternatively, it's just as reasonable to assume that
>>> json.load/json.loads expect to be passed a single "JSON text" as
>>> defined by the spec.
>>>
>>> If the spec was clear on how multiple objects in a single stream
>>> should be handled, then yes the json module should support that. But
>>> without anything explicit in the spec, it's not as obvious. What do
>>> other languages do?
>>>
>>> Paul
>>> ___
>>> Python-ideas mailing list
>>> Python-ideas@python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Wes Turner]

On Mon, Mar 27, 2017 at 10:34 AM, Chris Barker 
wrote:

> On Mon, Mar 27, 2017 at 7:59 AM, Paul Moore  wrote:
>
>> On 27 March 2017 at 15:40, Ram Rachum  wrote:
>> > Another idea: Maybe make json.load and json.dump support Path objects?
>>
>> If they currently supported filenames, I'd say that's a reasonable
>> extension. Given that they don't, it still seems like more effort than
>> it's worth to save a few characters
>>
>
> Sure, but they probably should -- it's a REALLY common (most common)
> use-case to read and write JSON from a file. And many APIs support
> "filename or open file-like object".
>
> I'd love to see that added, and, or course, support for Path objects as
> well.
>




https://docs.python.org/2/library/json.html#encoders-and-decoders

# https://docs.python.org/2/library/json.html#json.JSONEncoder

class PathJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, pathlib.Path):
return unicode(obj)  # ? (what about bytes)
return OrderedDict((
('@type', 'pydatatypes:pathlib.Path'),  # JSON-LD
('path', unicode(obj)), )
 return json.JSONEncoder.default(self, obj)


# https://docs.python.org/2/library/json.html#json.JSONDecoder
def as_pathlib_Path(obj):
if obj.get('@type') == 'pydatatypes:pathlib.Path':
return pathlib.Path(obj.get('path'))
return obj


def read_json(self, **kwargs):
object_pairs_hook = kwargs.pop('object_pairs_hook',
collections.OrderedDict) # OrderedDefaultDict
object_hook = kwargs.pop('object_hook', as_pathlib_Path)
encoding = kwargs.pop('encoding', 'utf8')
with codecs.open(self, 'r ', encoding=encoding) as _file:
return json.load(_file,
object_pairs_hook=object_pairs_hook,
object_hook=object_hook,
**kwargs)

def write_json(self, obj, **kwargs):
kwargs['cls'] = kwargs.pop('cls', PathJSONEncoder)
encoding = kwargs.pop('encoding', 'utf8')
with codecs.open(self, 'w', encoding=encoding) as _file:
return json.dump(obj, _file, **kwargs)


def test_pathlib_json_encoder_decoder():
p = pathlib.Path('./test.json')
obj = dict(path=p, _path=str(unicode(p)))
p.write_json(obj)
obj2 = p.read_json()
assert obj['path'] == obj2['path']
assert isinstance(obj['path'], pathlib.Path)



https://github.com/jaraco/path.py/blob/master/path.py#L735
open()
bytes()
chunks()
write_bytes()
text()
def write_text(self, text, encoding=None, errors='strict',
   linesep=os.linesep, append=False):
lines()
write_lines()

read_hash()
read_md5()
read_hexhash()




>
> -CHB
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding an 'errors' argument to print

[Barry]

I took to using

 chcp 65001

This puts cmd.exe into unicode mode.

Of course the python 3.6 make this uneccesary i understand.

Barry


> On 24 Mar 2017, at 15:41, Ryan Gonzalez  wrote:
> 
> Recently, I was working on a Windows GUI application that ends up running 
> ffmpeg, and I wanted to see the command that was being run. However, the file 
> name had a Unicode character in it (it's a Sawano song), and when I tried to 
> print it to the console, it crashed during the encode/decode. (The encoding 
> used in cmd doesn't support Unicode characters.)
> 
> The workaround was to do:
> 
> 
> print(mystring.encode(sys.stdout.encoding, 
> errors='replace).decode(sys.stdout.encoding))
> 
> 
> Not fun, especially since this was *just* a debug print.
> 
> The proposal: why not add an 'errors' argument to print? That way, I could've 
> just done:
> 
> 
> print(mystring, errors='replace')
> 
> 
> without having to worry about it crashing.
> 
> --
> Ryan (ライアン)
> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
> http://refi64.com
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[David Mertz]

This is a better link: https://en.m.wikipedia.org/wiki/JSON_Streaming

On Mar 27, 2017 3:45 PM, "David Mertz"  wrote:

> The format JSON lines (http://jsonlines.org/) is pretty widely used, but
> is an extension of JSON itself. Basically, it's the idea that you can put
> one object per physical line to allow incremental reading or spending of
> objects.
>
> It's a good idea, and I think the `json` module should support it. But it
> definitely doesn't belong in `pathlib`.
>
> On Mar 27, 2017 3:36 PM, "Paul Moore"  wrote:
>
>> On 27 March 2017 at 17:43, Bruce Leban  wrote:
>> > the ability to read one json object from the input rather than reading
>> the
>> > entire input
>>
>> Is this a well-defined idea? From a quick read of the JSON spec (which
>> is remarkably short on details of how JSON is stored in files, etc)
>> the only reference I can see is to a "JSON text" which is a JSON
>> representation of a single value. There's nothing describing how
>> multiple values would be stored in the same file/transmitted in the
>> same stream. It's not unreasonable to assume "read one object, then
>> read another" but without an analysis of the grammar, it's not 100%
>> clear if the grammar supports that (you sort of have to assume that
>> when you hit "the end of the object" you skip some whitespace then
>> start on the next - but the spec doesn't say anything like that.
>> Alternatively, it's just as reasonable to assume that
>> json.load/json.loads expect to be passed a single "JSON text" as
>> defined by the spec.
>>
>> If the spec was clear on how multiple objects in a single stream
>> should be handled, then yes the json module should support that. But
>> without anything explicit in the spec, it's not as obvious. What do
>> other languages do?
>>
>> Paul
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[David Mertz]

The format JSON lines (http://jsonlines.org/) is pretty widely used, but is
an extension of JSON itself. Basically, it's the idea that you can put one
object per physical line to allow incremental reading or spending of
objects.

It's a good idea, and I think the `json` module should support it. But it
definitely doesn't belong in `pathlib`.

On Mar 27, 2017 3:36 PM, "Paul Moore"  wrote:

> On 27 March 2017 at 17:43, Bruce Leban  wrote:
> > the ability to read one json object from the input rather than reading
> the
> > entire input
>
> Is this a well-defined idea? From a quick read of the JSON spec (which
> is remarkably short on details of how JSON is stored in files, etc)
> the only reference I can see is to a "JSON text" which is a JSON
> representation of a single value. There's nothing describing how
> multiple values would be stored in the same file/transmitted in the
> same stream. It's not unreasonable to assume "read one object, then
> read another" but without an analysis of the grammar, it's not 100%
> clear if the grammar supports that (you sort of have to assume that
> when you hit "the end of the object" you skip some whitespace then
> start on the next - but the spec doesn't say anything like that.
> Alternatively, it's just as reasonable to assume that
> json.load/json.loads expect to be passed a single "JSON text" as
> defined by the spec.
>
> If the spec was clear on how multiple objects in a single stream
> should be handled, then yes the json module should support that. But
> without anything explicit in the spec, it's not as obvious. What do
> other languages do?
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Paul Moore]

On 27 March 2017 at 17:43, Bruce Leban  wrote:
> the ability to read one json object from the input rather than reading the
> entire input

Is this a well-defined idea? From a quick read of the JSON spec (which
is remarkably short on details of how JSON is stored in files, etc)
the only reference I can see is to a "JSON text" which is a JSON
representation of a single value. There's nothing describing how
multiple values would be stored in the same file/transmitted in the
same stream. It's not unreasonable to assume "read one object, then
read another" but without an analysis of the grammar, it's not 100%
clear if the grammar supports that (you sort of have to assume that
when you hit "the end of the object" you skip some whitespace then
start on the next - but the spec doesn't say anything like that.
Alternatively, it's just as reasonable to assume that
json.load/json.loads expect to be passed a single "JSON text" as
defined by the spec.

If the spec was clear on how multiple objects in a single stream
should be handled, then yes the json module should support that. But
without anything explicit in the spec, it's not as obvious. What do
other languages do?

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Chris Barker]

On Mon, Mar 27, 2017 at 9:43 AM, Bruce Leban  wrote:

> I'm not in favor of this idea for the reason mentioned by many of the
> other posters. BUT ... this does bring up something missing from json
> readers: *the ability to read one json object from the input rather than
> reading the entire input* and attempting to interpret it as one object.
>

I can't tell from the JSON spec (at least not quickly), but it is possible
to have more than one object at the top level?

Experimenting with the python json module seems to indicate that it is not
-- you can only have one "thing" in a JSON file -- either an "object" or an
array.

then, of course you can arbitrarily nest stuff inside that top-level
container.

Since the nesting is arbitrary, I'm not sure it's clear how a
one-object-at-a-time reader would work in the general case?

-CHB



> For my use case, it would be sufficient to read whole lines only but I can
> imagine other use cases.
>
> The basic rule would be to read as much of the input as necessary (and no
> more) to read a single json object, ignoring leading white space.
>
> In practical terms:
>
>- if the first character is [ or { or " read to the matching ] or } or
>"
>- otherwise if the first character is a digit or '-' read as many
>characters as possible to parse a number
>- otherwise attempt to match 'true', 'false' or 'null'
>- otherwise fail
>
>
> --- Bruce
> Check out my puzzle book and get it free here:
> http://J.mp/ingToConclusionsFree (available on iOS)
>
>
>
> On Mon, Mar 27, 2017 at 5:50 AM, Ram Rachum  wrote:
>
>> Hi guys,
>>
>> What do you think about adding methods pathlib.Path.write_json and
>> pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
>> read_bytes?
>>
>> This would make writing / reading JSON to a file a one liner instead of a
>> two-line with clause.
>>
>>
>> Thanks,
>> Ram.
>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Bruce Leban]

I'm not in favor of this idea for the reason mentioned by many of the other
posters. BUT ... this does bring up something missing from json readers: *the
ability to read one json object from the input rather than reading the
entire input* and attempting to interpret it as one object. For my use
case, it would be sufficient to read whole lines only but I can imagine
other use cases.

The basic rule would be to read as much of the input as necessary (and no
more) to read a single json object, ignoring leading white space.

In practical terms:

   - if the first character is [ or { or " read to the matching ] or } or "
   - otherwise if the first character is a digit or '-' read as many
   characters as possible to parse a number
   - otherwise attempt to match 'true', 'false' or 'null'
   - otherwise fail


--- Bruce
Check out my puzzle book and get it free here:
http://J.mp/ingToConclusionsFree (available on iOS)



On Mon, Mar 27, 2017 at 5:50 AM, Ram Rachum  wrote:

> Hi guys,
>
> What do you think about adding methods pathlib.Path.write_json and
> pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
> read_bytes?
>
> This would make writing / reading JSON to a file a one liner instead of a
> two-line with clause.
>
>
> Thanks,
> Ram.
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Ethan Furman]

On 03/27/2017 08:04 AM, Steven D'Aprano wrote:

On Mon, Mar 27, 2017 at 02:50:38PM +0200, Ram Rachum wrote:



What do you think about adding methods pathlib.Path.write_json and
pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
read_bytes?


That's not pathlib's responsibility, and there is nothing wrong with
writing two lines of code.


+1

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Paul Moore]

On 27 March 2017 at 15:48, Eric V. Smith  wrote:
> On 3/27/17 10:40 AM, Ram Rachum wrote:
>>
>> Another idea: Maybe make json.load and json.dump support Path objects?
>
>
> json.dump requires open file objects, not strings or Paths representing
> filenames.
>
> But does this not already do what you want:
>
> Path('foo.json').write_text(json.dumps(obj))
> ?

Indeed. There have now been a few posts quoting ways of reading and
writing JSON, all of which are pretty short (if that matters). Do we
*really* need another way?

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] What about regexp string litterals : re".*" ?

[Serhiy Storchaka]

On 27.03.17 18:17, Simon D. wrote:

After some french discussions about this idea, I subscribed here to
suggest adding a new string litteral, for regexp, inspired by other
types like : u"", r"", b"", br"", f""…

The regexp string litteral could be represented by : re""

It would ease the use of regexps in Python, allowing to have some regexp
litterals, like in Perl or JavaScript.


There are several regular expression libraries for Python. One of them 
is included in the stdlib, but this is not the first regular expression 
library in the stdlib and may be not the last. Particular project can 
choose using an alternative regular expression library (because it has 
additional features or is faster for particular cases).



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Eric V. Smith]

On 3/27/17 10:40 AM, Ram Rachum wrote:

Another idea: Maybe make json.load and json.dump support Path objects?


json.dump requires open file objects, not strings or Paths representing 
filenames.


But does this not already do what you want:

Path('foo.json').write_text(json.dumps(obj))
?

Eric.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Chris Barker]

On Mon, Mar 27, 2017 at 7:59 AM, Paul Moore  wrote:

> On 27 March 2017 at 15:40, Ram Rachum  wrote:
> > Another idea: Maybe make json.load and json.dump support Path objects?
>
> If they currently supported filenames, I'd say that's a reasonable
> extension. Given that they don't, it still seems like more effort than
> it's worth to save a few characters
>

Sure, but they probably should -- it's a REALLY common (most common)
use-case to read and write JSON from a file. And many APIs support
"filename or open file-like object".

I'd love to see that added, and, or course, support for Path objects as
well.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] What about regexp string litterals : re".*" ?

[Simon D.]

Hello,

After some french discussions about this idea, I subscribed here to
suggest adding a new string litteral, for regexp, inspired by other
types like : u"", r"", b"", br"", f""…

The regexp string litteral could be represented by : re""

It would ease the use of regexps in Python, allowing to have some regexp
litterals, like in Perl or JavaScript.

We may end up with an integration like :

>>> import re
>>> if re".k" in 'ok':
... print "ok"
ok
>>>

Regexps are part of the language in Perl, and the rather complicated
integration of regexp in other languages, especially in Python, is
something that comes up easily in language comparing discussion.

I've always felt JavaScript integration being half the way it should,
and new string litterals types in Python (like f"") looked like a good
compromise to have a tight integration of regexps without asking to make
them part of the language (as I imagine it has already been discussed
years ago, and obviously denied…).

As per XKCD illustration, using a regexp may be a problem on its own,
but really, the "each-language a new and complicated approach" is
another difficulty, of the level of writing regexps I think. And then,
when you get the trick for Python, it feels to me still to much letters
to type regarding the numerous problems one can solve using regexps.

I know regexps are slower than string-based workflow (like .startswith)
but regexps can do the most and the least, so they are rapide to come up
with, once you started to think with them. As Python philosophy is to
spare brain-cycles, sacrificing CPU-cycles, allowing to easily use
regexps is a brain-cycle savior trick.

What do you think ?

--
Simon Descarpentries
 +336 769 702 53
http://acoeuro.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Steven D'Aprano]

On Mon, Mar 27, 2017 at 02:50:38PM +0200, Ram Rachum wrote:
> Hi guys,
> 
> What do you think about adding methods pathlib.Path.write_json and
> pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
> read_bytes?
> 
> This would make writing / reading JSON to a file a one liner instead of a
> two-line with clause.

Reading/writing JSON is already a one liner, for people who care about 
writing one liners:

obj = json.load(open("foo.json"))
json.dump(obj, open("foo.json"))

Pathlib exists as an OO interface to low-level path and file operations. 
It understands how to read and write to files, but it doesn't understand 
the content of those files. I don't think it should.

Of course pathlib can already read JSON, or for that matter ReST text 
or JPG binary files. It can read anything as text or bytes, including 
JSON:

some_path.write_text(json.dumps(obj))
json.loads(some_path.read_text())


I don't think it should be pathlib's responsibility to deal with the 
file format (besides text). Today you want to add JSON support. What 
about XML and plists and ini files? Tomorrow you'll ask for HTML 
support, next week someone will want pathlib to support .wav files as a 
one liner, and before you know it pathlib is responsible for a hundred 
different file formats with separate read_* and write_* methods.

That's not pathlib's responsibility, and there is nothing wrong with 
writing two lines of code.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Paul Moore]

On 27 March 2017 at 15:40, Ram Rachum  wrote:
> Another idea: Maybe make json.load and json.dump support Path objects?

If they currently supported filenames, I'd say that's a reasonable
extension. Given that they don't, it still seems like more effort than
it's worth to save a few characters

with path.open('w'): json.dump(obj, f)
with path.open() as f: obj = json.load(f)

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Markus Meskanen]

Another idea: Maybe make json.load and json.dump support Path objects?


Much better. Or maybe add json.load_path and dump_path
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Serhiy Storchaka]

On 27.03.17 15:50, Ram Rachum wrote:

Hi guys,

What do you think about adding methods pathlib.Path.write_json and
pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
read_bytes?

This would make writing / reading JSON to a file a one liner instead of
a two-line with clause.


Good try, but you have published this idea 5 days ahead of schedule.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Donald Stufft]

> On Mar 27, 2017, at 10:36 AM, Paul Moore  wrote:
> 
> On 27 March 2017 at 15:33, Donald Stufft  > wrote:
>> What do you think about adding methods pathlib.Path.write_json and
>> pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
>> read_bytes?
>> 
>> 
>> 
>> -1, I also think that write_* and read_* were mistakes to begin with.
> 
> Text is (much) more general-use than JSON.


Sure. I also think touch() and all the others are the same :) I think they’re 
just an unfortunate detritus of a time before PathLike and that it’s super 
weird to have some operations you do to a file path (compared to things you do 
to generate, modify, or resolve a path) be hung off of the Path object and 
every other be an independent thing that takes it as an input. I’d find it 
equally weird if dictionary objects supported a print() or a .json() method.

—
Donald Stufft



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Ram Rachum]

Another idea: Maybe make json.load and json.dump support Path objects?

On Mon, Mar 27, 2017 at 4:36 PM, Paul Moore  wrote:

> On 27 March 2017 at 15:33, Donald Stufft  wrote:
> > What do you think about adding methods pathlib.Path.write_json and
> > pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
> > read_bytes?
> >
> >
> >
> > -1, I also think that write_* and read_* were mistakes to begin with.
>
> Text is (much) more general-use than JSON.
> Paul
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Paul Moore]

On 27 March 2017 at 15:33, Donald Stufft  wrote:
> What do you think about adding methods pathlib.Path.write_json and
> pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
> read_bytes?
>
>
>
> -1, I also think that write_* and read_* were mistakes to begin with.

Text is (much) more general-use than JSON.
Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Donald Stufft]

> On Mar 27, 2017, at 8:50 AM, Ram Rachum  wrote:
> 
> What do you think about adding methods pathlib.Path.write_json and 
> pathlib.Path.read_json , similar to write_text, write_bytes, read_text, 
> read_bytes?
> 


-1, I also think that write_* and read_* were mistakes to begin with.

—
Donald Stufft



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json andpathlib.Path.read_json

[Markus Meskanen]

-1, should we also include write_ini, write_yaml, etc?

A class cannot account for everyone who wants to use it in different ways.

On Mar 27, 2017 17:07, "Steve Dower"  wrote:

> It was enough of a benefit for text (and I never forget the argument order
> for writing text to a file, unlike json.dump(file_or_data?, data_or_file?) )
>
> +1
>
> Top-posted from my Windows Phone
> --
> From: Paul Moore 
> Sent: ‎3/‎27/‎2017 5:57
> To: Ram Rachum 
> Cc: python-ideas 
> Subject: Re: [Python-ideas] Add pathlib.Path.write_json
> andpathlib.Path.read_json
>
> On 27 March 2017 at 13:50, Ram Rachum  wrote:
> > This would make writing / reading JSON to a file a one liner instead of a
> > two-line with clause.
>
> That hardly seems like a significant benefit...
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json andpathlib.Path.read_json

[Steve Dower]

It was enough of a benefit for text (and I never forget the argument order for 
writing text to a file, unlike json.dump(file_or_data?, data_or_file?) )

+1

Top-posted from my Windows Phone

-Original Message-
From: "Paul Moore" 
Sent: ‎3/‎27/‎2017 5:57
To: "Ram Rachum" 
Cc: "python-ideas" 
Subject: Re: [Python-ideas] Add pathlib.Path.write_json 
andpathlib.Path.read_json

On 27 March 2017 at 13:50, Ram Rachum  wrote:
> This would make writing / reading JSON to a file a one liner instead of a
> two-line with clause.

That hardly seems like a significant benefit...

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Paul Moore]

On 27 March 2017 at 13:50, Ram Rachum  wrote:
> This would make writing / reading JSON to a file a one liner instead of a
> two-line with clause.

That hardly seems like a significant benefit...

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Ram Rachum]

Oh, and also it saves you from having to import json.

On Mon, Mar 27, 2017 at 2:50 PM, Ram Rachum  wrote:

> Hi guys,
>
> What do you think about adding methods pathlib.Path.write_json and
> pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
> read_bytes?
>
> This would make writing / reading JSON to a file a one liner instead of a
> two-line with clause.
>
>
> Thanks,
> Ram.
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add pathlib.Path.write_json and pathlib.Path.read_json

[Ram Rachum]

Hi guys,

What do you think about adding methods pathlib.Path.write_json and
pathlib.Path.read_json , similar to write_text, write_bytes, read_text,
read_bytes?

This would make writing / reading JSON to a file a one liner instead of a
two-line with clause.


Thanks,
Ram.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Proposal: Query language extension to Python (PythonQL)

[Paul Moore]

On 27 March 2017 at 10:54, Brice PARENT  wrote:
> I get it, but it's more a matter of perception. To me, the version I
> described is just Python, while yours is Python + specific syntax. As this
> syntax is only used in PyQL sub-language, it's not really Python any more...

... which is why I suspect that this discussion would be better
expressed as a suggestion that Python provide better support for
domain specific languages like the one PythonQL offers. In that
context, the "extended comprehension" format *would* be Python,
specifically it would simply be a DSL embedded in Python using
Python's standard features for doing that. Of course, that's just a
re-framing of the perception, and the people who don't like
sub-languages will be just as uncomfortable with DSLs.

However, it does put this request into the context of DSL support,
which is something that many languages provide, to a greater or lesser
extent. For Python, Guido's traditionally been against allowing the
language to be mutable in the way that DSLs permit, so in the first
instance it's likely that the PythonQL proposal will face a lot of
resistance. It's possible that PythonQL could provide a use case that
shows the benefits of allowing DSLs to such an extent that Guido
changes his mind, but that's not yet proven (and it's not really
something that's been argued here yet). And it does change the
discussion from being about who prefers which syntax, to being about
where we want the language to go in terms of DSLs.

Personally, I quite like limited DSL support (things like allowing
no-parenthesis function calls can make it possible to write code that
uses functions as if they were keywords). But it does impose a burden
on people supporting the code because they have to understand the
non-standard syntax. So I'm happy with Python's current choice to not
go down that route, even though I do find it occasionally limiting.

If I needed PythonQL features, I'd personally find Brice's class-based
approach quite readable/acceptable. I find PythonQL form nice also,
but not enough of an advantage to warrant all the extra
keywords/syntax etc.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Proposal: Query language extension to Python (PythonQL)

[Brice PARENT]

Le 27/03/17 à 10:55, Pavel Velikhov a écrit :

Hi Brice,

On 27 Mar 2017, at 10:17, Brice PARENT > wrote:


I prefer this a lot to the original syntax, and I really think this 
has much better chances to be integrated (if such an integration had 
to be done, and not kept as a separate module).


Also, maybe managing this with classes instead of syntax could also 
be done easily (without any change to Python), like this:


from pyql import PQL, Select, For, Where, GroupBy, Let

result = PQL(
Select("x", "sum_y"),
For("x", range(1,8)),
For("y",range(1,7)),
Where(lambda x, y: x %2==0andy %2!=0andx >y, "x", "y"),  # 
function, *[arguments to pass to the function]

Where("sum_y", lambda sum_y: sum_y %2!=0)
GroupBy("x"),
Let("sum_y", lambda y: sum(y), "y")
)




So here’s the deal: small queries will look pretty decent in pretty 
much all paradigms, ORM, or PythonQL or your proposal.
Once they get bigger and combine multiple pain points (say outerjoins, 
grouping and nested data) - then unless you have a

really clear and minimal language, folks will get confused and lost.

We’ve gone through a few query languages that failed, including XQuery 
and others, and the main reason was the need to learn
a whole new language and a bunch of libraries, nobody wanted to do it. 
So the main selling point behind PythonQL is: its Python

that folks hopefully know already, with just a few extensions.
I get it, but it's more a matter of perception. To me, the version I 
described is just Python, while yours is Python + specific syntax. As 
this syntax is only used in PyQL sub-language, it's not really Python 
any more...
Also, what I like with what I used, is that it is object-based, which 
allows any part of the query to be reusable or built dynamically. We 
might also extend such a PQL object's constructor to embed automatically 
whatever default parameters or database connection we want, or shared 
behaviours, like:


class MyPQL(PQL):
def get_limit(self):
if self.limit is not None:
return self.limit

return 10

def __init__(self, *args):
args.append(Let("sum_y", lambda y: sum(y), "y"))
args.append(GroupBy("x"))
super().__init__(*args)

result = MyPQL(
Select("x", "sum_y"),
For("x", range(1,8)),
For("y",range(1,7)),
Where(lambda x, y: x %2==0andy %2!=0andx >y, "x", "y"),
Where("sum_y", lambda sum_y: sum_y %2!=0)
)

Big queries, this way, may be split into smaller parts. And it allows 
you to do the following in a single query, instead of having to write 
one big for each condition


where_from = [For("x", range(1,8)),For("y",range(1,7))]
where = [Where(lambda x, y: x %2==0andy %2!=0andx >y, "x", "y")]
if filter_sum_y:
where.append(Where("sum_y", lambda sum_y: sum_y %2!=0))

if group_by is not None:
grouping = GroupBy("x")

result = MyPQL(Select("x", "sum_y"), *where_from, *where, *grouping)

Side note : I'm not a big database user, I mostly use ORMs (Django's and 
PonyORM depending on the projects) to access PgSQL and SQLite (for unit 
testing), so I might not even have use cases for what you're trying to 
solve. I just give my point of view here to explain what I think could 
be more easily integrated and (re)used. And as I'm a big fan of the DRY 
mentality, I'm not a fan of the syntax-chaining things (as well as I 
don't really like big nested comprehensions).


-Brice
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Proposal: Query language extension to Python (PythonQL)

[Pavel Velikhov]

Hi Brice,

> On 27 Mar 2017, at 10:17, Brice PARENT  wrote:
> 
> I prefer this a lot to the original syntax, and I really think this has much 
> better chances to be integrated (if such an integration had to be done, and 
> not kept as a separate module).
> 
> Also, maybe managing this with classes instead of syntax could also be done 
> easily (without any change to Python), like this: 
> from pyql import PQL, Select, For, Where, GroupBy, Let
> 
> result = PQL(
> Select("x", "sum_y"),
> For("x", range(1, 8)),
> For("y", range(1, 7)),
> Where(lambda x, y: x % 2 == 0 and y % 2 != 0 and x > y, "x", "y"),  # 
> function, *[arguments to pass to the function]
> Where("sum_y", lambda sum_y: sum_y % 2 != 0)
> GroupBy("x"),
> Let("sum_y", lambda y: sum(y), "y")
> )
> 
> 

So here’s the deal: small queries will look pretty decent in pretty much all 
paradigms, ORM, or PythonQL or your proposal.
Once they get bigger and combine multiple pain points (say outerjoins, grouping 
and nested data) - then unless you have a
really clear and minimal language, folks will get confused and lost.

We’ve gone through a few query languages that failed, including XQuery and 
others, and the main reason was the need to learn
a whole new language and a bunch of libraries, nobody wanted to do it. So the 
main selling point behind PythonQL is: its Python
that folks hopefully know already, with just a few extensions.
> (to be defined more precisely, I don't really like to rely on case to 
> differentiate the "for" keyword and the "For" class, which by the way could 
> be inherited from a more general "From" class, allowing to get the data from 
> a database, pure python, a JSON/csv/xml file/object, or anything else.)
> 
> With a nice lazy evaluation, in the order of the arguments of the 
> constructor, I suppose you could achieve everything you need, and have an 
> easily extendable syntax (create new objects between versions of the module, 
> without ever having to create new keywords). There is no new parsing, and you 
> already have the autocompletion of your IDE if you annotate correctly your 
> code.
> 
> You could even have this : 
> 
> query = PQL(
> Select("x", "sum_y"),
> Where("x", "y", lambda x, y: x % 2 == 0 and y % 2 != 0 and x > y),
> Where("sum_y", lambda sum_y: sum_y % 2 != 0)
> GroupBy("x"),
> Let("sum_y", lambda y: sum(y), "y")  # [name of the new var], function, 
> *[arguments to pass to the function]
> )
> query.execute(x=range(1, 8), y=range(1, 7))
> or 
> query.execute(PgDatabase(**dbsettings))
> 
> -Brice
> 
> Le 25/03/17 à 16:40, Kyle Lahnakoski a écrit :
>> 
>> Pavel,
>> 
>> I like PythonQL. I perform a lot of data transformation, and often find 
>> Python's list comprehensions too limiting; leaving me wishing for LINQ-like 
>> language features. 
>> As an alternative to extending Python with PythonQL, Terry Reedy suggested 
>> interpreting a DSL string, and Pavel Velikhov alluded to using magic method 
>> tricks found in ORM libraries. I can see how both these are not 
>> satisfactory.  
>> A third alternative could be to encode the query clauses as JSON objects. 
>> For example:  
>> result = [ select (x, sum_y)
>>for x in range(1,8), 
>>y in range(1,7)
>>where x % 2 == 0 and y % 2 != 0 and x > y
>>group by x
>>let sum_y = sum(y)
>>where sum_y % 2 != 0
>>]
>> result = pq([
>> {"select":["x", "sum_y"]},
>> {"for":{"x": range(1,8), "y": range(1,7)}},
>> {"where": lambda x,y: x % 2 == 0 and y % 2 != 0 and x > y},
>> {"groupby": "x"},
>> {"with":{"sum_y":{"SUM":"y"}},
>> {"where": {"neq":[{"mod":["sum_y", 2]}, 0]}}
>>  ])
>> This representation does look a little lispy, and it may resemble PythonQL's 
>> parse tree. I think the benefits are:
>> 
>> 1) no python language change
>> 2) easier to parse
>> 3) better than string-based DSL for catching syntax errors
>> 4) {"clause": parameters} format is flexible for handling common query 
>> patterns **
>> 5) works in javascript too
>> 6) easy to compose with automation (my favorite) 
>> 
>> It is probably easy for you to see the drawbacks.
>> 
>> 
>> ** The `where` clause can accept a native lambda function, or an expression 
>> tree
>> 
>> 
>> 
>> 
>> 
>> "If you are writing a loop, you are doing it wrong!" :)
>> 
>> 
>> On 2017-03-24 11:10, Pavel Velikhov wrote:
>>> Hi folks!
>>> 
>>>   We started a project to extend Python with a full-blown query language 
>>> about a year ago. The project is call PythonQL, the links are given below 
>>> in the references section. We have implemented what is kind of an alpha 
>>> version now, and gained some experience and insights about why and where 
>>> this is really useful. So I’d like to share those with you and gather some 
>>> opinions whether you think we should try to include these extensions in the 
>>> Python core.
>>> 
>>> Intro
>>> 
>>>   What we have done is (m