[Python-ideas] Re: Improvement: __json__

[Andrew Barnert via Python-ideas]

On Apr 7, 2020, at 18:10, Wes Turner  wrote:
> 
> 
> *That* should read as "are not sufficient".
> 
> Stuffing all of those into annotations is going to be cumbersome; resulting 
> in there being multiple schema definitions to keep synchronized and validate 
> data according to.
> 
> I think generating some amalgamation of JSONLD @context & SHACL and JSON 
> schema would be an interesting exercise. You'd certainly want to add more 
> information to the generated schema than just the corresponding Python types :
> - type URI(s) that correspond to the Python primitive types
> - JSON schema format strings
> - JSON schema length
> - TODO JSON schema [...]

Not everything in the world has to be built around RDF semantic triples. In 
fact, most things don’t have to be. That’s why are a lot more things out there 
using plain old JSON Schema for their APIs and formats than using JSON-LD. And 
even more things just using free form JSON.

And for either of those, type annotations are sufficient. You can serialize any 
instance of Spam to JSON, and deserialize JSON (that you know represents a 
Spam) to an equal Spam instance, as long as you know what the name and type of 
every attribute of Spam is (and all of those types are number/string/book/null, 
types that match the same qualifications as Spam, lists of such a type, and 
dicts mapping str to such a type). Which is guaranteed to be knowable for 
dataclasses even without any external information. Or any classes with a 
(correct) accompanying schema. Or just any classes you design around such a 
serialization system.

The fact that you don’t have, e.g., a URI with metadata about Spam doesn’t in 
any way stop any of that from working, or being useful. Type annotations are 
sufficient for this purpose.

In fact, even type annotations aren’t necessary. Any value that can pickle, you 
can just msg=json.dumps(b64_encode(pickle.dumps(obj and 
obj=pickle.loads(b64_decode(json.loads(msg and you’ve got working JSON 
serialization. What type annotations add is JSON serialization that’s human 
readable/editable, or computer verifiable, or both. You don’t need JSON-LD 
unless you’re not just building APIs, but meta-indexes of APIs or automatic API 
generators or something.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/P7PYTO663PJLJ45W6TPRDEDMA2NG4RXZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improvement: __json__

[Wes Turner]

You don't need JSON at all in order to serialize and deserialize instances
of Python objects and primitives.

Pickle (or e.g. Arrow + [parquet,]) handles nested, arbitrary complex types
efficiently and without dataclasses and/or type annotations.

I don't see the value in using JSON to round-trip from Python to the same
Python code.

External schema is far more useful than embedding part of an ad-hoc nested
object schema in type annotations that can't also do or even specify data
validations.

You can already jsonpickle data classes. If you want to share or just
publish data, external schema using a web standard is your best bet.

On Wed, Apr 8, 2020, 3:30 AM Andrew Barnert  wrote:

> On Apr 7, 2020, at 18:10, Wes Turner  wrote:
> >
> > 
> > *That* should read as "are not sufficient".
> >
> > Stuffing all of those into annotations is going to be cumbersome;
> resulting in there being multiple schema definitions to keep synchronized
> and validate data according to.
> >
> > I think generating some amalgamation of JSONLD @context & SHACL and JSON
> schema would be an interesting exercise. You'd certainly want to add more
> information to the generated schema than just the corresponding Python
> types :
> > - type URI(s) that correspond to the Python primitive types
> > - JSON schema format strings
> > - JSON schema length
> > - TODO JSON schema [...]
>
> Not everything in the world has to be built around RDF semantic triples.
> In fact, most things don’t have to be. That’s why are a lot more things out
> there using plain old JSON Schema for their APIs and formats than using
> JSON-LD. And even more things just using free form JSON.
>
> And for either of those, type annotations are sufficient. You can
> serialize any instance of Spam to JSON, and deserialize JSON (that you know
> represents a Spam) to an equal Spam instance, as long as you know what the
> name and type of every attribute of Spam is (and all of those types are
> number/string/book/null, types that match the same qualifications as Spam,
> lists of such a type, and dicts mapping str to such a type). Which is
> guaranteed to be knowable for dataclasses even without any external
> information. Or any classes with a (correct) accompanying schema. Or just
> any classes you design around such a serialization system.
>
> The fact that you don’t have, e.g., a URI with metadata about Spam doesn’t
> in any way stop any of that from working, or being useful. Type annotations
> are sufficient for this purpose.
>
> In fact, even type annotations aren’t necessary. Any value that can
> pickle, you can just msg=json.dumps(b64_encode(pickle.dumps(obj and
> obj=pickle.loads(b64_decode(json.loads(msg and you’ve got working JSON
> serialization. What type annotations add is JSON serialization that’s human
> readable/editable, or computer verifiable, or both. You don’t need JSON-LD
> unless you’re not just building APIs, but meta-indexes of APIs or automatic
> API generators or something.
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PZZC3LHS56NUVDK4I7WHXMUPD24ZX4J5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "except BaseException with f as result"

[Stephen J. Turnbull]

Andrew Barnert writes:

 > I don’t like the idea either; but I think I like your version even less.
 > 
 > It reads perfectly well, but with the wrong meaning. Even though I
 > know what you’re intending, I can’t make myself read that as result
 > getting bound to what f returns, only as result getting bound to
 > the exception.

No, you don't know what I was intending. ;-)  *I* did intend 'result'
to be bound to the exception (that's *why* I changed the order of
clauses), but I think you're right: *Soni* intended it to bound to the
return of the handler.  We'll have to ask to be sure, though...

 > clearly result is getting bound to something to do with f.

Well, I had in mind that it would be f's *argument* (what else are you
going to pass to the handler?) and have the usual semantics of giving
a name to the exception in case you wanted further processing in the
body.

Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VVXKUS3QKTTQYFMQPHAREHYDCFCRXUHS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "except BaseException with f as result"

[Joao S. O. Bueno]

On Tue, 7 Apr 2020 at 23:17, Greg Ewing  wrote:

> On 8/04/20 1:14 pm, Soni L. wrote:
>
> >  def get_property_values(self, prop):
> >  try:
> >  factory = self.get_supported_properties()[prop]
> >  except KeyError with keyerror_handler;
> >  iterator = factory(self._obj)
> >  try:
> >  first = next(iterator)
> >  except abdl.exceptions.ValidationError with validation_handler;
> >  except StopIteration with stop_handler as return
> >  return itertools.chain([first], iterator)
>
> I don't think special syntax is warranted for this. You can write:
>
>  try:
>  first = next(iterator)
>  except abdl.exceptions.ValidationError as e:
>  validation_handler(e)
>  except StopIteration as e:
>  return stop_handler(e)
>
>
I agree -  and this is far more readable than any
of the propositions so far. (Like, for an order of magnitude).

I'd wait for Soni to try to explain why this would not
fit his use case, before proceeding with the
bikeshedding.





> --
> Greg
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/AERIZI4MRPFBJ62YAOIWO4OSXLDSMTRK/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Z5QFJEQVW4TTB54X5MV3HUKE5TEBK63N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "except BaseException with f as result"

[Soni L.]

On 2020-04-08 10:12 a.m., Joao S. O. Bueno wrote:



On Tue, 7 Apr 2020 at 23:17, Greg Ewing > wrote:


On 8/04/20 1:14 pm, Soni L. wrote:

>      def get_property_values(self, prop):
>      try:
>      factory = self.get_supported_properties()[prop]
>      except KeyError with keyerror_handler;
>      iterator = factory(self._obj)
>      try:
>      first = next(iterator)
>      except abdl.exceptions.ValidationError with
validation_handler;
>      except StopIteration with stop_handler as return
>      return itertools.chain([first], iterator)

I don't think special syntax is warranted for this. You can write:

     try:
         first = next(iterator)
     except abdl.exceptions.ValidationError as e:
         validation_handler(e)
     except StopIteration as e:
         return stop_handler(e)


I agree -  and this is far more readable than any
of the propositions so far. (Like, for an order of magnitude).

I'd wait for Soni to try to explain why this would not
fit his use case, before proceeding with the
bikeshedding.


have you looked at the syntax highlighting of the thing?

it sees keywords *everywhere* (which is right) it's basically 
unmaintainable. having the exception handlers have an exceptional syntax 
would set them apart from the rest of the code and make the whole thing 
much easier to read and maintain.


it's hard to reason about it if your eyes just keep jumping into the 
except bodies because they have the same indentation and everything as 
the normal code. I am gonna say my proposal improves accessibility, even 
if it doesn't make a difference to most ppl.






-- 
Greg

___
Python-ideas mailing list -- python-ideas@python.org

To unsubscribe send an email to python-ideas-le...@python.org

https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at

https://mail.python.org/archives/list/python-ideas@python.org/message/AERIZI4MRPFBJ62YAOIWO4OSXLDSMTRK/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Z5QFJEQVW4TTB54X5MV3HUKE5TEBK63N/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MB5DGQCZWFJSEMMS3U4Q2E6LRFJ76WVX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "except BaseException with f as result"

[Soni L.]

On 2020-04-08 10:22 a.m., Soni L. wrote:



On 2020-04-08 10:12 a.m., Joao S. O. Bueno wrote:



On Tue, 7 Apr 2020 at 23:17, Greg Ewing > wrote:


On 8/04/20 1:14 pm, Soni L. wrote:

>      def get_property_values(self, prop):
>      try:
>      factory = self.get_supported_properties()[prop]
>      except KeyError with keyerror_handler;
>      iterator = factory(self._obj)
>      try:
>      first = next(iterator)
>      except abdl.exceptions.ValidationError with
validation_handler;
>      except StopIteration with stop_handler as return
>      return itertools.chain([first], iterator)

I don't think special syntax is warranted for this. You can write:

     try:
         first = next(iterator)
     except abdl.exceptions.ValidationError as e:
         validation_handler(e)
     except StopIteration as e:
         return stop_handler(e)


I agree -  and this is far more readable than any
of the propositions so far. (Like, for an order of magnitude).

I'd wait for Soni to try to explain why this would not
fit his use case, before proceeding with the
bikeshedding.


have you looked at the syntax highlighting of the thing?

it sees keywords *everywhere* (which is right) it's basically 
unmaintainable. having the exception handlers have an exceptional 
syntax would set them apart from the rest of the code and make the 
whole thing much easier to read and maintain.


it's hard to reason about it if your eyes just keep jumping into the 
except bodies because they have the same indentation and everything as 
the normal code. I am gonna say my proposal improves accessibility, 
even if it doesn't make a difference to most ppl.


(also: I'm a they. forgot to include that in the reply. sorry.)







-- 
Greg

___
Python-ideas mailing list -- python-ideas@python.org

To unsubscribe send an email to python-ideas-le...@python.org

https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at

https://mail.python.org/archives/list/python-ideas@python.org/message/AERIZI4MRPFBJ62YAOIWO4OSXLDSMTRK/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list --python-ideas@python.org
To unsubscribe send an email topython-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived 
athttps://mail.python.org/archives/list/python-ideas@python.org/message/Z5QFJEQVW4TTB54X5MV3HUKE5TEBK63N/
Code of Conduct:http://python.org/psf/codeofconduct/




___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LGSGCMMK73HPWJKQRD7LSX6DJ4UPTU5Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "except BaseException with f as result"

[Greg Ewing]

On 9/04/20 1:22 am, Soni L. wrote:

it's hard to reason about it if your eyes just keep jumping into the 
except bodies because they have the same indentation and everything as 
the normal code. I am gonna say my proposal improves accessibility, even 
if it doesn't make a difference to most ppl.


Would you find this easier to read?

try:
first = next(iterator)
except abdl.exceptions.ValidationError as e: validation_handler(e)
except StopIteration as e: return stop_handler(e)

--
Greg
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VHYH3WOQZDG56PJ4CISYMH3W6HIT5Y76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "except BaseException with f as result"

[Soni L.]

On 2020-04-08 10:33 a.m., Greg Ewing wrote:

On 9/04/20 1:22 am, Soni L. wrote:

it's hard to reason about it if your eyes just keep jumping into the 
except bodies because they have the same indentation and everything 
as the normal code. I am gonna say my proposal improves 
accessibility, even if it doesn't make a difference to most ppl.


Would you find this easier to read?

    try:
    first = next(iterator)
    except abdl.exceptions.ValidationError as e: validation_handler(e)
    except StopIteration as e: return stop_handler(e)


yeah that'd work

... I can do that today, can't I?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LLFCO2SHGIUQ543XHOIBKEG7ITX3ZFFN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "except BaseException with f as result"

[Greg Ewing]

On 9/04/20 1:37 am, Soni L. wrote:


On 2020-04-08 10:33 a.m., Greg Ewing wrote:


    try:
    first = next(iterator)
    except abdl.exceptions.ValidationError as e: validation_handler(e)
    except StopIteration as e: return stop_handler(e)


... I can do that today, can't I?


Yep!

--
Greg
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WO23EYGCMGKWYG5HUED4QBSMMJE2QVCR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Live variable analysis -> earlier release?

[Guido van Rossum]

Look at the following code.

def foo(a, b):
x = a + b
if not x:
return None
sleep(1)  # A calculation that does not use x
return a*b

This code DECREFs x when the frame is exited (at the return statement). But
(assuming) we can clearly see that x is not needed during the sleep
(representing a big calculation), we could insert a "del x" statement
before the sleep.

I think our compiler is smart enough to find out *some* cases where it
could safely insert such del instructions. And this would potentially save
some memory. (For example, in the above example, if a and b are large
lists, x would be an even larger list, and its memory could be freed
earlier.)

For sure, we could manually insert del statements in code where it matters
to us. But if the compiler could do it in all code, regardless of whether
it matters to us or not, it would probably catch some useful places where
we wouldn't even have thought of this idea, and we might see a modest
memory saving for most programs.

Can anyone tear this idea apart?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HLIWSU3WBGFCWBF4N5WPALAIX5XBTXRV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Brandt Bucher]

> Can anyone tear this idea apart?

`exec` and `eval` come to mind... I'm sure there are also nasty `inspect` hacks 
this could break, too?

def foo(a, b):
x = a + b
if not x:
return None
sleep(1)  # A calculation that does not use x
return eval("x")
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GXWFXRV7GQRISIS727RQCS6WRTEBDHN4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Alex Hall]

This would break uses of locals(), e.g.

def foo(a, b):
x = a + b
if not x:
return None
del x
print('{x}, {a}, {b}'.format(**locals()))
return a * b

foo(1, 2)

Plus if the calculation raises an exception and I'm looking at the report
on Sentry, I'd like to see the values of all variables. In particular I
might have expected the function to return early and I want to see what `x`
was.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SDFPACSEXIQJPRJV334G5553IPDLD5JL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Tim Peters]

[Guido]
> Look at the following code.
>
> def foo(a, b):
> x = a + b
> if not x:
> return None
> sleep(1)  # A calculation that does not use x
> return a*b
>
> This code DECREFs x when the frame is exited (at the return statement).
> But (assuming) we can clearly see that x is not needed during the sleep
> (representing a big calculation), we could insert a "del x" statement before
> the sleep.
>
> I think our compiler is smart enough to find out *some* cases where it could
> safely insert such del instructions. And this would potentially save some
> memory. (For example, in the above example, if a and b are large lists,
> x would be an even larger list, and its memory could be freed earlier.)
>
> For sure, we could manually insert del statements in code where it
> matters to us. But if the compiler could do it in all code, regardless
> of whether it matters to us or not, it would probably catch some useful
> places where we wouldn't even have thought of this idea, and we
> might see a modest memory saving for most programs.
>
> Can anyone tear this idea apart?

My guess:  it would overwhelmingly free tiny objects, giving a
literally unmeasurable (just theoretically provable) memory savings,
at the cost of adding extra trips around the eval loop.  So not really
attractive to me.  But when I leave "large" temp objects hanging and
give a rip, I already stick in "del" statements anyway.  Very rarely,
but it happens.

Which is addressing it at a higher level than any other feedback
you're going to get ;-)  Of course there can be visible consequences
when people are playing with introspection gimmicks.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LFUJ4SLORLT6J75IULDFL7MK3VUKVQMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Antoine Pitrou]

On Wed, 8 Apr 2020 09:53:41 -0700
Guido van Rossum  wrote:
> Look at the following code.
> 
> def foo(a, b):
> x = a + b
> if not x:
> return None
> sleep(1)  # A calculation that does not use x
> return a*b
> 
> This code DECREFs x when the frame is exited (at the return statement). But
> (assuming) we can clearly see that x is not needed during the sleep
> (representing a big calculation), we could insert a "del x" statement
> before the sleep.
> 
> I think our compiler is smart enough to find out *some* cases where it
> could safely insert such del instructions. And this would potentially save
> some memory. (For example, in the above example, if a and b are large
> lists, x would be an even larger list, and its memory could be freed
> earlier.)
> 
> For sure, we could manually insert del statements in code where it matters
> to us. But if the compiler could do it in all code, regardless of whether
> it matters to us or not, it would probably catch some useful places where
> we wouldn't even have thought of this idea, and we might see a modest
> memory saving for most programs.
> 
> Can anyone tear this idea apart?

The problem is if variable `x` has a side-effect destructor.  It's
certainly not a common idiom to keep a resource alive simply by keeping
a Python object around (you should probably use `with` instead), but I'm
sure some people do it.

FWIW, Numba does something similar to try and release memory earlier.
But Numba certainly doesn't claim to support general-purpose Python
code :-)

Regards

Antoine.



___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FRGMBNJ4CULCW6VWKMV43WU4SQWVLFM2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Guido van Rossum]

On Wed, Apr 8, 2020 at 10:05 AM Alex Hall  wrote:

> This would break uses of locals(), e.g.
>

Hm, okay, so suppose the code analysis was good enough to recognize most
un-obfuscated uses of locals(), exec() and eval() (and presumably
sys._getframe() -- IIUC there's already a Python implementation that
generates less optimal code for functions where it detects usage of
sys._getframe(), maybe IronPython).


> Plus if the calculation raises an exception and I'm looking at the report
> on Sentry, I'd like to see the values of all variables. In particular I
> might have expected the function to return early and I want to see what `x`
> was.
>

That's a very valid objection. For simpletons like myself who just use pdb
it could also be problematic. So at the very least there would have to be a
way to turn it off, and probably it should have to be requested explicitly
(maybe just with -O).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CNON3MYEASWG6WLW5C2QWRSTMQVZDDCX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[MRAB]

On 2020-04-08 17:53, Guido van Rossum wrote:

Look at the following code.

def foo(a, b):
     x = a + b
     if not x:
     return None
     sleep(1)  # A calculation that does not use x
     return a*b

This code DECREFs x when the frame is exited (at the return statement). 
But (assuming) we can clearly see that x is not needed during the sleep 
(representing a big calculation), we could insert a "del x" statement 
before the sleep.


I think our compiler is smart enough to find out *some* cases where it 
could safely insert such del instructions. And this would potentially 
save some memory. (For example, in the above example, if a and b are 
large lists, x would be an even larger list, and its memory could be 
freed earlier.)


For sure, we could manually insert del statements in code where it 
matters to us. But if the compiler could do it in all code, regardless 
of whether it matters to us or not, it would probably catch some useful 
places where we wouldn't even have thought of this idea, and we might 
see a modest memory saving for most programs.


Can anyone tear this idea apart?


Can you be sure that 'sleep' isn't inspecting the stack and using x?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5KYLXDL6MZV3FETTIA5JIB5YSRUHAKZ2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Guido van Rossum]

On Wed, Apr 8, 2020 at 10:13 AM Tim Peters  wrote:

> My guess:  it would overwhelmingly free tiny objects, giving a
> literally unmeasurable (just theoretically provable) memory savings,
> at the cost of adding extra trips around the eval loop.  So not really
> attractive to me.


Yeah, the extra opcodes might well kill the idea for good.


> But when I leave "large" temp objects hanging and
> give a rip, I already stick in "del" statements anyway.  Very rarely,
> but it happens.
>

I recall that in the development of asyncio there were a few places where
we had to insert del statements, not so much to free a chunk of memory, but
to cause some destructor or finalizer to run early enough. (IIRC not right
at that moment, but at some later moment even if some Futures are still
alive.) Those issues took considerable effort to find, and would
conceivably have been prevented by this proposal.


> Which is addressing it at a higher level than any other feedback
> you're going to get ;-)  Of course there can be visible consequences
> when people are playing with introspection gimmicks.
>

Plus about all the debugging that would ensue because
destructors/finalizers are running *earlier* than expected. ;-)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZRTC2LOEFVGKHXXNVZLLOQSQJ24IAMP2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Antoine Pitrou]

On Wed, 8 Apr 2020 10:18:47 -0700
Guido van Rossum  wrote:
> 
> > But when I leave "large" temp objects hanging and
> > give a rip, I already stick in "del" statements anyway.  Very rarely,
> > but it happens.
> >  
> 
> I recall that in the development of asyncio there were a few places where
> we had to insert del statements, not so much to free a chunk of memory, but
> to cause some destructor or finalizer to run early enough. (IIRC not right
> at that moment, but at some later moment even if some Futures are still
> alive.) Those issues took considerable effort to find, and would
> conceivably have been prevented by this proposal.

If code relies on life variable analysis for correctness, it implies
other Python implementations must implement it with exactly the same
results.

Regards

Antoine.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/V6AGHX43FB2EF6RZGQQJTYGQTRVX3W2F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Gregory P. Smith]

On Wed, Apr 8, 2020 at 10:21 AM Guido van Rossum  wrote:

> On Wed, Apr 8, 2020 at 10:05 AM Alex Hall  wrote:
>
>> This would break uses of locals(), e.g.
>>
>
> Hm, okay, so suppose the code analysis was good enough to recognize most
> un-obfuscated uses of locals(), exec() and eval() (and presumably
> sys._getframe() -- IIUC there's already a Python implementation that
> generates less optimal code for functions where it detects usage of
> sys._getframe(), maybe IronPython).
>
>
>> Plus if the calculation raises an exception and I'm looking at the report
>> on Sentry, I'd like to see the values of all variables. In particular I
>> might have expected the function to return early and I want to see what `x`
>> was.
>>
>
> That's a very valid objection. For simpletons like myself who just use pdb
> it could also be problematic. So at the very least there would have to be a
> way to turn it off, and probably it should have to be requested explicitly
> (maybe just with -O).
>

Even though it seems like a pedantically correct behavior to toss something
after no future direct references to it are detected, it would just break
so much existing code that the upgrade to an interpreter doing this would
be a nightmare.  Yes, people should in theory be using a context manager or
explicit reference to things that need to live, but what _is_ an explicit
reference if not the local scope?  In practice code all over the place
assumes "still in local scope" means the reference lives on.  Including
wrapped C/C++ code where the destructor frees underlying resources that are
needed by other things outside of CPython's view.

Different behavior when debugging or adding a debug print vs when running
normally is bad.  Seeing surprising behavior changes due to the addition or
removal of code later in a scope that inadvertently changes when
something's destructor is called is action at a distance and hard to
debug.  In my experience, people understand scope-based lifetime.  It is
effectively the same as modern C++ STL based pointer management semantics;
destruction only happens after going out of scope or when explicitly called
for.

Making it optional?  It'd need to be at least at a per-file level (from
__future__ style... but called something else as this isn't likely to be
our future default).  A behavior change of this magnitude globally for a
program with -O would just reinforce the existing practice of nobody
practically using -O.  (does anyone actually run their tests under -O let
alone deploy with -O?  I expect that to be a single digit % or lower
minority)

-gps



>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/CNON3MYEASWG6WLW5C2QWRSTMQVZDDCX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/R7EAK4Q2MVED44EZQMKBQASIN24UIYFX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Brett Cannon]

On Wed, Apr 8, 2020 at 10:23 AM Guido van Rossum  wrote:

> On Wed, Apr 8, 2020 at 10:05 AM Alex Hall  wrote:
>
>> This would break uses of locals(), e.g.
>>
>
> Hm, okay, so suppose the code analysis was good enough to recognize most
> un-obfuscated uses of locals(), exec() and eval() (and presumably
> sys._getframe() -- IIUC there's already a Python implementation that
> generates less optimal code for functions where it detects usage of
> sys._getframe(), maybe IronPython).
>

Yep, it's IronPython.

-Brett


>
>
>> Plus if the calculation raises an exception and I'm looking at the report
>> on Sentry, I'd like to see the values of all variables. In particular I
>> might have expected the function to return early and I want to see what `x`
>> was.
>>
>
> That's a very valid objection. For simpletons like myself who just use pdb
> it could also be problematic. So at the very least there would have to be a
> way to turn it off, and probably it should have to be requested explicitly
> (maybe just with -O).
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/CNON3MYEASWG6WLW5C2QWRSTMQVZDDCX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VBP4ETELD4GCLFGYBPVEI7S3W7AJZBYQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Andrew Barnert via Python-ideas]

On Apr 8, 2020, at 09:57, Guido van Rossum  wrote:
> 
> 
> Look at the following code.
> 
> def foo(a, b):
> x = a + b
> if not x:
> return None
> sleep(1)  # A calculation that does not use x
> return a*b
> 
> This code DECREFs x when the frame is exited (at the return statement). But 
> (assuming) we can clearly see that x is not needed during the sleep 
> (representing a big calculation), we could insert a "del x" statement before 
> the sleep.
> 
> I think our compiler is smart enough to find out *some* cases where it could 
> safely insert such del instructions.

It depends on how much you’re willing to break and still call it “safely”.

def sleep(n):
global store
store = inspect.current_frame().f_back.f_locals['x']

This is a ridiculous example, but it shows that you can’t have all of Python’s 
dynamic functionality and still know when locals are dead. And there are less 
ridiculous examples with different code. If foo actually calls eval, exec, 
locals, vars, etc., or if it has a nested function that nonlocals x, etc., how 
can we spot that at compile time and keep x alive?

Maybe that’s ok. After all, that code doesn’t work in a Python implementation 
that doesn’t have stack frame support. Some of the other possibilities might be 
more portable, but I don’t know without digging in further.

Or maybe you can add new restrictions to what locals and eval and so on 
guarantee that will make it ok? Some code will break, but only rare “expert” 
code, where the authors will know how to work around it.

Or, if not, it’s definitely fine as an opt-in optimization: decorate the 
function with @deadlocals and that decorator scans the bytecode and finds any 
locals that are dead assuming there’s no use of locals/eval/cells/etc. and, 
because you told it to assume that by opting in to the decorator, it can insert 
a DELETE_FAST safely.

People already do similar things today—e.g., I’ve (only once in live code, but 
that’s still more than zero) used a @fastconst decorator that turns globals 
into consts on functions that I know are safe and are bottlenecks, and this 
would be no different. And of course you can add a recursive class decorator, 
or an import hook (or maybe even a command line flag or something) that enables 
it everywhere (maybe with a @nodeadlocals decorator for people who want it 
_almost_ everywhere but need to opt out one or two functions).

Did Victor Stinner explore this as one of the optimizations for FAT Python/PEP 
511/etc.? Maybe not, since it’s not something you can insert a guard, 
speculatively do, and then undo if the guard triggers, which was I think his 
key idea.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OIGCRV464VJW3FRRBBK25XSNQYGWID7N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Optimize out unused variables

[Serhiy Storchaka]

I doubted whether or not to write, but since Guido has already touched a 
similar topic (see "Live variable analysis -> earlier release"), so will 
I write.


It is common to assign some values which are never used to variables 
because the syntax demands this. For example:


head, _, rest = path.partition('/')
first, second, *_ = line.split()
for _ in range(10): ...
[k for k, _ in pairs]

What if make such assignments no-op? It will only work with some 
limitations:


1. The variable is local. Not global, not nonlocal, not cell.
2. The variable name must start with '_'. Otherwise there would be 
larger risk of breaking a legal code which uses locals() with 
str.format() or like.
3. "del" is considered a use of the variable. So explicitly deleted 
variables will not be optimized out.
4. Star-assignment still consumes the iterator. It just might not to 
keep all values in memory.


STORE_FAST will be replaced with POP_TOP in these cases.

I wrote some code few weeks ago, and it is not too complex. My doubts 
are only that the benefit of the optimization with the above limitations 
is very restricted.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PH7R2R3FTLT3I734FTTSN57P4PKUZ7L7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improvement: __json__

[Christopher Barker]

On Wed, Apr 8, 2020 at 1:18 AM Wes Turner  wrote:

>
> I don't see the value in using JSON to round-trip from Python to the same
> Python code.
>

There is a bit of a value: it's a human-readable format that matches the
Python Data model pretty well. I've used it for that reason. Maybe I should
be using yaml or something instead, but it's nice to use something common.

I have thought about using "PYSON" -- which would better match the Python
data model, but never got around to formalizing that.

But yes, the real advantage to JSON is interaction with non-python systems.

>
> External schema is far more useful than embedding part of an ad-hoc nested
> object schema in type annotations that can't also do or even specify data
> validations.
>

I suppose so -- I really need to see if I can make use of JSONSchema -- it
would be nice to specify teh schema in one place, and be able to build
Python objects, and Javascript objects, and theoretically all kinds of
other implementations as well. Someone may have done that, I need to go
look.

NOTE: JSON-LD keeps coming up in these thrteads, but that really seems like
an orthogonal issue. Maybe there should be JSON-LD support for Python (is
there already?) but that shouldn't impact the core json library, nor a
__json__ magic method.

-CHB

-- 
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J3BCRNHEC3CLE2LLKB5UTAMSGTZUXZZN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improvement: __json__

[Andrew Barnert via Python-ideas]

On Apr 8, 2020, at 01:18, Wes Turner  wrote:
> 
> I don't see the value in using JSON to round-trip from Python to the same 
> Python code.
> 
> External schema is far more useful than embedding part of an ad-hoc nested 
> object schema in type annotations that can't also do or even specify data 
> validations.

But dataclasses with type annotations can express a complete JSON Schema. Or, 
of course, in an ad hoc schema only published in human readable form, as most 
web APIs use today.

> You can already jsonpickle data classes. If you want to share or just publish 
> data, external schema using a web standard is your best bet.

Sure, but you don’t need JSON-LD for that. Again, the fact that type 
annotations are insufficient to represent semantic triples is irrelevant. They 
are sufficient for the case of writing code to parse what YouTube gives you, or 
to provide a documented API that you design to be consumed by other people in 
JS, or to generate a JSON Schema from your legacy collection of classes that 
you can then publish, or to validate that a dataclass hierarchy matches a 
published schema, and so on. All of which are useful.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XDJELGTCVSDHFNPQA6556CUWC5XUCXTW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improvement: __json__

[Wes Turner]

I'm aware of a couple Python implementations of JSON-LD: pyld [1] and
rdflib-jsonld [2]. But you don't need a JSON-LD parser to parse or produce
JSON-LD: You can just frame the data in the json document correctly such
that other tools can easily parse your necessarily complex data types
stored within the limited set of primitive datatypes supported by JSON.

We should welcome efforts to support linked data in Python. TimBL created
the web on top of the internet in order that we could share resources in
order to collaborate on science. In order to collaborate on science, we
need to be able to share, discover, merge, join, concatenate, analyze, and
compare Datasets. TimBL's 5-star Open Data plan [3] justifies the costs and
benefits of sharing LOD: Linked Open Data.

★
   make your stuff available on the Web (whatever format) under an open
license

★★
   make it available as structured data (e.g., Excel instead of image scan
of a table)

★★★
   make it available in a non-proprietary open format (e.g., CSV instead of
Excel)


   use URIs to denote things, so that people can point at your stuff

★
   link your data to other data to provide context


JSON is ★★★ data. JSON-LD, RDFa, and Microformats are  or ★ data.

We can link our data to other data with URIs in linked data formats.
JSON-LD is one representation of RDF. RDF* (read as "RDF star") extends RDF
for use with property graphs.
No one cares whether you believe that "semantic web failed" or "those
standards are useless": being able to share, discover, merge, join,
concatenate, analyze, and compare Datasets is of significant value to the
progress of the sciences and useful arts; so, I think that we should
support the linked data use case with at least:
1. a __json__(obj, spec=None) method and
2. a more-easily modifiable make_iterencode/iterencode implementation in
the json module of the standard library.


[1] https://github.com/digitalbazaar/pyld
[2] https://github.com/RDFLib/rdflib-jsonld
[3] https://5stardata.info/en/

*
Here's this that merges JSON-LD, SHACL, and JSON schema: It has 3 stars.
Validating JSON documents is indeed somewhat orthogonal to the __json__ /
iterencode implementation details we're discussing; but specifying types in
type annotations and then creating an additional complete data validation
specification is not DRY.
https://github.com/mulesoft-labs/json-ld-schema

*
re: generating JSON schema from type annotations

You can go from python:Str to jsonschema:format:string easily enough,
but, again, going from python:str to jsonschema:format:email will require
either extending the type annotation syntax or modifying a generated schema
stub and then changes to which will then need to be ported back to the type
annotations.

I suppose if all you're working with are data classes, that generating part
of the JSON schema from data class type annotations could be useful to you
in your quest to develop a new subset of JSON thats supports deserializing
(and validating) complex types in at least Python and JS (when there are
existing standards for doing so).


On Wed, Apr 8, 2020 at 3:08 PM Andrew Barnert  wrote:

> On Apr 8, 2020, at 01:18, Wes Turner  wrote:
> >
> > I don't see the value in using JSON to round-trip from Python to the
> same Python code.
> >
> > External schema is far more useful than embedding part of an ad-hoc
> nested object schema in type annotations that can't also do or even specify
> data validations.
>
> But dataclasses with type annotations can express a complete JSON Schema.
> Or, of course, in an ad hoc schema only published in human readable form,
> as most web APIs use today.
>
> > You can already jsonpickle data classes. If you want to share or just
> publish data, external schema using a web standard is your best bet.
>
> Sure, but you don’t need JSON-LD for that. Again, the fact that type
> annotations are insufficient to represent semantic triples is irrelevant.
> They are sufficient for the case of writing code to parse what YouTube
> gives you, or to provide a documented API that you design to be consumed by
> other people in JS, or to generate a JSON Schema from your legacy
> collection of classes that you can then publish, or to validate that a
> dataclass hierarchy matches a published schema, and so on. All of which are
> useful.
>
>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/D7WZX6AGC2OP3G6IYI36LACABQCWTS4J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improvement: __json__

[Andrew Barnert via Python-ideas]

On Apr 8, 2020, at 12:55, Wes Turner  wrote:
> 
> We should welcome efforts to support linked data in Python.

Fine, but that’s doesn’t meant we should derail every proposal that has 
anything to do with JSON by turning it into a proposal for linked data, or 
consider a useful proposal to be useless because it doesn’t give us linked data 
support on top of whatever it was intended to give us.

> No one cares whether you believe that "semantic web failed" or "those 
> standards are useless":

But no one is saying either of those things. People are saying that the 
semantic web is irrelevant to this discussion. JSON is useful, JSON Schema is 
useful, improving Python to make working with ad hoc or Schema-specified JSON 
is useful, and that’s all equally true whether the semantic web is the one true 
future or a total failure.

Would a __json__ method protocol improve Python? I don’t think so, but the 
reasons I don’t think so have nothing to do with LD. Other people do think so, 
and their reasons also have nothing to do with LD. And the same is true for 
most of the counter- and side-ideas that have come up in this thread and the 
last one.

> re: generating JSON schema from type annotations
> 
> You can go from python:Str to jsonschema:format:string easily enough,
> but, again, going from python:str to jsonschema:format:email will require 
> either extending the type annotation syntax or modifying a generated schema 
> stub and then changes to which will then need to be ported back to the type 
> annotations.

Sure, but who says you have to store email addresses as str?

@dataclass
class Person:
name: str
email: Email
address: Address

The fact that str is a builtin type, Address is a dataclass with a bunch of 
builtin-typed attributes, and Email is (say) a str subclass (or a dataclass 
with just a str member or whatever) that knows to render to and from 
type:string, format:email instead of just type:string doesn’t change the fact 
that they’re all perfectly good Python types and can all be used as type 
annotations and can all be mapped to a JSON Schema. How does it know that? Lots 
of things would work. Which one you use would be up to your JSON Schema-driven 
serialization library, or to your Python-code-from-JSON-Schema generator tool 
or your static Schema-from-code generator tool or.whatever. Maybe if we have 
multiple such libraries in wide use fighting it out, one of them will win. But 
even if there’s never a category killer, any of them will work fine for the 
applications that use it.

And of course often just using str for email addresses is fine. There’s a 
reason the Formats section of the JSON Schema spec is optional and explicitly 
calls out that many implementations don’t include it. And in some applications 
it would make more sense to break an email address down into two parts (user 
and host) and store a dict of those two values instead, and that’s also fine. 
Would it better serve the needs of the semantic web if the email format were a 
mandatory rather than optional part of the spec and everyone were required by 
law to always use it when storing email addresses? Maybe. But neither of those 
things are true.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QTDOOH4ASKWNUFYRZHUOSNQB62MAL5YK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improvement: __json__

[Wes Turner]

I think that we should support the linked data use case with at least:
1. a __json__(obj, spec=None) method and
2. a more-easily modifiable make_iterencode/iterencode implementation in
the json module of the standard library.

On Wed, Apr 8, 2020 at 5:00 PM Andrew Barnert  wrote:

> On Apr 8, 2020, at 12:55, Wes Turner  wrote:
> >
> > We should welcome efforts to support linked data in Python.
>
> Fine, but that’s doesn’t meant we should derail every proposal that has
> anything to do with JSON by turning it into a proposal for linked data, or
> consider a useful proposal to be useless because it doesn’t give us linked
> data support on top of whatever it was intended to give us.
>
> > No one cares whether you believe that "semantic web failed" or "those
> standards are useless":
>
> But no one is saying either of those things. People are saying that the
> semantic web is irrelevant to this discussion. JSON is useful, JSON Schema
> is useful, improving Python to make working with ad hoc or Schema-specified
> JSON is useful, and that’s all equally true whether the semantic web is the
> one true future or a total failure.
>
> Would a __json__ method protocol improve Python? I don’t think so, but the
> reasons I don’t think so have nothing to do with LD. Other people do think
> so, and their reasons also have nothing to do with LD. And the same is true
> for most of the counter- and side-ideas that have come up in this thread
> and the last one.
>
> > re: generating JSON schema from type annotations
> >
> > You can go from python:Str to jsonschema:format:string easily enough,
> > but, again, going from python:str to jsonschema:format:email will
> require either extending the type annotation syntax or modifying a
> generated schema stub and then changes to which will then need to be ported
> back to the type annotations.
>
> Sure, but who says you have to store email addresses as str?
>
> @dataclass
> class Person:
> name: str
> email: Email
> address: Address
>
> The fact that str is a builtin type, Address is a dataclass with a bunch
> of builtin-typed attributes, and Email is (say) a str subclass (or a
> dataclass with just a str member or whatever) that knows to render to and
> from type:string, format:email instead of just type:string doesn’t change
> the fact that they’re all perfectly good Python types and can all be used
> as type annotations and can all be mapped to a JSON Schema. How does it
> know that? Lots of things would work. Which one you use would be up to your
> JSON Schema-driven serialization library, or to your
> Python-code-from-JSON-Schema generator tool or your static Schema-from-code
> generator tool or.whatever. Maybe if we have multiple such libraries in
> wide use fighting it out, one of them will win. But even if there’s never a
> category killer, any of them will work fine for the applications that use
> it.
>
> And of course often just using str for email addresses is fine. There’s a
> reason the Formats section of the JSON Schema spec is optional and
> explicitly calls out that many implementations don’t include it. And in
> some applications it would make more sense to break an email address down
> into two parts (user and host) and store a dict of those two values
> instead, and that’s also fine. Would it better serve the needs of the
> semantic web if the email format were a mandatory rather than optional part
> of the spec and everyone were required by law to always use it when storing
> email addresses? Maybe. But neither of those things are true.
>
>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EDWP54EOMECBKUIDWZMU5Q4ADSI36Z27/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Optimize out unused variables

[Andrew Barnert via Python-ideas]

On Apr 8, 2020, at 10:59, Serhiy Storchaka  wrote:
> 
> I doubted whether or not to write, but since Guido has already touched a 
> similar topic (see "Live variable analysis -> earlier release"), so will I 
> write.
> 
> It is common to assign some values which are never used to variables because 
> the syntax demands this. For example:
> 
>head, _, rest = path.partition('/')
>first, second, *_ = line.split()
>for _ in range(10): ...
>[k for k, _ in pairs]
> 
> What if make such assignments no-op? It will only work with some limitations:
> 
> 1. The variable is local. Not global, not nonlocal, not cell.
> 2. The variable name must start with '_'. Otherwise there would be larger 
> risk of breaking a legal code which uses locals() with str.format() or like.
> 3. "del" is considered a use of the variable. So explicitly deleted variables 
> will not be optimized out.
> 4. Star-assignment still consumes the iterator. It just might not to keep all 
> values in memory.
> 
> STORE_FAST will be replaced with POP_TOP in these cases.

Could you go so far as to remove the variable from the locals if its only 
assignment(s) are optimized out? I’m not sure how much benefit that would 
provide. (Surely it would sometimes mean an f_locals array fits into one cache 
line instead of two, or that a whole code object stays around in L2 cache for 
the next time it’s called instead of being ejected, but often enough to make a 
difference? Maybe not…)

> I wrote some code few weeks ago, and it is not too complex. My doubts are 
> only that the benefit of the optimization with the above limitations is very 
> restricted.

Like Guido’s idea, this seems like something that should definitely be safe 
enough as an opt-in decorator or whatever, and implementable that way. And that 
also seems like the best way to answer those doubts. Write or find some code 
that you think should benefit, add the decorator, benchmark, and see.

Also, with an opt-in mechanism, you could relax the restrictions. For example, 
by default @killunused only kills unused assignments that meet your 
restrictions, but if I know it’s safe I can @killunused("_”, “dummy”) and it 
kills unused assignments to those names even if it wouldn’t normally do so. 
Then you could see if there are any cases where it’s useful, but only with the 
restrictions relaxed, and maybe use that as a guide to whether it’s worth 
finding a way to aim for looser restrictions in the first place or not.


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UVKV7VH3GTC3IU5L6W6F2GD3XVZRLHMO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improvement: __json__

[Wes Turner]

In trying to do this (again), I've realized that you *can't* just check for
hasattr(obj, '__json__') in a JSONEncoder.default method because default
only get's called for types it doesn't know how to serialize; so if you
e.g. subclass a dict, default() never gets called for the dict subclass.

https://docs.python.org/3/library/json.html :
> If specified, default should be a function that gets called for objects
that can’t otherwise be serialized.

https://github.com/python/cpython/blob/master/Lib/json/encoder.py

https://stackoverflow.com/questions/16405969/how-to-change-json-encoding-behaviour-for-serializable-python-object/17684652#17684652
specifies how to overload _make_iterencode *in pure Python*, but AFAICS
that NOPs use of the C-optimized json encoder.

The python json module originally came from simplejson.

Here's simplejson's for_json implementation in pure Python:
https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8ebcb82eee72/simplejson/encoder.py#L515
:

for_json = _for_json and getattr(value, 'for_json', None)
if for_json and callable(for_json):
chunks = _iterencode(for_json(), _current_indent_level)


And simplejson's for_json implementation in C:
https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8ebcb82eee72/simplejson/_speedups.c#L2839

Is there a strong reason that the method would need to be called __json__
instead of 'for_json'?

Passing a spec=None kwarg through to the __json__()/for_json() method would
not be compatible with simplejson's existing implementation.
Passing a spec=None kwarg would be necessary to support different JSON
standards within the same method; which I think is desirable because
JSON/JSON5/JSONLD/JSON_future.

On Wed, Apr 8, 2020 at 5:03 PM Wes Turner  wrote:

> I think that we should support the linked data use case with at least:
> 1. a __json__(obj, spec=None) method and
> 2. a more-easily modifiable make_iterencode/iterencode implementation in
> the json module of the standard library.
>
> On Wed, Apr 8, 2020 at 5:00 PM Andrew Barnert  wrote:
>
>> On Apr 8, 2020, at 12:55, Wes Turner  wrote:
>> >
>> > We should welcome efforts to support linked data in Python.
>>
>> Fine, but that’s doesn’t meant we should derail every proposal that has
>> anything to do with JSON by turning it into a proposal for linked data, or
>> consider a useful proposal to be useless because it doesn’t give us linked
>> data support on top of whatever it was intended to give us.
>>
>> > No one cares whether you believe that "semantic web failed" or "those
>> standards are useless":
>>
>> But no one is saying either of those things. People are saying that the
>> semantic web is irrelevant to this discussion. JSON is useful, JSON Schema
>> is useful, improving Python to make working with ad hoc or Schema-specified
>> JSON is useful, and that’s all equally true whether the semantic web is the
>> one true future or a total failure.
>>
>> Would a __json__ method protocol improve Python? I don’t think so, but
>> the reasons I don’t think so have nothing to do with LD. Other people do
>> think so, and their reasons also have nothing to do with LD. And the same
>> is true for most of the counter- and side-ideas that have come up in this
>> thread and the last one.
>>
>> > re: generating JSON schema from type annotations
>> >
>> > You can go from python:Str to jsonschema:format:string easily enough,
>> > but, again, going from python:str to jsonschema:format:email will
>> require either extending the type annotation syntax or modifying a
>> generated schema stub and then changes to which will then need to be ported
>> back to the type annotations.
>>
>> Sure, but who says you have to store email addresses as str?
>>
>> @dataclass
>> class Person:
>> name: str
>> email: Email
>> address: Address
>>
>> The fact that str is a builtin type, Address is a dataclass with a bunch
>> of builtin-typed attributes, and Email is (say) a str subclass (or a
>> dataclass with just a str member or whatever) that knows to render to and
>> from type:string, format:email instead of just type:string doesn’t change
>> the fact that they’re all perfectly good Python types and can all be used
>> as type annotations and can all be mapped to a JSON Schema. How does it
>> know that? Lots of things would work. Which one you use would be up to your
>> JSON Schema-driven serialization library, or to your
>> Python-code-from-JSON-Schema generator tool or your static Schema-from-code
>> generator tool or.whatever. Maybe if we have multiple such libraries in
>> wide use fighting it out, one of them will win. But even if there’s never a
>> category killer, any of them will work fine for the applications that use
>> it.
>>
>> And of course often just using str for email addresses is fine. There’s a
>> reason the Formats section of the JSON Schema spec is optional and
>> explicitly calls out that many implementations don’t include it. And

[Python-ideas] Re: Optimize out unused variables

[Henk-Jaap Wagenaar]

I like the idea of formalizing "unused variables".

How about having a syntax for it? Allowing a "." instead of an
identifier to signify this behaviour [reusing Serhiy's examples]:

head, ., rest = path.partition('/')
first, second, *. = line.split()
for . in range(10): ...
[k for k, . in pairs]

Potentially for unpacking one could use nothing, e.g.

first, second, * = line.split()

I don't think "." is necessarily a good symbol for it, but I think the
grammar would cope.

On Wed, 8 Apr 2020 at 22:56, Andrew Barnert via Python-ideas <
python-ideas@python.org> wrote:

> On Apr 8, 2020, at 10:59, Serhiy Storchaka  wrote:
> >
> > I doubted whether or not to write, but since Guido has already touched
> a similar topic (see "Live variable analysis -> earlier release"), so will
> I write.
> >
> > It is common to assign some values which are never used to variables
> because the syntax demands this. For example:
> >
> >head, _, rest = path.partition('/')
> >first, second, *_ = line.split()
> >for _ in range(10): ...
> >[k for k, _ in pairs]
> >
> > What if make such assignments no-op? It will only work with some
> limitations:
> >
> > 1. The variable is local. Not global, not nonlocal, not cell.
> > 2. The variable name must start with '_'. Otherwise there would be
> larger risk of breaking a legal code which uses locals() with str.format()
> or like.
> > 3. "del" is considered a use of the variable. So explicitly deleted
> variables will not be optimized out.
> > 4. Star-assignment still consumes the iterator. It just might not to
> keep all values in memory.
> >
> > STORE_FAST will be replaced with POP_TOP in these cases.
>
> Could you go so far as to remove the variable from the locals if its only
> assignment(s) are optimized out? I’m not sure how much benefit that would
> provide. (Surely it would sometimes mean an f_locals array fits into one
> cache line instead of two, or that a whole code object stays around in L2
> cache for the next time it’s called instead of being ejected, but often
> enough to make a difference? Maybe not…)
>
> > I wrote some code few weeks ago, and it is not too complex. My doubts
> are only that the benefit of the optimization with the above limitations is
> very restricted.
>
> Like Guido’s idea, this seems like something that should definitely be
> safe enough as an opt-in decorator or whatever, and implementable that way.
> And that also seems like the best way to answer those doubts. Write or find
> some code that you think should benefit, add the decorator, benchmark, and
> see.
>
> Also, with an opt-in mechanism, you could relax the restrictions. For
> example, by default @killunused only kills unused assignments that meet
> your restrictions, but if I know it’s safe I can @killunused("_”, “dummy”)
> and it kills unused assignments to those names even if it wouldn’t normally
> do so. Then you could see if there are any cases where it’s useful, but
> only with the restrictions relaxed, and maybe use that as a guide to
> whether it’s worth finding a way to aim for looser restrictions in the
> first place or not.
>
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/UVKV7VH3GTC3IU5L6W6F2GD3XVZRLHMO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/B52WJS53AFJHDRBRB55NG226BTTGMNSV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Caleb Donovick]

This would almost certainly break my code.   As a DSL developer I do a lot
of (exec | eval | introspection | ... ) shenanigans, that would make doing
liveness analysis undecidable.

On Wed, Apr 8, 2020 at 10:51 AM Andrew Barnert via Python-ideas <
python-ideas@python.org> wrote:

> On Apr 8, 2020, at 09:57, Guido van Rossum  wrote:
> >
> > 
> > Look at the following code.
> >
> > def foo(a, b):
> > x = a + b
> > if not x:
> > return None
> > sleep(1)  # A calculation that does not use x
> > return a*b
> >
> > This code DECREFs x when the frame is exited (at the return statement).
> But (assuming) we can clearly see that x is not needed during the sleep
> (representing a big calculation), we could insert a "del x" statement
> before the sleep.
> >
> > I think our compiler is smart enough to find out *some* cases where it
> could safely insert such del instructions.
>
> It depends on how much you’re willing to break and still call it “safely”.
>
> def sleep(n):
> global store
> store = inspect.current_frame().f_back.f_locals['x']
>
> This is a ridiculous example, but it shows that you can’t have all of
> Python’s dynamic functionality and still know when locals are dead. And
> there are less ridiculous examples with different code. If foo actually
> calls eval, exec, locals, vars, etc., or if it has a nested function that
> nonlocals x, etc., how can we spot that at compile time and keep x alive?
>
> Maybe that’s ok. After all, that code doesn’t work in a Python
> implementation that doesn’t have stack frame support. Some of the other
> possibilities might be more portable, but I don’t know without digging in
> further.
>
> Or maybe you can add new restrictions to what locals and eval and so on
> guarantee that will make it ok? Some code will break, but only rare
> “expert” code, where the authors will know how to work around it.
>
> Or, if not, it’s definitely fine as an opt-in optimization: decorate the
> function with @deadlocals and that decorator scans the bytecode and finds
> any locals that are dead assuming there’s no use of locals/eval/cells/etc.
> and, because you told it to assume that by opting in to the decorator, it
> can insert a DELETE_FAST safely.
>
> People already do similar things today—e.g., I’ve (only once in live code,
> but that’s still more than zero) used a @fastconst decorator that turns
> globals into consts on functions that I know are safe and are bottlenecks,
> and this would be no different. And of course you can add a recursive class
> decorator, or an import hook (or maybe even a command line flag or
> something) that enables it everywhere (maybe with a @nodeadlocals decorator
> for people who want it _almost_ everywhere but need to opt out one or two
> functions).
>
> Did Victor Stinner explore this as one of the optimizations for FAT
> Python/PEP 511/etc.? Maybe not, since it’s not something you can insert a
> guard, speculatively do, and then undo if the guard triggers, which was I
> think his key idea.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/OIGCRV464VJW3FRRBBK25XSNQYGWID7N/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DSIMIUM6QCUMC2GRTFV646KVWYIR45DR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Wes Turner]

Could something just heuristically add del statements with an AST
transformation that we could review with source control before committing?

When the gc pause occurs is something I don't fully understand. For example:

FWIW, this segfaults CPython in 2 lines:

import ctypes
ctypes.cast(1, ctypes.py_object)

Interestingly, this (tends to?) work; even when there are ah scope
closures?:

import ctypes, gc
x = 22
_id = id(x)
del x
gc.collect()
y = ctypes.cast(_id, ctypes.py_object).value
assert y == 22

Adding explicit calls to del with e.g. redbaron or similar would likely be
less surprising.
https://redbaron.readthedocs.io/en/latest/

Or something like a @jit decorator that pros would be aware of (who could
just add del statements to free as necessary)

On Wed, Apr 8, 2020, 10:29 PM Caleb Donovick 
wrote:

> This would almost certainly break my code.   As a DSL developer I do a lot
> of (exec | eval | introspection | ... ) shenanigans, that would make doing
> liveness analysis undecidable.
>
> On Wed, Apr 8, 2020 at 10:51 AM Andrew Barnert via Python-ideas <
> python-ideas@python.org> wrote:
>
>> On Apr 8, 2020, at 09:57, Guido van Rossum  wrote:
>> >
>> > 
>> > Look at the following code.
>> >
>> > def foo(a, b):
>> > x = a + b
>> > if not x:
>> > return None
>> > sleep(1)  # A calculation that does not use x
>> > return a*b
>> >
>> > This code DECREFs x when the frame is exited (at the return statement).
>> But (assuming) we can clearly see that x is not needed during the sleep
>> (representing a big calculation), we could insert a "del x" statement
>> before the sleep.
>> >
>> > I think our compiler is smart enough to find out *some* cases where it
>> could safely insert such del instructions.
>>
>> It depends on how much you’re willing to break and still call it “safely”.
>>
>> def sleep(n):
>> global store
>> store = inspect.current_frame().f_back.f_locals['x']
>>
>> This is a ridiculous example, but it shows that you can’t have all of
>> Python’s dynamic functionality and still know when locals are dead. And
>> there are less ridiculous examples with different code. If foo actually
>> calls eval, exec, locals, vars, etc., or if it has a nested function that
>> nonlocals x, etc., how can we spot that at compile time and keep x alive?
>>
>> Maybe that’s ok. After all, that code doesn’t work in a Python
>> implementation that doesn’t have stack frame support. Some of the other
>> possibilities might be more portable, but I don’t know without digging in
>> further.
>>
>> Or maybe you can add new restrictions to what locals and eval and so on
>> guarantee that will make it ok? Some code will break, but only rare
>> “expert” code, where the authors will know how to work around it.
>>
>> Or, if not, it’s definitely fine as an opt-in optimization: decorate the
>> function with @deadlocals and that decorator scans the bytecode and finds
>> any locals that are dead assuming there’s no use of locals/eval/cells/etc.
>> and, because you told it to assume that by opting in to the decorator, it
>> can insert a DELETE_FAST safely.
>>
>> People already do similar things today—e.g., I’ve (only once in live
>> code, but that’s still more than zero) used a @fastconst decorator that
>> turns globals into consts on functions that I know are safe and are
>> bottlenecks, and this would be no different. And of course you can add a
>> recursive class decorator, or an import hook (or maybe even a command line
>> flag or something) that enables it everywhere (maybe with a @nodeadlocals
>> decorator for people who want it _almost_ everywhere but need to opt out
>> one or two functions).
>>
>> Did Victor Stinner explore this as one of the optimizations for FAT
>> Python/PEP 511/etc.? Maybe not, since it’s not something you can insert a
>> guard, speculatively do, and then undo if the guard triggers, which was I
>> think his key idea.
>>
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/OIGCRV464VJW3FRRBBK25XSNQYGWID7N/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/DSIMIUM6QCUMC2GRTFV646KVWYIR45DR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.pytho