> On 13 May 2021, at 02:09, Mike Miller wrote:
>
>
>> On 2021-05-11 16:12, Guido van Rossum wrote:
>> On Tue, May 11, 2021 at 4:07 PM Gregory P. Smith > There's a difference between tracebacks dumped as plain text (utf-8) by
>>traceback.print_exc() appearing on stderr or directed into log
On 2021-05-11 16:12, Guido van Rossum wrote:
On Tue, May 11, 2021 at 4:07 PM Gregory P. Smith There's a difference between tracebacks dumped as plain text (utf-8) by
traceback.print_exc() appearing on stderr or directed into log files and
what can be displayed within a terminal. It
On Tue, May 11, 2021 at 4:07 PM Gregory P. Smith wrote:
>
>
> On Tue, May 11, 2021 at 3:33 PM Mike Miller
> wrote:
>
>>
>> On 5/11/21 1:57 AM, Baptiste Carvello wrote:
>> > Le 11/05/2021 à 09:35, Steven D'Aprano a écrit :
>> >> On Mon, May 10, 2021 at 09:44:05PM -0400, Terry Reedy wrote:
>> >>
>
On Tue, May 11, 2021 at 3:33 PM Mike Miller wrote:
>
> On 5/11/21 1:57 AM, Baptiste Carvello wrote:
> > Le 11/05/2021 à 09:35, Steven D'Aprano a écrit :
> >> On Mon, May 10, 2021 at 09:44:05PM -0400, Terry Reedy wrote:
> >>
> >>> The vanilla interpreter could be updated to recognize when it is
>
On 5/11/21 1:57 AM, Baptiste Carvello wrote:
Le 11/05/2021 à 09:35, Steven D'Aprano a écrit :
On Mon, May 10, 2021 at 09:44:05PM -0400, Terry Reedy wrote:
The vanilla interpreter could be updated to recognize when it is running
on a similated 35-year-old terminal that implements ansi-vt100 co
Le 11/05/2021 à 09:35, Steven D'Aprano a écrit :
> On Mon, May 10, 2021 at 09:44:05PM -0400, Terry Reedy wrote:
>
>> The vanilla interpreter could be updated to recognize when it is running
>> on a similated 35-year-old terminal that implements ansi-vt100 color
>> codes rather than a similated 4
On Mon, May 10, 2021 at 09:44:05PM -0400, Terry Reedy wrote:
> The vanilla interpreter could be updated to recognize when it is running
> on a similated 35-year-old terminal that implements ansi-vt100 color
> codes rather than a similated 40+-year-old black-and-white teletype-like
> terminal.
On 5/10/2021 6:07 AM, Steven D'Aprano wrote:
On Mon, May 10, 2021 at 05:34:12AM -0400, Terry Reedy wrote:
On 5/10/2021 3:28 AM, M.-A. Lemburg wrote:
I'm mostly thinking of tracebacks which go >10 levels deep, which is
rather common in larger applications. For those tracebacks, the top
entries
That is going to be very hard to read, unfortunately. Especially when the
line is
not simple. Highlighting the range is quite a fundamental part of the
proposal and
is driven by the great welcoming of highlighting ranges for syntax errors,
which many
users have reached to say that they find it extr
Another alternative is instead of
File blah.py line 3:
return x/0
^^^
to have
File blah.py line 3 cols 12-14:
x/0
On Mon, May 10, 2021 at 11:12 AM Steven D'Aprano
wrote:
> On Mon, May 10, 2021 at 05:34:12AM -0400, Terry Reedy wrote:
> > On 5/10/2021 3:28 AM, M
On Mon, May 10, 2021 at 05:34:12AM -0400, Terry Reedy wrote:
> On 5/10/2021 3:28 AM, M.-A. Lemburg wrote:
>
> >I'm mostly thinking of tracebacks which go >10 levels deep, which is
> >rather common in larger applications. For those tracebacks, the top
> >entries are mostly noise you never look at w
On 5/10/2021 3:28 AM, M.-A. Lemburg wrote:
I'm mostly thinking of tracebacks which go >10 levels deep, which is
rather common in larger applications. For those tracebacks, the top
entries are mostly noise you never look at when debugging. The proposal
now adds another 10 extra lines to jump over
On Mon, 10 May 2021 at 08:34, M.-A. Lemburg wrote:
> [...]
PS: It looks like the discussion has wondered off to Discourse
> now. Should we continue there ?
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
Pablo seems to want to redirect the discussion there yes, in particular to:
https://discuss.pyt
On 09.05.2021 14:22, Larry Hastings wrote:
> On 5/9/21 3:00 AM, M.-A. Lemburg wrote:
>> BTW: For better readability, I'd also not output the lines
>> for every stack level in the traceback, but just the last one,
>> since it's usually clear where the call to the next stack
>> level happens in
On Sun, May 9, 2021 at 9:13 AM Antoine Pitrou wrote:
> On Sun, 09 May 2021 02:16:02 -
> "Jim J. Jewett" wrote:
> > Antoine Pitrou wrote:
> > > On Sat, 8 May 2021 02:58:40 +
> > > Neil Schemenauer nas-pyt...@arctrix.com wrote:
> >
> > > > It would be cool if we could mmap the pyc files an
On Sat, 8 May 2021 21:59:49 +0100
Pablo Galindo Salgado wrote:
> > That could work, but in my personal opinion, I would prefer not to do
> that as it complicates things and I think is overkill.
>
> Let me expand on this:
>
> I recognize the problem that -OO can be quite unusable if some of you
On Sun, 09 May 2021 02:16:02 -
"Jim J. Jewett" wrote:
> Antoine Pitrou wrote:
> > On Sat, 8 May 2021 02:58:40 +
> > Neil Schemenauer nas-pyt...@arctrix.com wrote:
>
> > > It would be cool if we could mmap the pyc files and have the VM run
> > > code without an unmarshal step.
> > > What
On 5/9/21 3:00 AM, M.-A. Lemburg wrote:
> BTW: For better readability, I'd also not output the lines
> for every stack level in the traceback, but just the last one,
> since it's usually clear where the call to the next stack
> level happens in the upper ones.
Usually, sure -- but in the un
On 5/9/21 3:00 AM, M.-A. Lemburg wrote:
BTW: For better readability, I'd also not output the lines
for every stack level in the traceback, but just the last one,
since it's usually clear where the call to the next stack
level happens in the upper ones.
Playing devil's advocate: in the un-
On 08.05.2021 23:55, Gregory P. Smith wrote:
> Who non-hypothetically cares about a 22% pyc file size increase? I don't
> think
> we should be concerned. I'm in favor of always writing them and the 20% size
> increase that results in. If pyc size is an issue that should be its own
> separate en
On 5/8/21 10:16 PM, Jim J. Jewett wrote:
> Antoine Pitrou wrote:
>> On Sat, 8 May 2021 02:58:40 +
>> Neil Schemenauer nas-pyt...@arctrix.com wrote:
>>> It would be cool if we could mmap the pyc files and have the VM run
>>> code without an unmarshal step.
>>> What happens if another process mut
Antoine Pitrou wrote:
> On Sat, 8 May 2021 02:58:40 +
> Neil Schemenauer nas-pyt...@arctrix.com wrote:
> > It would be cool if we could mmap the pyc files and have the VM run
> > code without an unmarshal step.
> > What happens if another process mutates or truncates the file while the
> CPyth
Hi Chris,
On Fri, May 07, 2021 at 07:13:16PM -0700, Chris Jerdonek wrote:
> I'm not sure why you're sounding so negative. Pablo asked for ideas in his
> first message to the list:
I know that Pablo asked for ideas, but that doesn't mean that we are
obliged to agree with every idea. This is a di
On Sat, May 8, 2021 at 2:59 PM Gregory P. Smith wrote:
>
>
> On Sat, May 8, 2021 at 2:09 PM Pablo Galindo Salgado
> wrote:
>
>> > Why not put in it -O instead? Then -O means lose asserts and lose
>> fine-grained tracebacks, while -OO continues to also
>> strip out doc strings.
>>
>> What if som
Thanks Greg for the great, detailed response
I think I understand now better your proposal and I think is a good idea
and I would like to explore that. I have some questions:
* One problem I see is that that will make the constructor of the code
object depend on global options in the interpreter.
On Sat, May 8, 2021 at 2:40 PM Jonathan Goble wrote:
> On Sat, May 8, 2021 at 5:08 PM Pablo Galindo Salgado
> wrote:
>
>> > Why not put in it -O instead? Then -O means lose asserts and lose
>> fine-grained tracebacks, while -OO continues to also
>> strip out doc strings.
>>
>> What if someone w
On Sat, May 8, 2021 at 2:09 PM Pablo Galindo Salgado
wrote:
> > Why not put in it -O instead? Then -O means lose asserts and lose
> fine-grained tracebacks, while -OO continues to also
> strip out doc strings.
>
> What if someone wants to keep asserts but do not want the extra data?
>
exactly m
On Sat, May 8, 2021 at 5:08 PM Pablo Galindo Salgado
wrote:
> > Why not put in it -O instead? Then -O means lose asserts and lose
> fine-grained tracebacks, while -OO continues to also
> strip out doc strings.
>
> What if someone wants to keep asserts but do not want the extra data?
>
What if I
> I don't think the optional existence of column number information needs a
different kind of pyc file. Just a flag in a pyc file's header at most.
It isn't a new type of file.
Greg, what do you think if instead of not writing it to the pyc file with
-OO or adding a header entry to decide to read
> Why not put in it -O instead? Then -O means lose asserts and lose
fine-grained tracebacks, while -OO continues to also
strip out doc strings.
What if someone wants to keep asserts but do not want the extra data?
On Sat, 8 May 2021 at 22:05, Ethan Furman wrote:
> On 5/8/21 1:31 PM, Pablo Gali
On 5/8/21 1:31 PM, Pablo Galindo Salgado wrote:
>> We can't piggy back on -OO as the only way to disable this, it needs to
>> have an option of its own. -OO is unusable as code that relies on "doc"
>> strings as application data such as http://www.dabeaz.com/ply/ply.html
>> exists.
>
> -OO is the
> That could work, but in my personal opinion, I would prefer not to do
that as it complicates things and I think is overkill.
Let me expand on this:
I recognize the problem that -OO can be quite unusable if some of your
dependencies depend on docstrings and that It would be good to separate
this
> I don't think the optional existence of column number information needs a
different kind of pyc file. Just a flag in a pyc file's header at most.
It isn't a new type of file.
That could work, but in my personal opinion, I would prefer not to do that
as it complicates things and I think is overk
On Sat, May 8, 2021 at 1:32 PM Pablo Galindo Salgado
wrote:
> > We can't piggy back on -OO as the only way to disable this, it needs to
> have an option of its own. -OO is unusable as code that relies on
> "doc"strings as application data such as
> http://www.dabeaz.com/ply/ply.html exists.
>
>
> We can't piggy back on -OO as the only way to disable this, it needs to
have an option of its own. -OO is unusable as code that relies on
"doc"strings as application data such as http://www.dabeaz.com/ply/ply.html
exists.
-OO is the only sensible way to disable the data. There are two things to
On Sat, May 8, 2021 at 11:58 AM Pablo Galindo Salgado
wrote:
> Hi Brett,
>
> Just to be clear, .pyo files have not existed for a while:
>> https://www.python.org/dev/peps/pep-0488/.
>
>
> Whoops, my bad, I wanted to refer to the pyc files that are generated
> with -OO, which have the "opt-2" pref
Hi Brett,
Just to be clear, .pyo files have not existed for a while:
> https://www.python.org/dev/peps/pep-0488/.
Whoops, my bad, I wanted to refer to the pyc files that are generated
with -OO, which have the "opt-2" prefix.
This only kicks in at the -OO level.
I will correct the PEP so it re
On Fri, May 7, 2021 at 7:31 PM Pablo Galindo Salgado
wrote:
> Although we were originally not sympathetic with it, we may need to offer
> an opt-out mechanism for those users that care about the impact of the
> overhead of the new data in pyc files
> and in in-memory code objectsas was suggested
El sáb, 8 may 2021 a las 10:00, Devin Jeanpierre ()
escribió:
> > What are people thoughts on the feature?
>
> I'm +1, this level of detail in the bytecode is very useful. My main
> interest is actually from the AST though. :) In order to be in the
> bytecode, one assumes it must first be in the A
> What are people thoughts on the feature?
I'm +1, this level of detail in the bytecode is very useful. My main
interest is actually from the AST though. :) In order to be in the
bytecode, one assumes it must first be in the AST. That information is
incredibly useful for refactoring tools like htt
I really like this idea Nathaniel.
We already have a section considering lazy-loading column information in the
draft PEP but obviously it has a pretty high implementation complexity since it
necessitates a change in the pyc format and the importlib machinery.
Long term this might be the way to g
You can certainly get fancy and apply delta encoding + entropy
compression, such as done in Parquet, a high-performance data storage
format:
https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5
(the linked paper from Lemire and Boytsov gives a l
On Fri, 7 May 2021 23:20:55 +0100
Irit Katriel via Python-Dev wrote:
> On Fri, May 7, 2021 at 10:52 PM Pablo Galindo Salgado
> wrote:
>
> >
> > The cost of this is having the start column number and end column number
> > information for every bytecode instruction
> >
>
>
> Is it really every
On Sat, 8 May 2021 02:58:40 +
Neil Schemenauer wrote:
> On 2021-05-07, Pablo Galindo Salgado wrote:
> > Technically the main concern may be the size of the unmarshalled
> > pyc files in memory, more than the storage size of disk.
>
> It would be cool if we could mmap the pyc files and have
On Fri, May 7, 2021 at 8:14 PM Neil Schemenauer wrote:
>
> On 2021-05-07, Pablo Galindo Salgado wrote:
> > Technically the main concern may be the size of the unmarshalled
> > pyc files in memory, more than the storage size of disk.
>
> It would be cool if we could mmap the pyc files and have the
On 2021-05-07, Pablo Galindo Salgado wrote:
> Technically the main concern may be the size of the unmarshalled
> pyc files in memory, more than the storage size of disk.
It would be cool if we could mmap the pyc files and have the VM run
code without an unmarshal step. One idea is something simil
Although we were originally not sympathetic with it, we may need to offer
an opt-out mechanism for those users that care about the impact of the
overhead of the new data in pyc files
and in in-memory code objectsas was suggested by some folks (Thomas, Yury,
and others). For this, we could propose t
On Fri, May 7, 2021 at 6:39 PM Steven D'Aprano wrote:
> On Fri, May 07, 2021 at 06:02:51PM -0700, Chris Jerdonek wrote:
>
> > To know what compression methods might be effective, I’m wondering if it
> > could be useful to see separate histograms of, say, the start column
> number
> > and width ov
> I'm wondering if it's possible to compromise with one position that's
not as complete but still gives a good hint:
Even if is possible, it will be quite less useful (a lot of users wanted to
highlight full ranges for syntax errors, and that change was very well
received
when we announce it in 3.
On 2021-05-08 01:43, Pablo Galindo Salgado wrote:
Some update on the numbers. We have made some draft implementation to
corroborate the
numbers with some more realistic tests and seems that our original
calculations were wrong.
The actual increase in size is quite bigger than previously adverti
On Fri, May 07, 2021 at 06:02:51PM -0700, Chris Jerdonek wrote:
> To know what compression methods might be effective, I’m wondering if it
> could be useful to see separate histograms of, say, the start column number
> and width over the code base. Or for people that really want to dig in,
> maybe
On Fri, May 7, 2021 at 5:44 PM Pablo Galindo Salgado
wrote:
> Some update on the numbers. We have made some draft implementation to
> corroborate the
> numbers with some more realistic tests and seems that our original
> calculations were wrong.
> The actual increase in size is quite bigger than
One last note for clarity: that's the increase of size in the stdlib, the
increase of size
for pyc files goes from 28.471296MB to 34.750464MB, which is an increase of
22%.
On Sat, 8 May 2021 at 01:43, Pablo Galindo Salgado
wrote:
> Some update on the numbers. We have made some draft implementati
Some update on the numbers. We have made some draft implementation to
corroborate the
numbers with some more realistic tests and seems that our original
calculations were wrong.
The actual increase in size is quite bigger than previously advertised:
Using bytes object to encode the final object an
On Sat, 8 May 2021, 8:53 am Pablo Galindo Salgado,
wrote:
> > One thought: could the stored column position not include the
> indentation? Would that help?
>
> The compiler doesn't have access easy access to the source unfortunately
> so we don't know how much is the indentation. This can make li
> One thought: could the stored column position not include the
indentation? Would that help?
The compiler doesn't have access easy access to the source unfortunately so
we don't know how much is the indentation. This can make life
a bit harder for other tools, although it can make it easier for r
Thanks, Irit for your comment!
> Is it really every instruction? Or only those that can raise exceptions?
Technically only the ones that can raise exceptions, but the majority can
and optimizing this to only restrict to the set that can raise exceptions
has
the danger than the mapping needs to be
> haha, true... Does our parser even have a maximum line length? (I'm not
suggesting being unlimited or matching that if huge, 64k is already
ridiculous
We use py_ssize_t in some places but at the end of the day the lines and
columns have a limit of INT_MAX IIRC
On Fri, 7 May 2021 at 23:35, Greg
This is actually a very good point. The only disadvantage is that it
complicates the parsing a bit and we loose the possibility of indexing
the table by instruction offset.
On Fri, 7 May 2021 at 23:01, Larry Hastings wrote:
> On 5/7/21 2:45 PM, Pablo Galindo Salgado wrote:
>
> Given that column
On Fri, May 7, 2021 at 3:24 PM Pablo Galindo Salgado
wrote:
> Thanks a lot Gregory for the comments!
>
> An additional cost to this is things that parse text tracebacks not
>> knowing how to handle it and things that log tracebacks
>> generating additional output.
>
> We should provide a way for
On 2021-05-07 22:56, Larry Hastings wrote:
On 5/7/21 2:45 PM, Pablo Galindo Salgado wrote:
Given that column numbers are not very big compared with line numbers,
we plan to store these as unsigned chars
or unsigned shorts. We ran some experiments over the standard library
and we found that the
On 2021-05-07 22:45, Pablo Galindo Salgado wrote:
Hi there,
We are preparing a PEP and we would like to start some early discussion
about one of the main aspects of the PEP.
The work we are preparing is to allow the interpreter to produce more
fine-grained error messages, pointing to
the so
Thanks a lot Gregory for the comments!
An additional cost to this is things that parse text tracebacks not knowing
> how to handle it and things that log tracebacks generating additional
> output.
We should provide a way for people to disable the feature on a process as
> part of this while they
On Fri, May 7, 2021 at 2:50 PM Pablo Galindo Salgado
wrote:
> Hi there,
>
> We are preparing a PEP and we would like to start some early discussion
> about one of the main aspects of the PEP.
>
> The work we are preparing is to allow the interpreter to produce more
> fine-grained error messages,
On Fri, May 7, 2021 at 10:52 PM Pablo Galindo Salgado
wrote:
>
> The cost of this is having the start column number and end column number
> information for every bytecode instruction
>
Is it really every instruction? Or only those that can raise exceptions?
_
Technically the main concern may be the size of the unmarshalled pyc files
in memory, more than the storage size of disk.
On Fri, 7 May 2021, 23:04 Antoine Pitrou, wrote:
> On Fri, 7 May 2021 22:45:38 +0100
> Pablo Galindo Salgado wrote:
> >
> > The cost of this is having the start column numbe
On Fri, May 7, 2021 at 3:01 PM Larry Hastings wrote:
> On 5/7/21 2:45 PM, Pablo Galindo Salgado wrote:
>
> Given that column numbers are not very big compared with line numbers, we
> plan to store these as unsigned chars
> or unsigned shorts. We ran some experiments over the standard library and
On Fri, 7 May 2021 22:45:38 +0100
Pablo Galindo Salgado wrote:
>
> The cost of this is having the start column number and end column number
> information for every bytecode instruction
> and this is what we want to discuss (there is also some stack cost to
> re-raise exceptions but that's not a b
On 5/7/21 2:45 PM, Pablo Galindo Salgado wrote:
Given that column numbers are not very big compared with line numbers,
we plan to store these as unsigned chars
or unsigned shorts. We ran some experiments over the standard library
and we found that the overhead of all pyc files is:
* If we use
69 matches
Mail list logo
