Re: getting rid of the recursion in __getattribute__

[Peter Otten]

On 24/05/2023 15:37, A KR wrote:

It is perfectly explained in the standards here [1] saying that:


In order to avoid infinite recursion in this method, its implementation should 
always call the base class method with the same name to access any attributes 
it needs, for example, object.__getattribute__(self, name).


Therefore, I wrote a code following what the standard says:


class Sample():
 def __init__(self):
 self.a = -10

 def __getattribute__(self, name):
 if name == 'a':
 return object.__getattribute__(self, name)

 raise AttributeError()

s = Sample()
result = s.a
print(result)

I did not fall into recursion, and the output was
-10


While this works it's not how I understand the recommended pattern. I'd
rather treat "special" attributes first and then use the
__getattribute__ method of the base class as a fallback:

>> class Demo:
def __getattribute__(self, name):
if name == "answer":
return 42
return super().__getattribute__(name)

That way your special arguments,

>>> d = Demo()
>>> d.answer
42


missing arguments

>>> d.whatever
Traceback (most recent call last):
  File "", line 1, in 
d.whatever
  File "", line 5, in __getattribute__
return super().__getattribute__(name)
AttributeError: 'Demo' object has no attribute 'whatever'

and "normal" arguments are treated as expected

>>> d.question = "What's up?"
>>> d.question
"What's up?"

Eventual "special" arguments in the superclass would also remain accessible.



However, when I try the code without deriving from a class:

class AnyClassNoRelation:
 pass

class Sample():
 def __init__(self):
 self.a = -10

 def __getattribute__(self, name):
 if name == 'a':
 return AnyClassNoRelation.__getattribute__(self, name)

 raise AttributeError()

s = Sample()

result = s.a
print(result)
and calling __getattribute__ via any class (in this example class 
AnyClassNoRelation) instead of object.__getattribute__(self, name) as the 
standard says call using the base class, I get the same output: no recursion 
and -10.

So my question:

How come this is possible (having the same output without using the base 
class's __getattribute__? Although the standards clearly states that 
__getattribute__ should be called from the base class.



AnyClassNoRelation does not override __getattribute__, so

>>> AnyClassNoRelation.__getattribute__ is object.__getattribute__
True


There is no sanity check whether a method that you call explicitly is
actually in an object's inheritance tree,

>>> class NoRelation:
def __getattribute__(self, name):
return name.upper()


>>> class Demo:
def __getattribute__(self, name):
return "<{}>".format(NoRelation.__getattribute__(self, name))


>>> Demo().some_arg
''

but the only purpose I can imagine of actually calling "someone else's"
method is to confuse the reader...


In order to avoid infinite recursion in this method, its implementation should 
always call the base class method with the same name to access any attributes 
it needs, for example, object.__getattribute__(self, name).


Literally, I can call __getattribute__ with anyclass (except Sample cause it 
will be infinite recursion) I define and it works just fine. Could you explain 
me why that happens?



--
https://mail.python.org/mailman/listinfo/python-list

getting rid of the recursion in __getattribute__

[A KR]

It is perfectly explained in the standards here [1] saying that: 


In order to avoid infinite recursion in this method, its implementation should 
always call the base class method with the same name to access any attributes 
it needs, for example, object.__getattribute__(self, name).


Therefore, I wrote a code following what the standard says:


class Sample():
def __init__(self):
self.a = -10

def __getattribute__(self, name):
if name == 'a':
return object.__getattribute__(self, name)

raise AttributeError()

s = Sample()
result = s.a
print(result)

I did not fall into recursion, and the output was
-10

I used here object.__getattribute__(self, name) cause the base class of Sample 
is object.

If I derive the Sample class from another class such as A, I should change 
object.__getattribute__(self, name) to A.__getattribute__(self, name) as the 
base class of class Sample is class A.


class A:
pass

class Sample(A):
def __init__(self):
self.a = -10

def __getattribute__(self, name):
if name == 'a':
return A.__getattribute__(self, name)

raise AttributeError()

s = Sample()

result = s.a
print(result)

which gives the same output as expected. No recursion and -10.

However, when I try the code without deriving from a class:

class AnyClassNoRelation:
pass

class Sample():
def __init__(self):
self.a = -10

def __getattribute__(self, name):
if name == 'a':
return AnyClassNoRelation.__getattribute__(self, name)

raise AttributeError()

s = Sample()

result = s.a
print(result)
and calling __getattribute__ via any class (in this example class 
AnyClassNoRelation) instead of object.__getattribute__(self, name) as the 
standard says call using the base class, I get the same output: no recursion 
and -10.

So my question:

How come this is possible (having the same output without using the base 
class's __getattribute__? Although the standards clearly states that 
__getattribute__ should be called from the base class.


In order to avoid infinite recursion in this method, its implementation should 
always call the base class method with the same name to access any attributes 
it needs, for example, object.__getattribute__(self, name). 


Literally, I can call __getattribute__ with anyclass (except Sample cause it 
will be infinite recursion) I define and it works just fine. Could you explain 
me why that happens?


[1] https://docs.python.org/3/reference/datamodel.html#object.__getattribute__
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: timedelta object recursion bug

[Dieter Maurer]

Please stay on the list (such that others can help, too)

Ben Hirsig wrote at 2022-7-29 06:53 +1000:
>Thanks for the replies, I'm just trying to understand why this would be
>useful?
>
>E.g. why does max need a min/max/resolution, and why would these attributes
>themselves need a min/max/resolution, etc, etc?

`max` is a `timedelta` and as such inherits (e.g.) `resolution`
from the class (as any other `timedelta` instance).

Note that `timedelta` instances do not have a `max` (`min|resolution`)
slot. When `max` is looked up, it is first searched in the instance
(and not found), then in the class where it is found:
all `max` accesses result in the same object.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: timedelta object recursion bug

[Dieter Maurer]

Ben Hirsig wrote at 2022-7-28 19:54 +1000:
>Hi, I noticed this when using the requests library in the response.elapsed
>object (type timedelta). Tested using the standard datetime library alone
>with the example displayed on
>https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta
>
>
>
>It appears as though the timedelta object recursively adds its own
>attributes (min, max, resolution) as further timedelta objects. I’m not
>sure how deep they go, but presumably hitting the recursion limit.

If you look at the source, you will see that `min`, `max`, `resolution`
are class level attributes. Their values are `timedelta` instances.
Therefore, you can access e.g. `timedelta(days=365).min.max.resolution`.
But this is nothing to worry about.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: timedelta object recursion bug

[Jon Ribbens via Python-list]

On 2022-07-28, Ben Hirsig  wrote:
> Hi, I noticed this when using the requests library in the response.elapsed
> object (type timedelta). Tested using the standard datetime library alone
> with the example displayed on
> https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta
>
> It appears as though the timedelta object recursively adds its own
> attributes (min, max, resolution) as further timedelta objects. I’m not
> sure how deep they go, but presumably hitting the recursion limit.
>
>>from datetime import timedelta
>>year = timedelta(days=365)
>>print(year.max)
>   9 days, 23:59:59.99
>>print(year.max.min.max.resolution.max.min)
>   -9 days, 0:00:00

Why do you think this is a bug?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: timedelta object recursion bug

[MRAB]

On 28/07/2022 10:54, Ben Hirsig wrote:

Hi, I noticed this when using the requests library in the response.elapsed
object (type timedelta). Tested using the standard datetime library alone
with the example displayed on
https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta



It appears as though the timedelta object recursively adds its own
attributes (min, max, resolution) as further timedelta objects. I’m not
sure how deep they go, but presumably hitting the recursion limit.




from datetime import timedelta



year = timedelta(days=365)



print(year.max)


   9 days, 23:59:59.99


print(year.max.min.max.resolution.max.min)


   -9 days, 0:00:00



I’m using 3.10.3


It's not recursion, it's a reference cycle. In fact, more than one:

>>> from datetime import timedelta
>>> year = timedelta(days=365)
>>> type(year)

>>> type(year.max)

>>> year.max is year.max.max
True
>>> type(year.min)

>>> year.min is year.min.min
True
--
https://mail.python.org/mailman/listinfo/python-list

Fwd: timedelta object recursion bug

[Ben Hirsig]

Hi, I noticed this when using the requests library in the response.elapsed
object (type timedelta). Tested using the standard datetime library alone
with the example displayed on
https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta



It appears as though the timedelta object recursively adds its own
attributes (min, max, resolution) as further timedelta objects. I’m not
sure how deep they go, but presumably hitting the recursion limit.



>from datetime import timedelta

>year = timedelta(days=365)

>print(year.max)

  9 days, 23:59:59.99

>print(year.max.min.max.resolution.max.min)

  -9 days, 0:00:00



I’m using 3.10.3



Cheers
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why this code giving recursion error????

[dn]

On 26/06/2022 23.00, נתי שטרן wrote:
> I FIXED THE CODE 

For the benefit of future-readers: how did you go about fixing it? What
was wrong?

-- 
Regards,
=dn
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why this code giving recursion error????

[נתי שטרן]

I FIXED THE CODE

‫בתאריך יום א׳, 26 ביוני 2022 ב-13:57 מאת ‪dn‬‏ <‪
pythonl...@danceswithmice.info‬‏>:‬

> On 26/06/2022 22.48, נתי שטרן wrote:
> > def compile(p, flags=0):
> > # internal: convert pattern list to internal format
> >
> > if (isinstance(p,str)):
> > pattern = p
> > p = sre_parse.parse(p, flags)
> > else:
> > pattern = None
> >
> > code = _code(p, flags)
> >
> > if flags & SRE_FLAG_DEBUG:
> > print()
> > dis(code)
> >
> > # map in either direction
> > groupindex = p.state.groupdict
> > indexgroup = [None] * p.state.groups
> > for k, i in groupindex.items():
> > indexgroup[i] = k
> >
> > return sre_compile.compile(
> > pattern, flags | p.state.flags, code,
> > p.state.groups-1,
> > groupindex, tuple(indexgroup)
> > )
>
>
> Why would any code give a recursion error?
>
> With recursion problems, the first question to ask is: how does this
> thing stop?
>
> Have you built some test-data with only a handful of items, and thus a
> predictable result. Does that result occur? If not, where are the
> differences occurring?
> --
> Regards,
> =dn
> --
> https://mail.python.org/mailman/listinfo/python-list
>


-- 
<https://netanel.ml>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why this code giving recursion error????

[נתי שטרן]

and i ask also what's the problem with this function:


def _code(p, flags):

flags = p.state.flags | flags
code = []

# compile info block
sre_compile._compile_info(code, p, flags)

# compile the pattern
sre_compile._compile(code, p.data, flags)

code.append(SUCCESS)

return code


‫בתאריך יום א׳, 26 ביוני 2022 ב-13:48 מאת נתי שטרן <‪nsh...@gmail.com‬‏>:‬

> def compile(p, flags=0):
> # internal: convert pattern list to internal format
>
> if (isinstance(p,str)):
> pattern = p
> p = sre_parse.parse(p, flags)
> else:
> pattern = None
>
> code = _code(p, flags)
>
> if flags & SRE_FLAG_DEBUG:
> print()
> dis(code)
>
> # map in either direction
> groupindex = p.state.groupdict
> indexgroup = [None] * p.state.groups
> for k, i in groupindex.items():
> indexgroup[i] = k
>
> return sre_compile.compile(
> pattern, flags | p.state.flags, code,
> p.state.groups-1,
> groupindex, tuple(indexgroup)
> )
>
>
> --
> 
>


-- 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why this code giving recursion error????

[dn]

On 26/06/2022 22.48, נתי שטרן wrote:
> def compile(p, flags=0):
> # internal: convert pattern list to internal format
> 
> if (isinstance(p,str)):
> pattern = p
> p = sre_parse.parse(p, flags)
> else:
> pattern = None
> 
> code = _code(p, flags)
> 
> if flags & SRE_FLAG_DEBUG:
> print()
> dis(code)
> 
> # map in either direction
> groupindex = p.state.groupdict
> indexgroup = [None] * p.state.groups
> for k, i in groupindex.items():
> indexgroup[i] = k
> 
> return sre_compile.compile(
> pattern, flags | p.state.flags, code,
> p.state.groups-1,
> groupindex, tuple(indexgroup)
> )


Why would any code give a recursion error?

With recursion problems, the first question to ask is: how does this
thing stop?

Have you built some test-data with only a handful of items, and thus a
predictable result. Does that result occur? If not, where are the
differences occurring?
-- 
Regards,
=dn
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why this code giving recursion error????

[נתי שטרן]

stack trace:



 File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 5387, in 
class bottle:
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 5694, in bottle
class Router(object):
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 5736, in Router
rule_syntax = re.compile('(*)'
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 375, in compile
return sre_compile.compile(pattern, flags)
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 2752, in compile
return sre_compile.compile(
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 2752, in compile
return sre_compile.compile(
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 2752, in compile
return sre_compile.compile(
  [Previous line repeated 1005 more times]
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 2736, in compile
p = sre_parse.parse(p, flags)
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 1822, in parse
p = sre_parse._parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 1234, in _parse_sub
itemsappend(sre_parse._parse(source, state, verbose, nested + 1,
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 1708, in _parse
p = sre_parse._parse_sub(source, state, sub_verbose, nested + 1)
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 1234, in _parse_sub
itemsappend(sre_parse._parse(source, state, verbose, nested + 1,
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 1708, in _parse
p = sre_parse._parse_sub(source, state, sub_verbose, nested + 1)
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 1234, in _parse_sub
itemsappend(sre_parse._parse(source, state, verbose, nested + 1,
  File "\\Weank-fs\users$\NetanelST\ORACLE\RTR.py", line 1707, in _parse
not (del_flags & SRE_FLAG_VERBOSE))
  File "C:\Program
Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\enum.py",
line 987, in __and__
if not isinstance(other, (self.__class__, int)):

‫בתאריך יום א׳, 26 ביוני 2022 ב-13:48 מאת נתי שטרן <‪nsh...@gmail.com‬‏>:‬

> def compile(p, flags=0):
> # internal: convert pattern list to internal format
>
> if (isinstance(p,str)):
> pattern = p
> p = sre_parse.parse(p, flags)
> else:
> pattern = None
>
> code = _code(p, flags)
>
> if flags & SRE_FLAG_DEBUG:
> print()
> dis(code)
>
> # map in either direction
> groupindex = p.state.groupdict
> indexgroup = [None] * p.state.groups
> for k, i in groupindex.items():
> indexgroup[i] = k
>
> return sre_compile.compile(
> pattern, flags | p.state.flags, code,
> p.state.groups-1,
> groupindex, tuple(indexgroup)
> )
>
>
> --
> 
>


-- 

-- 
https://mail.python.org/mailman/listinfo/python-list

why this code giving recursion error????

[נתי שטרן]

def compile(p, flags=0):
# internal: convert pattern list to internal format

if (isinstance(p,str)):
pattern = p
p = sre_parse.parse(p, flags)
else:
pattern = None

code = _code(p, flags)

if flags & SRE_FLAG_DEBUG:
print()
dis(code)

# map in either direction
groupindex = p.state.groupdict
indexgroup = [None] * p.state.groups
for k, i in groupindex.items():
indexgroup[i] = k

return sre_compile.compile(
pattern, flags | p.state.flags, code,
p.state.groups-1,
groupindex, tuple(indexgroup)
)


-- 

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue403757] catch server recursion in urrlib

[admin]

Change by admin :


--
github: None -> 33914

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401612] Prevent recursion limit when using ".*?xxx"

[admin]

Change by admin :


--
github: None -> 33182

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue493252] maximum recursion limit exceeded in match

[admin]

Change by admin :


--
github: None -> 35740

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue483469] crash on unbounded recursion in __del__ .

[admin]

Change by admin :


--
github: None -> 35549

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue437472] MacPy21: sre "recursion limit" bug

[admin]

Change by admin :


--
github: None -> 34685

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue437475] MacPy21: sre "recursion limit" bug

[admin]

Change by admin :


--
github: None -> 34686

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue432384] Recursion in PyString_AsEncodedString?

[admin]

Change by admin :


--
github: None -> 34614

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue418626] maximum recursion limit exceeded (2.1)

[admin]

Change by admin :


--
github: None -> 34410

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue216136] sre recursion limit hit in tokenize.py

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue215903] sre: split exceeds recursion limit

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401612] Prevent recursion limit when using ".*?xxx"

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue212943] infinite recursion bug

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210615] another unbounded recursion bug

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue215903] sre: split exceeds recursion limit

[admin]

Change by admin :


--
github: None -> 33258

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue216136] sre recursion limit hit in tokenize.py

[admin]

Change by admin :


--
github: None -> 33284

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue212943] infinite recursion bug

[admin]

Change by admin :


--
github: None -> 32997

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210615] another unbounded recursion bug

[admin]

Change by admin :


--
github: None -> 32686

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47073] Solution for recursion error when comparing dataclass objects

[Eric V. Smith]

Change by Eric V. Smith :


--
assignee:  -> eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47073] Solution for recursion error when comparing dataclass objects

[Alex Waygood]

Change by Alex Waygood :


--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47073] Solution for recursion error when comparing dataclass objects

[Yonatan Dankner]

New submission from Yonatan Dankner :

TL;DR: A possible method to avoid infinite recursion when comparing dataclasses 
with circular references, using a decorator.

>>> @dataclass
... class A:
... x: int = 1
... self_reference: Any = field(init=False)
...
... def __post_init__(self):
... self.self_reference = self
...
>>> A(x=1) == A(x=1)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in __eq__
  File "", line 3, in __eq__
  File "", line 3, in __eq__
  [Previous line repeated 330 more times]
RecursionError: maximum recursion depth exceeded in comparison

It should in my opinion return True.

To avoid a recursion error, I would like to suggest a solution using a wrapper, 
similar to the one implemented for repr (see dataclasses._recursive_repr):
If within the process of comparing two objects, we have encountered an attempt 
to compare these objects (meaning we have reached a loop), we should return 
true, since no inequalities will be discovered if we continue recursing.

I suggest an addition to dataclasses.py which would look something like this 
(drew inspiration from _recursive_repr):

def _recursive_eq(user_function):
# Decorator to make a eq function return True when a loop was detected.
eq_running = set()

@functools.wraps(user_function)
def wrapper(self, other):
key = id(self), id(other), _thread.get_ident()
if key in eq_running:
return True
eq_running.add(key)
   
try:
result = user_function(self, other)
finally:
eq_running.discard(key)
return result
return wrapper


# And then replacing the _cmp_fn with the following.

def _cmp_fn(name, op, self_tuple, other_tuple):
return _recursive_eq(_create_fn(name,
  ('self', 'other'),
  [ 'if other.__class__ is self.__class__:',
   f' return {self_tuple}{op}{other_tuple}',
'return NotImplemented']))

I would like to hear your thoughts!

--
components: Library (Lib)
messages: 415641
nosy: dankner
priority: normal
severity: normal
status: open
title: Solution for recursion error when comparing dataclass objects
type: behavior
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue47073>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26545] [doc] os.walk is limited by python's recursion limit

[Irit Katriel]

Irit Katriel  added the comment:

I agree with Stanley. The documentation for os is clear that recursion is used 
and the documentation for RecursionError links to getrecursionlimit(). This 
seems sufficient.

--
nosy: +iritkatriel
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue26545>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46938] dataclass __post_init__ recursion

[Eric V. Smith]

Eric V. Smith  added the comment:

Yeah, I've come to the conclusion that it's not so simple, either. I'm also 
thinking that advising to call the base __init__ is a mistake.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46938] dataclass __post_init__ recursion

[Bar Harel]

Bar Harel  added the comment:

Actually I'm not sure if the fix is so simple. What happens if B does not 
inherit from dataclass, but still inherits from A? Do we want to use the new 
__post_init__? I would assume we do, meaning we don't necessarily want to 
attach the __post_init__ to the dataclass, but to subclasses.

If so, the suggestion to call __init__() from __post_init__() might be the 
problematic part, and is what conceptually breaks the inheritance linearization.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46938] dataclass __post_init__ recursion

[Bar Harel]

Bar Harel  added the comment:

@Eric, I can easily fix it if you wish :-)

Just wondered if it's intended or not, as it looked like a bug but the 
documentation was somewhat endorsing it and I got confused.

Either case, a simple fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46938] dataclass __post_init__ recursion

[Eric V. Smith]

Eric V. Smith  added the comment:

I think this is a bug in the code. I'll have a PR ready shortly.

But since it's a non-trivial change, I'm going to target it for 3.11 only.

--
assignee: docs@python -> eric.smith
versions:  -Python 3.10, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46938] dataclass __post_init__ recursion

[Alex Waygood]

Change by Alex Waygood :


--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46938] dataclass __post_init__ recursion

[Bar Harel]

New submission from Bar Harel :

Not sure if a continuance of https://bugs.python.org/issue44365 or not, but the 
suggestion to call super().__init__() in __post__init__ will cause infinite 
recursion if the superclass also contains __post__init__:

>>> @d
... class A:
...  test: int
...  def __post_init__(self):
...pass

>>> @d
... class B(A):
...  test2: int
...  def __post_init__(self):
...super().__init__(test=1)

>>> B(test2=1, test=3) <-- infinite recursion.

This is caused by line 564 
(https://github.com/python/cpython/blob/4716f70c8543d12d18c64677af650d479b99edac/Lib/dataclasses.py#L564)
 checking for post init on current class, and calling it on self (child class).

Not sure if the bug is in the documentation/suggestion, or if it's a bug in the 
implementation, in which case we need to call the current class's __post_init__.

--
assignee: docs@python
components: Documentation, Library (Lib)
messages: 414613
nosy: bar.harel, docs@python
priority: normal
severity: normal
status: open
title: dataclass __post_init__ recursion
type: behavior
versions: Python 3.10, Python 3.11, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue46938>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26545] [doc] os.walk is limited by python's recursion limit

[Stanley]

Stanley  added the comment:

I'm not too sure about documenting the recursive limit here. There's a few 
other recursive functions in the os library (such as makedirs()) and if we note 
the recursive limit for os.walk then all of them should be noted too, but that 
doesn't seem quite right to me.

--
nosy: +slateny

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42259] pprint: infinite recursion for saferepr() when using nested objects, but str() works

[Irit Katriel]

Irit Katriel  added the comment:

The documentation change in the PR clarifies the current state of things. 

@serhiy.storchaka - do you have a view on whether we should aim for an actual 
fix, or just document and close?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42259] pprint: infinite recursion for saferepr() when using nested objects, but str() works

[Irit Katriel]

Change by Irit Katriel :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42259] pprint: infinite recursion for saferepr() when using nested objects, but str() works

[Andrei Kulakov]

Andrei Kulakov  added the comment:

The recursion protection in `saferepr` applies when two conditions are met: 

- the structure is subclassed from list, tuple or dict
- __repr__ is not overriden

In this case neither condition is met.

However, the recursion is caused by the `__repr__` so when it's removed, 
recursion doesn't happen (but not due to recursion protection).

Btw also note that recursive path must be continuous for recursion detection to 
apply, e.g. if it's list[cust_obj[list[cust_obj...]]], detection also won't 
work.

I don't think we can fix this in code in a straightforward way, because  we 
want to avoid recursively calling saferepr in case __repr__ does not recurse.

In other words, if we knew __repr__ DOES recurse, we could call saferepr 
recursively and apply recursion detection without any problems, but __repr__ 
might intentionally say something like "", and then 
recursively calling saferepr would be undesirable.

So unfortunately we lose the recursion detection because of that.

One possible option would be to add an optional param like *force_recursion*, 
to recurse with detection even on overridden *__repr__*. I'm not sure it's 
worth it. But that's something users can consider: subclass PrettyPrinter and 
override saferepr() and either remove the checks for __repr__ override or add a 
param to do just that.

Current docs really make it sound like any recursion that shows up in repr() 
will be protected; it's really much more limited than that. Adding PR to 
clarify the limitations.

--
versions: +Python 3.11 -Python 3.8

___
Python tracker 
<https://bugs.python.org/issue42259>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42259] pprint: infinite recursion for saferepr() when using nested objects, but str() works

[Andrei Kulakov]

Change by Andrei Kulakov :


--
keywords: +patch
nosy: +andrei.avk
nosy_count: 2.0 -> 3.0
pull_requests: +28477
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30256

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37578] Change Glob: Allow Recursion for Hidden Files

[Alex Waygood]

Change by Alex Waygood :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.11 -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37578] Change Glob: Allow Recursion for Hidden Files

[miss-islington]

miss-islington  added the comment:


New changeset ae36cd1e792db9d6db4c6847ec2a7d50a71f2b68 by andrei kulakov in 
branch 'main':
bpo-37578: glob.glob -- added include_hidden parameter (GH-30153)
https://github.com/python/cpython/commit/ae36cd1e792db9d6db4c6847ec2a7d50a71f2b68


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37578] Change Glob: Allow Recursion for Hidden Files

[Andrei Kulakov]

Andrei Kulakov  added the comment:

I want to clarify my first comment: my PR is similar to Zsh behaviour as 
described above; as I'm not 100% sure how bash behaves with `**` with globstar 
option, I can say that bash is the same as Zsh with ? and * magic characters 
(with dotglob option), and likely the same for `**` as well, but I can't test 
it (if someone can test it, please comment).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37578] Change Glob: Allow Recursion for Hidden Files

[Andrei Kulakov]

Andrei Kulakov  added the comment:

Isaac: my PR does allow [.] to match. But I probably need to update the docs to 
note that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37578] Change Glob: Allow Recursion for Hidden Files

[Isaac Muse]

Isaac Muse  added the comment:

If this was to be done, you'd want to make sure character sequences also match 
hidden files: [.]. Just * and ? would be incomplete. If allowing ** to match a 
leading dot, it would not match . or ..

--
nosy: +Isaac Muse

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37578] Change Glob: Allow Recursion for Hidden Files

[Andrei Kulakov]

Andrei Kulakov  added the comment:

Zsh allows use of *, ? and ** to match hidden files and directories.

Bash allows the same for * and ? with `dotglob` option. There is also a 
`globstar` but my version of bash (on macos) is too old to have it.

By the way setting options in bash is done with `shopt -s `.

I've put up a PR that enables behavior similar to Zsh and Bash (at least for ? 
and * chars).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37578] Change Glob: Allow Recursion for Hidden Files

[Andrei Kulakov]

Change by Andrei Kulakov :


--
keywords: +patch
nosy: +andrei.avk
nosy_count: 5.0 -> 6.0
pull_requests: +28372
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30153

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29221] ABC Recursion Error on isinstance() with less than recursion limit class hierarchy depth

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

It is still reproducible if increase the depth. In 3.8-3.10 it needs 329 nested 
classes, in 3.11 -- 496.

Seems the limit is sys.getrecursionlimit()//k - 4, where k=4 in 3.7 and older, 
k=3 in 3.8-3.10, and k=2 in 3.11. It is much better than was initially, but the 
ideal is k=1.

--
resolution: out of date -> 
stage: resolved -> 
status: closed -> open
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 2.7, Python 3.5, Python 
3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44475] Dataclass Causes Infinite Recursion when using type of bytes

[Eric V. Smith]

Eric V. Smith  added the comment:

Closing due to lack of feedback.

--
resolution:  -> not a bug
stage:  -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29221] ABC Recursion Error on isinstance() with less than recursion limit class hierarchy depth

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22684] message.as_bytes() produces recursion depth exceeded

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> works for me
stage:  -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29221] ABC Recursion Error on isinstance() with less than recursion limit class hierarchy depth

[Irit Katriel]

Irit Katriel  added the comment:

I am unable to reproduce this on 3.11. Is it still a problem?

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22684] message.as_bytes() produces recursion depth exceeded

[Irit Katriel]

Irit Katriel  added the comment:

I am unable to reproduce this on 3.11.

--
nosy: +iritkatriel
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42500] crash with unbounded recursion in except statement

[Łukasz Langa]

Change by Łukasz Langa :


--
superseder:  -> Cannot Recover From StackOverflow in 3.9 Tests

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43185] AssertRaises() causes core dump in handling recursion

[Łukasz Langa]

Change by Łukasz Langa :


--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed
superseder:  -> Cannot Recover From StackOverflow in 3.9 Tests

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Recursion on list

[Peter Pearson]

On Thu, 4 Nov 2021 08:57:14 +0100, ast  wrote:
> > li = []
> > li.append(li)
> > li
> [[...]]
>
> >li[0][0][0][0]
> [[...]]
>
> That's funny

After the coming AI upheaval, such cruelty to machines will
be considered punishable and not funny.


-- 
To email me, substitute nowhere->runbox, invalid->com.
-- 
https://mail.python.org/mailman/listinfo/python-list

Recursion on list

[ast]

> li = []
> li.append(li)
> li
[[...]]

>li[0][0][0][0]
[[...]]

That's funny
--
https://mail.python.org/mailman/listinfo/python-list

Re: Recursion on list

[Pieter van Oostrum]

ast  writes:

>> li = []
>> li.append(li)
>> li
> [[...]]
>
>>li[0][0][0][0]
> [[...]]
>
> That's funny
>

You made a list whose only element is itself.

In [1]: li = []

In [3]: li.append(li)

In [4]: li[0] is li
Out[4]: True

-- 
Pieter van Oostrum 
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue45687] Infinite recursion in Pickler.persistent_id

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components: +Interpreter Core
nosy: +serhiy.storchaka
versions: +Python 3.11 -Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45687] Infinite recursion in Pickler.persistent_id

[Michał Bartoszkiewicz]

New submission from Michał Bartoszkiewicz :

The following code, which seems reasonable:
import io
import pickle

class Pickler(pickle.Pickler):
  def persistent_id(self, obj):
return super().persistent_id(obj)

Pickler(io.BytesIO()).dump(42)

crashes with:
RecursionError: maximum recursion depth exceeded while calling a Python object

It works perfectly when inheriting from pickle._Pickler (the Python 
implementation).

--
components: Library (Lib)
files: pickle-bug.py
messages: 405494
nosy: embe-navalgo
priority: normal
severity: normal
status: open
title: Infinite recursion in Pickler.persistent_id
type: behavior
versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file50418/pickle-bug.py

___
Python tracker 
<https://bugs.python.org/issue45687>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[Eryk Sun]

Eryk Sun  added the comment:

In theory, a crash could be prevented in most cases by setting a larger stack 
guarantee (i.e. region of guard pages) via SetThreadStackGuarantee() [1] and 
using a vectored exception handler [2]. The exception handler can set a flag in 
the thread state that indicates stack-overflow recovery is in progress and then 
return EXCEPTION_CONTINUE_EXECUTION. The guaranteed stack space will be 
available, but there are no guard pages, so another stack overflow in this 
context will crash with an access violation. The stack guarantee should be 
large enough to raise and unwind a RecursionError. As the stack unwinds, if the 
recovery flag is still set, try calling _resetstkoflw() [3] to restore the 
guard region. If it succeeds, clear the flag in the thread state.

For giggles, here's a toy example using ctypes:

import ctypes
import sys

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
ucrt = ctypes.CDLL('ucrtbase', use_errno=True)

EXCEPTION_CONTINUE_EXECUTION = 0x

stack_overflow = False

@ctypes.WINFUNCTYPE(ctypes.c_long, ctypes.c_void_p)
def handler(p):
global stack_overflow
stack_overflow = True
return EXCEPTION_CONTINUE_EXECUTION

kernel32.AddVectoredExceptionHandler(1, handler)

def recursive():
if stack_overflow:
raise RecursionError
recursive()

# Normally the stack has 2 or 3 guard pages, which is actually
# enough to recover in this example, but let's increase it to
# 9 pages (8 plus an extra that the memory manager adds). You
# can inspect this with Sysinternals VMMap.
size = ctypes.c_ulong(8 * 4096)
kernel32.SetThreadStackGuarantee(ctypes.byref(size))

sys.setrecursionlimit(100)

for n in range(5):
try:
recursive()
except RecursionError:
if stack_overflow and ucrt._resetstkoflw():
stack_overflow = False
print("recovered from stack overflow:", n)

---
[1] 
https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-setthreadstackguarantee
[2] 
https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-addvectoredexceptionhandler
[3] 
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/resetstkoflw?view=msvc-160

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[PABLO LOBATO DE LA CRUZ]

PABLO LOBATO DE LA CRUZ  added the comment:

I see. Thank you very much Steve :)

El vie, 29 oct 2021 a las 0:29, Steve Dower ()
escribió:

>
> Steve Dower  added the comment:
>
> > But why does it happen only on Windows?
>
> Because it's the operating system that is terminating the process :)
> Other operating systems behave differently when you exceed the stack,
> and may also allocate different amounts of stack space, making it less
> likely that you hit it.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[Steve Dower]

Steve Dower  added the comment:

> But why does it happen only on Windows?

Because it's the operating system that is terminating the process :) 
Other operating systems behave differently when you exceed the stack, 
and may also allocate different amounts of stack space, making it less 
likely that you hit it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[PABLO LOBATO DE LA CRUZ]

PABLO LOBATO DE LA CRUZ  added the comment:

I see thanks for answering so quickly. But why does it happen only on
Windows?

El jue, 28 oct 2021 a las 23:09, Steve Dower ()
escribió:

>
> Steve Dower  added the comment:
>
> This is almost certainly because of how Windows handles stack overflow
> exceptions, and the fact that there's no way for us to detect it reliably.
>
> There's some work going on to reduce the C stack depth when calling
> heavily nested Python code (see issue45256), but there's nothing else we
> can really do I'm afraid.
>
> I'm marking *this* issue as wontfix, but hopefully we can make some
> improvement on this general issue through other issues. Thanks for
> reporting it!
>
> --
> resolution:  -> wont fix
> stage:  -> resolved
> status: open -> closed
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[Steve Dower]

Steve Dower  added the comment:

This is almost certainly because of how Windows handles stack overflow 
exceptions, and the fact that there's no way for us to detect it reliably.

There's some work going on to reduce the C stack depth when calling heavily 
nested Python code (see issue45256), but there's nothing else we can really do 
I'm afraid.

I'm marking *this* issue as wontfix, but hopefully we can make some improvement 
on this general issue through other issues. Thanks for reporting it!

--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[PABLO LOBATO DE LA CRUZ]

New submission from PABLO LOBATO DE LA CRUZ :

Deep recursion crashes on Windows (Python 3.9) when the depth limit is 
increased and no error is shown.
Seems to work fine on other systems that I have tried (Linux and MacOS).
Please find attached the script to reproduce the error.
Expected and other systems output:

Calculated N2
Try block finished correctly
This is the finally block

Output:

Calculated N2

--
components: Windows
files: forstack.py
messages: 405185
nosy: pablolob, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Deep recursion terminates script execution with no error (Windows, 
Python 3.9)
type: crash
versions: Python 3.9
Added file: https://bugs.python.org/file50407/forstack.py

___
Python tracker 
<https://bugs.python.org/issue45645>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45564] shutil.rmtree and os.walk are implemented using recursion, fail on deep hierarchies

[Alexander Patrakov]

New submission from Alexander Patrakov :

It is possible to create deep directory hierarchies that cannot be removed via 
shutil.rmtree or walked via os.walk, because these functions exceed the 
interpreter recursion limit. This may have security implications for web 
services (e.g. various webdisks) that have to clean up user-created mess or 
walk through it.

[aep@aep-haswell ~]$ mkdir /tmp/badstuff
[aep@aep-haswell ~]$ cd /tmp/badstuff
[aep@aep-haswell badstuff]$ for x in `seq 2048` ; do mkdir $x ; cd $x ; done
[aep@aep-haswell 103]$ cd
[aep@aep-haswell ~]$ python
Python 3.9.7 (default, Oct 10 2021, 15:13:22) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.rmtree('/tmp/badstuff')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.9/shutil.py", line 726, in rmtree
_rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 992 more times]
  File "/usr/lib/python3.9/shutil.py", line 642, in _rmtree_safe_fd
fullname = os.path.join(path, entry.name)
  File "/usr/lib/python3.9/posixpath.py", line 77, in join
sep = _get_sep(a)
  File "/usr/lib/python3.9/posixpath.py", line 42, in _get_sep
if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
>>> import os
>>> list(os.walk('/tmp/badstuff'))
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.9/os.py", line 418, in _walk
yield from _walk(new_path, topdown, onerror, followlinks)
  File "/usr/lib/python3.9/os.py", line 418, in _walk
yield from _walk(new_path, topdown, onerror, followlinks)
  File "/usr/lib/python3.9/os.py", line 418, in _walk
yield from _walk(new_path, topdown, onerror, followlinks)
  [Previous line repeated 993 more times]
  File "/usr/lib/python3.9/os.py", line 412, in _walk
new_path = join(top, dirname)
  File "/usr/lib/python3.9/posixpath.py", line 77, in join
sep = _get_sep(a)
  File "/usr/lib/python3.9/posixpath.py", line 42, in _get_sep
if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
>>>

--
components: Library (Lib)
messages: 404687
nosy: Alexander.Patrakov
priority: normal
severity: normal
status: open
title: shutil.rmtree and os.walk are implemented using recursion, fail on deep 
hierarchies
type: crash
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue45564>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[STINNER Victor]

STINNER Victor  added the comment:

Thanks for the report and the fix!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[STINNER Victor]

STINNER Victor  added the comment:


New changeset fd74d2680ef96c0140bc02cf94d1cf1f2ef814c2 by Miss Islington (bot) 
in branch '3.10':
bpo-45156: Fixes inifite loop on unittest.mock.seal() (GH-28300) (GH-28326)
https://github.com/python/cpython/commit/fd74d2680ef96c0140bc02cf94d1cf1f2ef814c2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[miss-islington]

miss-islington  added the comment:


New changeset 13257d9fca13dfa1bda5b802d68ddaec72f3a07e by Miss Islington (bot) 
in branch '3.9':
bpo-45156: Fixes inifite loop on unittest.mock.seal() (GH-28300)
https://github.com/python/cpython/commit/13257d9fca13dfa1bda5b802d68ddaec72f3a07e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[miss-islington]

Change by miss-islington :


--
pull_requests: +26738
pull_request: https://github.com/python/cpython/pull/28327

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +26737
pull_request: https://github.com/python/cpython/pull/28326

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 7f60c9e1c6e22cc0e846a872c318570926cd3094 by Nikita Sobolev in 
branch 'main':
bpo-45156: Fixes inifite loop on unittest.mock.seal() (GH-28300)
https://github.com/python/cpython/commit/7f60c9e1c6e22cc0e846a872c318570926cd3094


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[Nikita Sobolev]

Nikita Sobolev  added the comment:

I've proposed a solution, based on the assumption that we don't need to recurse 
into `_SpecState` props:

```
if isinstance(m._mock_children.get(attr), _SpecState):
   continue
```

It seems like a simple and reasonable thing to do.
Link: https://github.com/python/cpython/pull/28300

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[Nikita Sobolev]

Change by Nikita Sobolev :


--
keywords: +patch
nosy: +sobolevn
nosy_count: 3.0 -> 4.0
pull_requests: +26714
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28300

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[Irit Katriel]

Irit Katriel  added the comment:

Pdb) list
2916if m._mock_new_parent is mock:
2917  try:
2918seal(m)
2919  except:
2920breakpoint()
2921 -> raise
2922
2923
2924class _AsyncIterator:
2925"""
2926Wraps an iterator in an asynchronous iterator.
(Pdb) p m


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[Irit Katriel]

Irit Katriel  added the comment:

Reproduced on 3.11:

>>> foo = mock.create_autospec(Foo)
>>> mock.seal(foo)

^CTraceback (most recent call last):
  File "", line 1, in 
  File "/Users/iritkatriel/src/cpython-1/Lib/unittest/mock.py", line 2917, in 
seal
seal(m)
^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/unittest/mock.py", line 2917, in 
seal
seal(m)
^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/unittest/mock.py", line 2917, in 
seal
seal(m)
^^^
  [Previous line repeated 645 more times]
  File "/Users/iritkatriel/src/cpython-1/Lib/unittest/mock.py", line 2911, in 
seal
m = getattr(mock, attr)
^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/unittest/mock.py", line 662, in 
__getattr__
result = create_autospec(
 
  File "/Users/iritkatriel/src/cpython-1/Lib/unittest/mock.py", line 2672, in 
create_autospec
mock = Klass(parent=_parent, _new_parent=_parent, _new_name=_new_name,
   ^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/unittest/mock.py", line 416, in 
__new__
new = type(cls.__name__, bases, {'__doc__': cls.__doc__})
  ^^^
KeyboardInterrupt

--
nosy: +iritkatriel
versions: +Python 3.10, Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45156] mock.seal has infinite recursion with int class attributes

[David Mandelberg]

New submission from David Mandelberg :

The code below seems to have infinite recursion in the mock.seal call with 
python 3.9.2.

from unittest import mock
class Foo:
  foo = 0
foo = mock.create_autospec(Foo)
mock.seal(foo)

--
components: Library (Lib)
messages: 401525
nosy: dseomn
priority: normal
severity: normal
status: open
title: mock.seal has infinite recursion with int class attributes
type: crash
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue45156>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26545] [doc] os.walk is limited by python's recursion limit

[Ryan Mast (nightlark)]

Change by Ryan Mast (nightlark) :


--
nosy: +rmast

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44917] interpreter hangs on recursion in both body and handler of a try block

[Irit Katriel]

Irit Katriel  added the comment:

Right, this is a duplicate of issue42951.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> Random and infinite loop in dealing with recursion error for 
"try-except "

___
Python tracker 
<https://bugs.python.org/issue44917>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44917] interpreter hangs on recursion in both body and handler of a try block

[Mark Shannon]

Mark Shannon  added the comment:

A recursion limit of 30 is effectively infinite.
With a debug build of 3.11, the time to execute grows very fast indeed, 
probably super-exponentially.

mark@nero:~/repos/cpython$ time ./python ~/test/test.py 15

real0m1.107s
user0m1.099s
sys 0m0.008s
mark@nero:~/repos/cpython$ time ./python ~/test/test.py 16

real0m4.468s
user0m4.464s
sys 0m0.004s
mark@nero:~/repos/cpython$ time ./python ~/test/test.py 17

real0m20.968s
user0m20.928s
sys 0m0.040s
mark@nero:~/repos/cpython$ time ./python ~/test/test.py 18

real2m29.562s
user2m29.270s
sys 0m0.140s


I would expect ./python ~/test/test.py 30 to take millions of years.

--

___
Python tracker 
<https://bugs.python.org/issue44917>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44917] interpreter hangs on recursion in both body and handler of a try block

[Irit Katriel]

Irit Katriel  added the comment:

Not sure. Here we do set the recursion limit.

--

___
Python tracker 
<https://bugs.python.org/issue44917>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44917] interpreter hangs on recursion in both body and handler of a try block

[Mark Shannon]

Mark Shannon  added the comment:

This looks like a duplicate of https://bugs.python.org/issue42951

--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44917] interpreter hangs on recursion in both body and handler of a try block

[Irit Katriel]

Irit Katriel  added the comment:

Perhaps the interpreter should detect that it is about to raise a RecusionError 
whose context is another RecursionError, and raise a FatalError instead?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44917] interpreter hangs on recursion in both body and handler of a try block

[Irit Katriel]

New submission from Irit Katriel :

This was found while investigating issue44895. It may or may not be the cause 
of that issue.


The script below hangs on a mac (it's an extract from 
test_exceptions.test_recursion_in_except_handler).

---
import sys

count = 0
def main():

  def f():
global count
count += 1
try:
f()
except RecursionError:
f()

  sys.setrecursionlimit(30)

  try:
f()
  except RecursionError:
pass

main()
print(count)
---


When I kill it the traceback shows it alternating between the two recursive 
calls, but not in a regular pattern:


... [snipped a lot]

  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
  [Previous line repeated 2 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
    ^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
  File "/Users/iritkatriel/src/cpython/tt.py", line 13, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/tt.py", line 11, in f
f()
^^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most r

[issue33930] Segfault with deep recursion into object().__dir__

[miss-islington]

miss-islington  added the comment:


New changeset ef36dfe4de1ee0df11cde94fd76465336aa8141d by Benjamin Peterson in 
branch '3.10':
[3.10] bpo-33930: Fix typo in the test name. (GH-27736)
https://github.com/python/cpython/commit/ef36dfe4de1ee0df11cde94fd76465336aa8141d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Łukasz Langa]

Łukasz Langa  added the comment:

Since the refleak investigation moved to its own issue, I'm re-closing this one 
for clarity.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[miss-islington]

miss-islington  added the comment:


New changeset f7635f0e542c916bfd62542e7b60ee23ff681d0d by Miss Islington (bot) 
in branch '3.9':
[3.9] bpo-33930: Fix typo in the test name. (GH-27733) (GH-27734)
https://github.com/python/cpython/commit/f7635f0e542c916bfd62542e7b60ee23ff681d0d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Benjamin Peterson]

Benjamin Peterson  added the comment:


New changeset d7f5796a1ec7ba223f6a844d7580559abef05238 by Miss Islington (bot) 
in branch '3.8':
bpo-33930: Fix typo in the test name. (GH-27735)
https://github.com/python/cpython/commit/d7f5796a1ec7ba223f6a844d7580559abef05238


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Benjamin Peterson]

Change by Benjamin Peterson :


--
pull_requests: +26216
pull_request: https://github.com/python/cpython/pull/27736

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[miss-islington]

Change by miss-islington :


--
pull_requests: +26214
pull_request: https://github.com/python/cpython/pull/27734

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[miss-islington]

Change by miss-islington :


--
pull_requests: +26215
pull_request: https://github.com/python/cpython/pull/27735

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Benjamin Peterson]

Benjamin Peterson  added the comment:


New changeset f08e6d1bb3c5655f184af88c6793e90908bb6338 by Benjamin Peterson in 
branch 'main':
bpo-33930: Fix typo in the test name. (#27733)
https://github.com/python/cpython/commit/f08e6d1bb3c5655f184af88c6793e90908bb6338


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Benjamin Peterson]

Change by Benjamin Peterson :


--
nosy: +benjamin.peterson
nosy_count: 14.0 -> 15.0
pull_requests: +26213
pull_request: https://github.com/python/cpython/pull/27733

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com