On 2021-02-28 23:28, Peter Otten wrote:
On 28/02/2021 23:33, Marco Sulla wrote:
On Sun, 28 Feb 2021 at 01:19, Cameron Simpson <c...@cskk.id.au> wrote:
My object represents an MDAT box in an MP4 file: it is the ludicrously
large data box containing the raw audiovideo data; for a TV episode it
is often about 2GB and a movie is often 4GB to 6GB.
[...]
That length is presented via the object's __len__ method
[...]
I noticed that it was stalling, and investigation revealed it was
stalling at this line:
subboxes = list(self)
when doing the MDAT box. That box (a) has no subboxes at all and (b) has
a very large __len__ value.
BUT... It also has a __iter__ value, which like any Box iterates over
the subboxes. For MDAT that is implemented like this:
def __iter__(self):
yield from ()
What I was expecting was pretty much instant construction of an empty
list. What I was getting was a very time consuming (10 seconds or more)
construction of an empty list.
I can't reproduce, Am I missing something?
marco@buzz:~$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
class A:
... def __len__(self):
... return 1024**3
... def __iter__(self):
... yield from ()
...
a = A()
len(a)
1073741824
list(a)
[]
It takes milliseconds to run list(a)
Looks like you need at least Python 3.8 to see this. Quoting
https://docs.python.org/3/whatsnew/3.8.html:
"""
The list constructor does not overallocate the internal item buffer if
the input iterable has a known length (the input implements __len__).
This makes the created list 12% smaller on average. (Contributed by
Raymond Hettinger and Pablo Galindo in bpo-33234.)
"""
I'm not seeing a huge problem here:
Python 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> class A:
... def __len__(self):
... return 1024**3
... def __iter__(self):
... yield from ()
...
>>> a = A()
>>> len(a)
1073741824
>>> s = time.time()
>>> list(a)
[]
>>> print(time.time() - s)
0.16294455528259277
--
https://mail.python.org/mailman/listinfo/python-list
