Can you distill the program into something reproducible?
Maybe with something slightly less than 45Gb but still exhibiting some
degradation of exit performance?
I can try to point our commercial profiling tools at it and see what it is
doing.
K
-Original Message-
From:
On Sat, 20 Dec 2008 09:02:38 pm Kristján Valur Jónsson wrote:
Can you distill the program into something reproducible?
Maybe with something slightly less than 45Gb but still exhibiting
some degradation of exit performance? I can try to point our
commercial profiling tools at it and see what
Mike Coleman wrote:
I have a program that creates a huge (45GB) defaultdict. (The keys
are short strings, the values are short lists of pairs (string, int).)
Nothing but possibly the strings and ints is shared.
The program takes around 10 minutes to run, but longer than 20 minutes
to exit (I
Andrew MacIntyre wrote:
Mike Coleman wrote:
I have a program that creates a huge (45GB) defaultdict. (The keys
are short strings, the values are short lists of pairs (string, int).)
Nothing but possibly the strings and ints is shared.
The program takes around 10 minutes to run, but longer
Benjamin Peterson schrieb:
On Fri, Dec 19, 2008 at 12:55 PM, Tarek Ziadé ziade.ta...@gmail.com wrote:
Hello
I would like to request a commit access to work specifically on
distutils maintenance.
+1
We are currently without an active distutils maintainer, and many
stale distutil tickets
Barry Warsaw schrieb:
I'd like to get Python 3.0.1 out before the end of the year. There
are no showstoppers, but I haven't yet looked at the deferred blockers
or the buildbots.
Do you think we can get 3.0.1 out on December 24th? Or should we wait
until after Christmas and get it out, say
Steve Unfortunately there are doubtless programs out there that do rely
Steve on actions being taken at shutdown.
Indeed. I believe any code which calls atexit.register.
Steve Maybe os.exit() could be more widely advertised, though ...
That would be os._exit(). Calling it avoids
On Sat, Dec 20, 2008, Nick Coghlan wrote:
It turns out that _PyBuiltin_Init doesn't call PyType_Ready on any of
the builtin types - they're left to have it called implicitly when an
operation using them needs tp_dict filled in.
This seems like a release blocker for 3.0.1 to me
--
Aahz
Andrew, this is on an (intel) x86_64 box with 64GB of RAM. I don't
recall the maker or details of the architecture off the top of my
head, but it would be something off the rack from Dell or maybe HP.
There were other users on the box at the time, but nothing heavy or
that gave me any reason to
Mike Coleman wrote:
... Regarding interning, I thought this only worked with strings.
Is there some way to intern integers? I'm probably creating 300M
integers more or less uniformly distributed across range(1)?
held = list(range(1))
...
troublesome_dict[string] =
You can always try poor-man's profiling, which is surprisingly useful in the
face of massive performance problems.
Just attach a debugger to the program, and when it suffering from a performance
problem, break the execution on a regular basis. You are statistically very
likely to get a
Hi fellow snakemen and lizard ladies,
We have been recently done lots of Python work on Nokia Series 60 phones and
even managed to roll out some commercial Python based applications. In the
future we plan to create some iPhone Python apps also.
Python runs fine in phones - after it has been
Steven D'Aprano steve at pearwood.info writes:
In November 2007, a similar problem was reported on the comp.lang.python
newsgroup. 370MB was large enough to demonstrate the problem. I don't
know if a bug was ever reported.
Do you still reproduce it on trunk?
I've tried your scripts on my
On 2008-12-20 17:57, Mike Coleman wrote:
On Sat, Dec 20, 2008 at 4:02 AM, Kristján Valur Jónsson
krist...@ccpgames.com wrote:
Can you distill the program into something reproducible?
Maybe with something slightly less than 45Gb but still exhibiting some
degradation of exit performance?
I
On Sat, Dec 20, 2008 at 3:04 PM, M.-A. Lemburg m...@egenix.com wrote:
These long exit times are usually caused by the garbage collection
of objects. This can be a very time consuming task.
In that case, the question would be why is the interpreter collecting
garbage when it knows we're trying
Leif Walsh wrote:
On Sat, Dec 20, 2008 at 3:04 PM, M.-A. Lemburg m...@egenix.com wrote:
These long exit times are usually caused by the garbage collection
of objects. This can be a very time consuming task.
In that case, the question would be why is the interpreter collecting
garbage
Leif In that case, the question would be why is the interpreter
Leif collecting garbage when it knows we're trying to exit anyway?.
Because useful side effects are sometimes performed as a result of this
activity (flushing disk buffers, closing database connections, etc).
Skip
[M.-A. Lemburg]
These long exit times are usually caused by the garbage collection
of objects. This can be a very time consuming task.
[Leif Walsh]
In that case, the question would be why is the interpreter collecting
garbage when it knows we're trying to exit anyway?.
Because user-defined
On 2008-12-20 21:20, Leif Walsh wrote:
On Sat, Dec 20, 2008 at 3:04 PM, M.-A. Lemburg m...@egenix.com wrote:
These long exit times are usually caused by the garbage collection
of objects. This can be a very time consuming task.
In that case, the question would be why is the interpreter
(@Skip, Michael, Tim)
On Sat, Dec 20, 2008 at 3:26 PM, s...@pobox.com wrote:
Because useful side effects are sometimes performed as a result of this
activity (flushing disk buffers, closing database connections, etc).
Of course they are. But what about the case given above:
On Sat, Dec 20,
[Mike Coleman]
... Regarding interning, I thought this only worked with strings.
Implementation details. Recent versions of CPython also, e.g.,
intern the empty tuple, and very small integers.
Is there some way to intern integers? I'm probably creating 300M
integers more or less uniformly
[Leif Walsh]
...
It might be a semantic change that I'm looking for here, but it seems
to me that if you turn off the garbage collector, you should be able
to expect that either it also won't run on exit,
It won't then, but the garbage collector is the gc module, and that
only performs
Leif Walsh leif.walsh at gmail.com writes:
It might be a semantic change that I'm looking for here, but it seems
to me that if you turn off the garbage collector, you should be able
to expect that either it also won't run on exit, or it should have a
way of letting you tell it not to run on
It appears that this bug was already reported: http://bugs.python.org/issue4705
Any chance that it gets in the next 3.0.x bugfix release?
Just as a note, if I do: sys.stdout._line_buffering = True, it also
works, but doesn't seem right as it's accessing an internal attribute.
Note 2: the
Any opinions?
I would use a different marshal implementation. Instead of defining
a stream format for marshal, make marshal dump its graph of objects
along with the actual memory layout. On load, copying can
be avoided; just a few pointers need to be updated. The resulting
marshal files would be
On Sat, Dec 20, 2008 at 2:50 PM, M.-A. Lemburg m...@egenix.com wrote:
If you want a really fast exit, try this:
import os
os.kill(os.getpid(), 9)
But you better know what you're doing if you take this approach...
This would work, but I think os._exit(EX_OK) is probably just as fast,
and
I will try next week to see if I can come up with a smaller,
submittable example. Thanks.
These long exit times are usually caused by the garbage collection
of objects. This can be a very time consuming task.
I doubt that. The long exit times are usually caused by a bad
malloc
Python 2.6.1 documentation currently isn't available for download at:
http://docs.python.org/ftp/python/doc/
Additionally please include version numbers in documentation
archives (e.g. python-docs-html-2.6.1.tar.bz2).
--
Arfrever Frehtes Taifersar Arahesis
signature.asc
Description: This is a
It might be helpful if
http://www.python.org/download/releases/2.6.1/license/
said it was also the official license for the 2.6.1 release (though I
don't suppose it matters that it's still called the 2.5 license, since
that's its origin).
Another detail to go into the release manage PEP?
Antoine Pitrou wrote:
Leif Walsh leif.walsh at gmail.com writes:
It might be a semantic change that I'm looking for here, but it seems
to me that if you turn off the garbage collector, you should be able
to expect that either it also won't run on exit, or it should have a
way of letting you
On Sat, Dec 20, 2008 at 4:28 PM, Arfrever Frehtes Taifersar Arahesis
arfrever@gmail.com wrote:
Python 2.6.1 documentation currently isn't available for download at:
http://docs.python.org/ftp/python/doc/
It is avaiable here, though:
http://www.python.org/ftp/python/doc/current/
On Sat, Dec 20, 2008 at 4:37 PM, Steve Holden st...@holdenweb.com wrote:
It might be helpful if
http://www.python.org/download/releases/2.6.1/license/
said it was also the official license for the 2.6.1 release (though I
don't suppose it matters that it's still called the 2.5 license, since
2008-12-20 23:46:15 Benjamin Peterson napisał(a):
On Sat, Dec 20, 2008 at 4:28 PM, Arfrever Frehtes Taifersar Arahesis
arfrever@gmail.com wrote:
Python 2.6.1 documentation currently isn't available for download at:
http://docs.python.org/ftp/python/doc/
It is avaiable here, though:
On Sat, Dec 20, 2008 at 13:45, Fabio Zadrozny fabi...@gmail.com wrote:
It appears that this bug was already reported:
http://bugs.python.org/issue4705
Any chance that it gets in the next 3.0.x bugfix release?
Just as a note, if I do: sys.stdout._line_buffering = True, it also
works, but
Steve Holden steve at holdenweb.com writes:
I believe the OP engendered a certain amount of confusion by describing
object deallocation as being performed by the garbage collector. So he
perhaps didn't understand that even decref'ing all the objects only
referenced by the dict will take a huge
On Fri, Dec 19, 2008 at 6:29 PM, Mike Coleman tutu...@gmail.com wrote:
I have a program that creates a huge (45GB) defaultdict. (The keys
are short strings, the values are short lists of pairs (string, int).)
Nothing but possibly the strings and ints is shared.
That is, after executing
[Sorry, for the previous garbage post.]
On Fri, Dec 19, 2008 at 6:29 PM, Mike Coleman tutu...@gmail.com wrote:
I have a program that creates a huge (45GB) defaultdict. (The keys
are short strings, the values are short lists of pairs (string, int).)
Nothing but possibly the strings and ints
Tim Peters wrote:
If that is the case here, there's no evident general solution. If you
have millions of objects still alive at exit, refcount-based
reclamation has to visit all of them, and if they've been swapped out
to disk it can take a very long time to swap them all back into memory
s...@pobox.com wrote:
Steve Unfortunately there are doubtless programs out there that do rely
Steve on actions being taken at shutdown.
Indeed. I believe any code which calls atexit.register.
Steve Maybe os.exit() could be more widely advertised, though ...
That would be
Aahz wrote:
On Sat, Dec 20, 2008, Nick Coghlan wrote:
It turns out that _PyBuiltin_Init doesn't call PyType_Ready on any of
the builtin types - they're left to have it called implicitly when an
operation using them needs tp_dict filled in.
This seems like a release blocker for 3.0.1 to me
Tim, I left out some details that I believe probably rule out the
swapped out theory. The machine in question has 64GB RAM, but only
16GB swap. I'd prefer more swap, but in any case only around ~400MB
of the swap was actually in use during my program's entire run.
Furthermore, during my
Re held and intern_it: Haha! That's evil and extremely evil,
respectively. :-)
I will add these to the Python wiki if they're not already there...
Mike
___
Python-Dev mailing list
Python-Dev@python.org
On Sat, Dec 20, 2008 at 4:11 PM, Tim Peters tim.pet...@gmail.com wrote:
[Lots of answers]
Thanks. Wish I could have offered something useful.
--
Cheers,
Leif
___
Python-Dev mailing list
Python-Dev@python.org
Mike Coleman tutufan at gmail.com writes:
Just to clarify, I didn't gc.disable() to address this problem, but
rather because it destroys performance during the creation of the huge
dict. I don't have a specific number, but I think disabling gc
reduced construction from something like 70
On Sat, Dec 20, 2008 at 5:40 PM, Alexandre Vassalotti
alexan...@peadrop.com wrote:
Could you give us more information about the dictionary. For example,
how many objects does it contain? Is 45GB the actual size of the
dictionary or of the Python process?
The 45G was the VM size of the process
Mike Coleman tutufan at gmail.com writes:
The 45G was the VM size of the process (resident size was similar).
Can you reproduce it with a smaller working set? Something between 1 and 2GB,
possibly randomly-generated, and post both the generation script and the
problematic script on the bug
On Sat, Dec 20, 2008 at 5:02 PM, Arfrever Frehtes Taifersar Arahesis
arfrever@gmail.com wrote:
2008-12-20 23:46:15 Benjamin Peterson napisał(a):
On Sat, Dec 20, 2008 at 4:28 PM, Arfrever Frehtes Taifersar Arahesis
arfrever@gmail.com wrote:
Python 2.6.1 documentation currently isn't
4631 should be a release blocker. I'll have a bit of time on Monday
and Tuesday to wrap it up.
Jeremy
On Fri, Dec 19, 2008 at 5:28 PM, Barry Warsaw ba...@python.org wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I'd like to get Python 3.0.1 out before the end of the year. There are
Mike Coleman wrote:
Andrew, this is on an (intel) x86_64 box with 64GB of RAM. I don't
recall the maker or details of the architecture off the top of my
head, but it would be something off the rack from Dell or maybe HP.
There were other users on the box at the time, but nothing heavy or
that
49 matches
Mail list logo
