[Python-Dev] cProfile and threads

[Kristján Valur Jónsson]

Hello there.
I'd like to draw your attention to two feature requests / patches that I've 
subbmitted:
http://bugs.python.org/issue9609
http://bugs.python.org/issue9622

These patches are the result of work that we have done in profiling Stackless 
Python server applications at runtime, but they apply just as well to C Python.
The first patch makes _lsprof, the engine behind cProfile, multi-stack aware.  
This allows the same cProfiler.Profile() instance to be active on multiple 
python threads and still meaningful information is gathered.
The second patch allows to set the trace/profile function in python globally, 
so that all threads are affected.  This is essential if you want to take a 
profililng snapshot of a running application.

We now use this extensively, to monitor the live behaviour of our EVE game 
servers.  A HTTP backend server is used to control the profiler 
(profile.enable(), profile disable() at runtime) and explore its output.

I haven't seen any feadback on these submissions and would appreciate some.

Cheers,
Kristján
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cProfile and threads

[Nick Coghlan]

2010/8/17 Kristján Valur Jónsson :
> Hello there.
>
> I‘d like to draw your attention to two feature requests / patches that I‘ve
> subbmitted:
>
> http://bugs.python.org/issue9609
> http://bugs.python.org/issue9622
>
> These patches are the result of work that we have done in profiling
> Stackless Python server applications at runtime, but they apply just as well
> to C Python.

Both look like good ideas to me (multi-threaded profiling and
debugging is fairly painful and it would be good to be able to do
something to improve that situation).

> The first patch makes _lsprof, the engine behind cProfile, multi-stack
> aware.  This allows the same cProfiler.Profile() instance to be active on
> multiple python threads and still meaningful information is gathered.

I'm curious as to the memory impact this has on the profiler (it
obviously can't be too bad if you're able to run it against your live
servers).

> The second patch allows to set the trace/profile function in python
> globally, so that all threads are affected.  This is essential if you want
> to take a profililng snapshot of a running application.

One minor quibble here is that I would suggest using "global=False" in
your docstring signatures.

Both patches seem to be missing updates to the relevant documentation.
I expect this would be difficult to unit test properly, but at least
some basic tests to check that the new global configuration of
settrace and setprofile don't blow, and that a profiler can be shared
between two threads would be good.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cProfile and threads

[Kristján Valur Jónsson]

> -Original Message-
> From: Nick Coghlan [mailto:ncogh...@gmail.com]
> Sent: 17. ágúst 2010 10:04
> 
> Both look like good ideas to me (multi-threaded profiling and debugging is
> fairly painful and it would be good to be able to do something to improve
> that situation).
> 
Indeed.  I expect that profiling web server frameworks would be a good 
candidate, since a "profile page" can
be easily set up.

> > The first patch makes _lsprof, the engine behind cProfile, multi-stack
> > aware.  This allows the same cProfiler.Profile() instance to be active
> > on multiple python threads and still meaningful information is gathered.
> 
> I'm curious as to the memory impact this has on the profiler (it obviously 
> can't
> be too bad if you're able to run it against your live servers).
The change is minimal.  We have to have an extra rotatingtree to match tls to 
stack
anchor points.  There are also a few extra members in the profiling "Context" 
entries,
but these are ephemeral.

> 
> One minor quibble here is that I would suggest using "global=False" in your
> docstring signatures.
Okay.  The docs also need to be in line with the docstrings (e.g. the docs
say "setprofile(profilefunc)" while the docstring says "setprofile(function)"
> 
> Both patches seem to be missing updates to the relevant documentation.
Yes, this is intentional.  I didn't want to waste effort on writing 
documentation
before having exposed this to you.  Sometimes my good ideas turn out to be
not so good and end up being rejected.

> I expect this would be difficult to unit test properly, but at least some 
> basic
> tests to check that the new global configuration of settrace and setprofile
> don't blow, and that a profiler can be shared between two threads would be
> good.
This is my intention too, but again, I wanted to give this some airing first.
What't I probably end up doing is setting up a few threads, have them do
some token work, and then do analysis on the Profile.stats member to make
sure that all of them were accounted for and all and only callgraph relations
show up.

Kristján
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cProfile and threads

[Antoine Pitrou]

On Tue, 17 Aug 2010 09:22:15 +
Kristján Valur Jónsson  wrote:
> Hello there.
> I'd like to draw your attention to two feature requests / patches that I've 
> subbmitted:
> http://bugs.python.org/issue9609
> http://bugs.python.org/issue9622
> 
> These patches are the result of work that we have done in profiling Stackless 
> Python server applications at runtime, but they apply just as well to C 
> Python.
> The first patch makes _lsprof, the engine behind cProfile, multi-stack aware. 
>  This allows the same cProfiler.Profile() instance to be active on multiple 
> python threads and still meaningful information is gathered.

Does that mean you're proposing code for inclusion in CPython that can
only be tested with Stackless?
Can't Stackless use its own patches instead?

> The second patch allows to set the trace/profile function in python globally, 
> so that all threads are affected.  This is essential if you want to take a 
> profililng snapshot of a running application.

I've often heard that cProfile didn't give correct profiling
information with multiple threads. Is it true?

Thanks

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cProfile and threads

[Antoine Pitrou]

Ok, I've looked at the patch and it's actually stackless-agnostic.

Regards

Antoine.


On Tue, 17 Aug 2010 12:31:30 +0200
Antoine Pitrou  wrote:
> On Tue, 17 Aug 2010 09:22:15 +
> Kristján Valur Jónsson  wrote:
> > Hello there.
> > I'd like to draw your attention to two feature requests / patches that I've 
> > subbmitted:
> > http://bugs.python.org/issue9609
> > http://bugs.python.org/issue9622
> > 
> > These patches are the result of work that we have done in profiling 
> > Stackless Python server applications at runtime, but they apply just as 
> > well to C Python.
> > The first patch makes _lsprof, the engine behind cProfile, multi-stack 
> > aware.  This allows the same cProfiler.Profile() instance to be active on 
> > multiple python threads and still meaningful information is gathered.
> 
> Does that mean you're proposing code for inclusion in CPython that can
> only be tested with Stackless?
> Can't Stackless use its own patches instead?
> 
> > The second patch allows to set the trace/profile function in python 
> > globally, so that all threads are affected.  This is essential if you want 
> > to take a profililng snapshot of a running application.
> 
> I've often heard that cProfile didn't give correct profiling
> information with multiple threads. Is it true?
> 
> Thanks
> 
> Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cProfile and threads

[Kristján Valur Jónsson]

> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On
> Behalf Of Antoine Pitrou
> 
> Does that mean you're proposing code for inclusion in CPython that can only
> be tested with Stackless?
> Can't Stackless use its own patches instead?
> 
While this work originated with Stackless, I have gone through the trouble of 
porting it to
CPython, because it is relevant for CPython too.  Multiple "stacks" translates 
to multiple "threads"
in C Python.
 (Confusingly though, the _lsprof.c patch actually has a #ifdef STACKLESS that 
I forgot to remove.  Sorry about that)

> 
> I've often heard that cProfile didn't give correct profiling information with
> multiple threads. Is it true?
Yes, and that's the purpose of this submission:  To fix exactly that.


K
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cProfile and threads

[Nick Coghlan]

2010/8/17 Kristján Valur Jónsson :
> Yes, this is intentional.  I didn't want to waste effort on writing 
> documentation
> before having exposed this to you.  Sometimes my good ideas turn out to be
> not so good and end up being rejected.

Cool, I thought it would be something like that. In this case, at
least from where I'm standing, your good idea looks like a good one.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] i18n

[Anders Sandvig]

On Sat, Aug 14, 2010 at 10:18 AM, Jeroen Ruigrok van der Werven
 wrote:
> I doubt you will be able to localize much with regard to the interpreter.
> The only thing that really comes to mind are the error and exception
> messages, but you will never be able to localize the errors themselves. The
> problem there is that if they seek help on international fora for Python,
> other people might have no clue what the (error) message even means.

I think one way to solve this might be to include the original
(English) error message as well as the translation. I've noticed this
is how error messages are handled in localized versions of Oracle, and
although I'm personally annoyed by it, I can see how some people might
find it useful.

For example:

>>> cause.error()
Traceback (most recent call last):
File "", line 1, in 
NameError: name 'cause' is not defined

localized to Norwegian, could become:

>>> cause.error()
Tilbakesporing (nyeste kall sist):
Fil "", linje 1, i 
NameError: navn 'cause' er ikke definert (name 'cause' is not defined)

I think translating the actual error text would make sense, but I'm
not so sure about localizing the traceback output itself...


Anders
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dev FAQ update request for svnmerge

[Steve Holden]

On 8/17/2010 1:45 AM, Senthil Kumaran wrote:
> On Tue, Aug 17, 2010 at 03:08:31PM +1000, Nick Coghlan wrote:
>> Could someone who knows how it is currently set up suggest a
>> correction to the dev FAQ for svnmerge usage?
> 
> 
> 2.26   How do I merge between branches?
> 
> All development occurs under the py3k branch and bug fixes are expected to be

All development occurs under the py3k branch. Bug fixes (and ONLY bug
fixes: no new features should be added to micro releases) should be

> merged into two additional branches, which are release2x-maint and
> release3x-maint.
> 
> Assuming a change is committed into py3k as revision 0001, you
> merge into the release2x-maint by doing:
> 
> # In the release2x-maint branch checkout.
> svnmerge.py merge -r 0001
> svn commit -F svnmerge-commit-message.txt  # r0002
> 
> # In a release3x-maint checkout.
> svnmerge.py merge -r 0001
> svn commit -F svnmerge-commit-message.txt  # r0003
> 
> 
> # Optional
> 
> In rare situations where you want to backport a security fix or a
> documentation fix into release26-maint branch: 
> 
> #In the release26-maint checkout.
> 
> svnmerge merge -S /python/branches/release27-maint -r0002
> svn commit -F svnmerge-commit-message.txt  # r0004
> 
> 
> 
regards
 Steve
-- 
Steve Holden   +1 571 484 6266   +1 800 494 3119
DjangoCon US September 7-9, 2010http://djangocon.us/
See Python Video!   http://python.mirocommunity.org/
Holden Web LLC http://www.holdenweb.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cProfile and threads

[Daniel Stutzbach]

2010/8/17 Kristján Valur Jónsson 

> These patches are the result of work that we have done in profiling
> Stackless Python server applications at runtime, but they apply just as well
> to C Python.
>
> The first patch makes _lsprof, the engine behind cProfile, multi-stack
> aware.  This allows the same cProfiler.Profile() instance to be active on
> multiple python threads and still meaningful information is gathered.
>

+1


> The second patch allows to set the trace/profile function in python *
> globally*, so that all threads are affected.  This is essential if you
> want to take a profililng snapshot of a running application.
>

+1
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dev FAQ update request for svnmerge

[Terry Reedy]

On 8/17/2010 1:45 AM, Senthil Kumaran wrote:

On Tue, Aug 17, 2010 at 03:08:31PM +1000, Nick Coghlan wrote:

Could someone who knows how it is currently set up suggest a
correction to the dev FAQ for svnmerge usage?



2.26   How do I merge between branches?

All development occurs under the py3k branch and bug fixes are expected to be
merged into two additional branches, which are release2x-maint and


That can and now should specifically be release27_maint as 26 is now 
effectively closed to routine bug fixes.



release3x-maint.

Assuming a change is committed into py3k as revision 0001, you
merge into the release2x-maint by doing:

# In the release2x-maint branch checkout.
svnmerge.py merge -r 0001
svn commit -F svnmerge-commit-message.txt  # r0002

# In a release3x-maint checkout.
svnmerge.py merge -r 0001
svn commit -F svnmerge-commit-message.txt  # r0003


# Optional

In rare situations where you want to backport a security fix or a
documentation fix into release26-maint branch:


I think the only doc fix would be part of a security fix, so no need to 
mention separately I think.


#In the release26-maint checkout.

svnmerge merge -S /python/branches/release27-maint -r0002
svn commit -F svnmerge-commit-message.txt  # r0004






--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 2.6.6 release candidate 2 now available.

[Barry Warsaw]

Hello fellow Python enthusiasts,

The source tarballs and Windows installers for the second (and hopefully last)
Python 2.6.6 release candidate is now available:

http://www.python.org/download/releases/2.6.6/

We've had a handful of important fixes since rc1, and of course a huge number
of bugs have been fixed since 2.6.5, with the full NEWS file available here:

http://www.python.org/download/releases/2.6.6/NEWS.txt

We would love it if you can download, install, and test this version with your
favorite projects and on your favorite platforms.  We expect to release Python
2.6.6 final on August 24, 2010.

Please note that with the release of Python 2.7 final on July 3, 2010, and in
accordance with Python policy, Python 2.6.6 is the last scheduled bug fix
maintenance release of the 2.6 series.  Because of this, your testing of this
release candidate will help immensely.  We plan on continuing to support
source-only security fixes in Python 2.6 for the next five years.

My thanks go out to everyone who has contributed with code, testing and bug
tracker gardening for Python 2.6.6.  The excellent folks on #python-dev are
true Pythonic heros.

Enjoy,
-Barry
(on behalf of the Python development community)


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 3149 - updated patch

[Barry Warsaw]

I've re-merged the py3k trunk to my PEP 3149 branch and uploaded a new diff.
For reference, here's the PEP:

http://www.python.org/dev/peps/pep-3149/

and the tracker issue:

http://bugs.python.org/issue9193

along with the updated patch:

http://bugs.python.org/file18558/pep3149.txt

and the Launchpad branch:

https://code.edge.launchpad.net/~barry/python/sovers

I'd like to know if anybody has additional feedback, suggestions, or
objections that you think still need to be addressed.  You can follow up here
or in the tracker.

Now that 2.6.6 is almost off my plate, I'd like to bring this PEP to its
(hopefully successful) conclusion.

Cheers,
-Barry



signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r84166 - python/branches/py3k/Misc/developers.txt

[Benjamin Peterson]

2010/8/17 martin.v.loewis :
> Author: martin.v.loewis
> Date: Wed Aug 18 00:58:42 2010
> New Revision: 84166
>
> Log:
> Add Ask Solem.
>
>
> Modified:
>   python/branches/py3k/Misc/developers.txt
>
> Modified: python/branches/py3k/Misc/developers.txt
> ==
> --- python/branches/py3k/Misc/developers.txt    (original)
> +++ python/branches/py3k/Misc/developers.txt    Wed Aug 18 00:58:42 2010
> @@ -20,6 +20,10 @@
>  Permissions History
>  ---
>
> +- Ask Solem was given commit access on Aug 17 2010 by MvL,
> +  on recommendation by Jesse Noller, for work on the subprocess
> +  library.

IIRC it was multiprocessing.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com