[issue401229] Optional memory profiler

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401229] Optional memory profiler

[admin]

Change by admin :


--
github: None -> 32959

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Fil v0.11.0, a memory profiler for scientists and data scientists

[Itamar Turner-Trauring]

Your code reads some data, processes it, and uses too much memory. In order to 
reduce memory usage, you need to figure out:

 1. Where peak memory usage is, also known as the high-water mark.
 2. What code was responsible for allocating the memory that was present at 
that peak moment.
That's exactly what Fil will help you find.

Fil an open source memory profiler designed for data processing applications 
written in Python, and includes native support for Jupyter. It is designed to 
be high-performance and easy to use.
At the moment it only runs on Linux and macOS.

You can learn more about Fil at https://pythonspeed.com/fil or on GitHub at 
https://github.com/pythonspeed/filprofiler/.

v0.11 includes performance improvements and less intrusive behavior under 
Jupyter.

Fil vs. other Python memory tools

There are two distinct patterns of Python usage, each with its own source of 
memory problems.

In a long-running server, memory usage can grow indefinitely due to memory 
leaks. That is, some memory is not being freed.

 * If the issue is in Python code, tools like `tracemalloc` 
<https://docs.python.org/3/library/tracemalloc.html> and Pympler 
<https://pypi.org/project/Pympler/> can tell you which objects are leaking and 
what is preventing them from being leaked.
 * If you're leaking memory in C code, you can use tools like Valgrind 
<https://valgrind.org/>.
Fil, however, is not aimed at memory leaks, but at the other use case: data 
processing applications. These applications load in data, process it somehow, 
and then finish running.

The problem with these applications is that they can, on purpose or by mistake, 
allocate huge amounts of memory. It might get freed soon after, but if you 
allocate 16GB RAM and only have 8GB in your computer, the lack of leaks doesn't 
help you.

Fil will therefore tell you, in an easy to understand way:

 1. Where peak memory usage is, also known as the high-water mark.
 2. What code was responsible for allocating the memory that was present at 
that peak moment.
 3. This includes C/Fortran/C++/whatever extensions that don't use Python's 
memory allocation API (`tracemalloc` only does Python memory APIs).
___
Python-announce-list mailing list -- python-announce-list@python.org
To unsubscribe send an email to python-announce-list-le...@python.org
https://mail.python.org/mailman3/lists/python-announce-list.python.org/
Member address: arch...@mail-archive.com

memprof - a memory profiler for Python

[dana . develop]

Hello,

I'd like to announce memprof, a memory profiler for Python:

http://jmdana.github.io/memprof/

memprof logs and plots the memory usage of all the variables during the 
execution of the decorated methods.

The source code is available in GitHub:

https://github.com/jmdana/memprof

And the mailing list is here:

http://groups.google.com/group/memprof

The project is licensed under GPLv3 and the current version is 0.2.2.

Please don't hesitate to contact me through the mailing list and/or GitHub 
(e.g. opening a new issue). 

I hope you find it useful.
Jose.



-- 
http://mail.python.org/mailman/listinfo/python-announce-list

Support the Python Software Foundation:
http://www.python.org/psf/donations/

Memory Profiler

[Dave]

Hi,

Is there any memory profiler for Python programs? I
would like to see how much memory objects take up,
cache misses, etc. I tried to patch PySizer, but got a
build error (Error 138). 

Thanks!

__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Memory Profiler

[Grig Gheorghiu]

See

http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy#MiscellaneousPythonTestingTools

In particular, PySizer and HeapPy might be what you're looking for. I
can't say for sure, since I haven't used these tools myself.

Grig

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Memory Profiler

[Stephen Kellett]

In message [EMAIL PROTECTED], Dave 
[EMAIL PROTECTED] writes
Is there any memory profiler for Python programs? I

Python Memory Validator

http://www.softwareverify.com/pythonMemoryValidator/index.html

Stephen
-- 
Stephen Kellett
Object Media Limitedhttp://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: memory profiler?

[John Reese]

On Wed, 20 Apr 2005 23:06:51 -0400, Peter Hansen [EMAIL PROTECTED] wrote:
 John Reese wrote:
 Is there a memory or heap profiler for python programs?  So that, for
 example, if a program was bloating over time I could see how many of
 each object there were and maybe even where the references were?

 The gc module has a variety of helpful features
 like that.

 -Peter

In particular, get_objects() and get_referrers().  Good point.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: memory profiler?

[Diez B. Roggisch]

Peter Hansen wrote:

 John Reese wrote:
 Is there a memory or heap profiler for python programs?  So that, for
 example, if a program was bloating over time I could see how many of
 each object there were and maybe even where the references were?
 
 The gc module has a variety of helpful features
 like that.

I created a qt-based memory analyzer on top of the gc module. I plan to
release it soon - it has been helpful to me for a few projects. Its a
client-server application that collects data through an in-process running
thread that delivers object histograms. The client fetches these and allows
to display the development of objects based various criteria over the time.
It can also record these data for offline-analysis. So far the in-proccess
threads communicate their histograms either by pyro or xmlrpc servers.  

Unfortunately debugging ZOPE with the xmlrpc server lead to a mem-leak
itself :( - so currently I'm no confident enough to release it. 
-- 
Regards,

Diez B. Roggisch
-- 
http://mail.python.org/mailman/listinfo/python-list

memory profiler?

[John Reese]

Good afternoon, ha ha ha!

Is there a memory or heap profiler for python programs?  So that, for
example, if a program was bloating over time I could see how many of
each object there were and maybe even where the references were?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: memory profiler?

[Peter Hansen]

John Reese wrote:
Is there a memory or heap profiler for python programs?  So that, for
example, if a program was bloating over time I could see how many of
each object there were and maybe even where the references were?
The gc module has a variety of helpful features
like that.
-Peter
--
http://mail.python.org/mailman/listinfo/python-list