Re: Pickle caching objects?

2019-12-16 Thread Peter J. Holzer
On 2019-12-16 08:56:26 +0100, dieter wrote: > Note also that Python memeory management is quite elaborate: > not every memory block is immediately obtained from and released > to the operating system: Python has its own memory management > data structures (to fill the gap between the fine grained

Re: Pickle caching objects?

2019-12-16 Thread Inada Naoki
Would you try the pull request in this issue? https://bugs.python.org/issue36694 I'm not sure this issue is relating to you because I don't know about your data. Regards, On Sun, Dec 1, 2019 at 10:14 AM José María Mateos wrote: > > Hi, > > I just asked this question on the IRC channel but

Re: Pickle caching objects?

2019-12-15 Thread dieter
José María Mateos writes: > I just asked this question on the IRC channel but didn't manage to get > a response, though some people replied with suggestions that expanded > this question a bit. > > I have a program that has to read some pickle files, perform some > operations on them, and then

Re: Pickle caching objects?

2019-12-01 Thread José María Mateos
On Sun, Dec 01, 2019 at 12:26:15PM +1100, Chris Angelico wrote: I can't answer your question authoritatively, but I can suggest a place to look. Python's memory allocator doesn't always return memory to the system when the objects are freed up, for various reasons including the way that memory

Re: Pickle caching objects?

2019-11-30 Thread Richard Damon
On 11/30/19 5:05 PM, José María Mateos wrote: > Hi, > > I just asked this question on the IRC channel but didn't manage to get > a response, though some people replied with suggestions that expanded > this question a bit. > > I have a program that has to read some pickle files, perform some >

Re: Pickle caching objects?

2019-11-30 Thread Chris Angelico
On Sun, Dec 1, 2019 at 12:15 PM José María Mateos wrote: > print("Memory usage:", psutil.Process(os.getpid()).memory_info().rss) > > Notice that memory usage increases noticeably specially on files 4 and > 5, the biggest ones, and doesn't come down as I would expect it to. But > the

Pickle caching objects?

2019-11-30 Thread José María Mateos
Hi, I just asked this question on the IRC channel but didn't manage to get a response, though some people replied with suggestions that expanded this question a bit. I have a program that has to read some pickle files, perform some operations on them, and then return. The pickle objects I