[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Also attaching massif output. ** Attachment added: "zg-massif.txt" https://bugs.launchpad.net/zeitgeist/+bug/624310/+attachment/2916667/+files/zg-massif.txt ** Changed in: zeitgeist Status: In Progress => Invalid ** Changed in: zeitgeist-extensions Status: New => Invalid ** Ch

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

After some very thorough analysis, I've concluded that zeitgeist is not at blame here. If my understanding of the kernel slab allocator is correct, free() doesn't actually free the memory from a program's space until the kernel decides it needs reclaimed. As such, zeitgeist-daemon is not at fault

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Changed in: zeitgeist (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extensions. https://bugs.launchpad.net/bugs/624310 Title: Large requests increase memory usage conside

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Also affects: zeitgeist (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extensions. https://bugs.launchpad.net/bugs/624310 Title: Large requests increase memory

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Changed in: zeitgeist Importance: Undecided => Low ** Changed in: zeitgeist Status: Confirmed => In Progress ** Changed in: zeitgeist Milestone: 0.8.1 => None -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zei

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Branch linked: lp:~stephanieodok/zeitgeist/generator -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extensions. https://bugs.launchpad.net/bugs/624310 Title: Large requests increase memory usage considerably Sta

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

for the last 2 scripts make sure you change the path to the DB -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extensions. https://bugs.launchpad.net/bugs/624310 Title: Large requests increase memory usage considerab

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

and here is the vala test ?field.comment=and here is the vala test ** Attachment added: "test.vala" https://bugs.launchpad.net/zeitgeist/+bug/624310/+attachment/2153966/+files/test.vala -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the regis

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

also here are is the python test script ** Attachment added: "test.py" https://bugs.launchpad.net/zeitgeist/+bug/624310/+attachment/2153965/+files/test.py -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extensions.

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

So i created a shitty extension too see how much memory consumption happens before things are sent over dbus. The idea is I ask for ALL events directly from the engine, 10 seconds after zeitgeist starts Here are my observations find_eventids: starts with 16.8 MB and end up with 25.4 MB find_eve

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Here's an easy way to check this: [rainct, zeitgeist-trunk]$ ps -u $USERNAME -o comm,rss | grep zeitgeist-dae zeitgeist-daemo 13948 [rainct, zeitgeist-trunk]$ ./reproducer.py 96466 [rainct, zeitgeist-trunk]$ ps -u $USERNAME -o comm,rss | grep zeitgeist-dae zeitgeist-daemo 31032 ** Attachment add

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Changed in: zeitgeist Milestone: 0.8.0 => 0.8.1 -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extensions. https://bugs.launchpad.net/bugs/624310 Title: Large requests increase memory usage considerably Stat

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Before we freak about about __slots__ and structs I think we need hard evidence that we lose all this memory to Events and Subjects. Attaching some profiler to the Python VM should sort this out quickly. And why doesn't Ubuntu show the same symptoms? -- You received this bug notification because

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Based on what I posted above I noticed that in our datamodel.py Event and Subject both inherit from list which could cause the excessive memory usage. I think we should look into re-implementing them as objects with all the necessary additions like __getitem__ and __setitem__ to avoid breaking th

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

I had a brief discussion with the guys on #python --- can u explain to me why inheriting from object with __slots__ uses less memory than inheriting from list? make it for dummies so i can explain it to my team seiflotfy: do you know what a struct looks like in memory? verte, nope seiflot

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

So using fedora 32-bit Zeitgeist consumes about 90 MB after a few hours I am using Synapse and GNOME Shell. -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extensions. https://bugs.launchpad.net/bugs/624310 Title: La

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Is there a posibility this bug is related with this one in Unity?: https://bugs.launchpad.net/ubuntu/+source/zeitgeist/+bug/757727 If so, is it being worked on? -- You received this bug notification because you are a member of Zeitgeist Extensions, which is the registrant for Zeitgeist Extension

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Changed in: zeitgeist Milestone: None => 0.8.0 -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/624310 Title: Large requests increase memory usage considerably Status

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Changed in: zeitgeist Milestone: 0.7.0 => None -- Large requests increase memory usage considerably https://bugs.launchpad.net/bugs/624310 You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. Status in Zeitgei

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Ok I don't think this issue will be tackled for 0.6 now so postponing for 0.7 ** Changed in: zeitgeist Milestone: 0.6 => 0.7 -- Large requests increase memory usage considerably https://bugs.launchpad.net/bugs/624310 You received this bug notification because you are a member of Zeitgeist Fr

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

I've made some tests tonight, on an activitylog with 18k random web history events. The in memory size of the daemon is about 7MiB on startup. After running FindEvents(TimeRange.until_now(), [],StorageState.Any, 0,1) (same query is used by sezen *over and over again*) the daemon's memory consump

Re: [Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

OK now it all makes sense again. Sorry guys. On another note I found a nifty tool for memory profiling. http://guppy-pe.sourceforge.net/ Let's see what we can do with that Cheers Seif On Sat, Oct 2, 2010 at 11:13 AM, Markus Korn wrote: > As far as I remember, th

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

As far as I remember, the main reason why we dropped this iterator thing being part of our API redisign efforts a year ago was that there is no easy (and performant) way to do batched database queries. The problem is: we don't have a 'stable' order of events, just think of a query which returns

Re: [Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

I know yet the iteration is handeled then from the client side... I am suggesting a way to do it completely from the engine side :) On Fri, Oct 1, 2010 at 10:00 PM, Siegfried Gevatter wrote: > Seif, that's what we have FindEventIds for. > > -- > Large requests increase memory usage considerably >

Re: [Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Seif, that's what we have FindEventIds for. -- Large requests increase memory usage considerably https://bugs.launchpad.net/bugs/624310 You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. Status in Zeitgeist Framewo

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Thinking of it I somehow remember the iterator idea. Maybe this is something we should really look at. An extension with extra cursors for each iterator could be an idea, with an 1) get_iterator(time_range, event_template, result_type, transaction_size) which returns an iterator object with nex

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Changed in: zeitgeist Milestone: 0.5.1 => 0.6 ** Changed in: zeitgeist Importance: High => Undecided -- Large requests increase memory usage considerably https://bugs.launchpad.net/bugs/624310 You received this bug notification because you are a member of Zeitgeist Framework Team, whic

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Markus is already working on a profiling tool for this... https://code.edge.launchpad.net/~zeitgeist-extensions/zeitgeist-extensions/trunk ** Also affects: zeitgeist-extensions Importance: Undecided Status: New -- Large requests increase memory usage considerably https://bugs.launchpad

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

Come to think of it - the fact that Sezen requests 11k events also suggests that there is something wrong in the design scope of either Zeitgeist or Sezen. Before we loose our selves on an optimization cache-pruning spree (which may very well consume a lot of development time for some very small g

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

I'd actually guess that the memory was allocated inside sqlite rather than python-dbus, but that's hard to tell without really profiling it. -- Large requests increase memory usage considerably https://bugs.launchpad.net/bugs/624310 You received this bug notification because you are a member of Z

[Zeitgeist] [Bug 624310] Re: Large requests increase memory usage considerably

** Changed in: zeitgeist Status: New => Confirmed ** Changed in: zeitgeist Importance: Undecided => High ** Changed in: zeitgeist Milestone: None => 0.5.1 -- Large requests increase memory usage considerably https://bugs.launchpad.net/bugs/624310 You received this bug notification