I'm well aware that changing the API is a pain for everyone and thus it might be required to introduce another method (i.e. conn.get_event_conduit()) instead of changing the behavior of conn.event_conduit().
Nevertheless, it's awesome that you're already into fixing this issue and I hope to hear from you soon. Cheers -----Ursprüngliche Nachricht----- Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 7. November 2014 15:24 An: firebird-python@yahoogroups.com Betreff: Re: AW: AW: AW: [firebird-python] Memory leak when using events Hi, I have to think more about implementation details and consequences but I like your solution. If no problem arises I'm going to make a minor release with this fix next week. We're also working with Vlad on "Unknown error" issue which is most likely a bug in FB 2.5 engine. best regards Pavel Cisar IBPhoenix Dne 7.11.2014 v 13:57 Dominik Psenner dominik.psen...@topcontrol.it [firebird-python] napsal(a): > Bug or pitfall does not matter since there's no way to work around the issue. > The problem is mainly caused by the invoke of self.__wait_for_events() in the > constructor, isn't it? So what was if you changed the API such that one would: > > eventNames = ['EVENT'] > handle = conn.event_conduit(eventNames) > try: > # begin listening for events > handle.begin() > # wait until events are there > events = handle.wait(timeout) > # process events > except: > pass #or whatever would be appropriate > finally: > handle.close() > > or maybe even fancier by providing an __enter__ and __exit__ method: > > with conn.event_conduit(eventNames) as handle: > events = handle.wait(timeout) > # process events > > Where the __enter__ method called self.begin() and __exit__ called > self.close(). This way the constructor would no longer have to do fancy > things that could break. Handling exceptional cases would after all be left > up to who uses the API. > > Cheers > > -----Ursprüngliche Nachricht----- > Von: firebird-python@yahoogroups.com > [mailto:firebird-python@yahoogroups.com] > Gesendet: Freitag, 7. November 2014 13:33 > An: firebird-python@yahoogroups.com > Betreff: Re: AW: AW: [firebird-python] Memory leak when using events > > Hi, > > thanks for the app, it helped to look into this issue. However, after deep > look I start to think that this is actually not a bug, just pitfall. The > problem is that when some problem arises while (re)registering interest in > event notifications (isc_que_events API call), there is no good way how to > handle it and continue with event processing using given conduit (threading > issues aside). So exception has to be propagated up to the application level > and handled there properly, i.e. EventConduit has to be considered invalid > and discarded. > > I can do next things to improve the situation: > > 1. Use weakref.proxy to queue from EventBlock, so circular reference through > queue content would be broken. > 2. Provide access to internal events dictionary in EventConduit (it would > return a copy like wait does), so event counts accumulated so far could be > retrieved in case of failure. > > The side issue is that events seems to be quite fragile (isc_que_events > shouldn't fail so easily and especially with no good reason like "Unknown ISC > error" I'm getting from your sample app), and I don't know why. I'll consult > that with core developers and report here. > > best regards > Pavel Cisar > IBPhoenix > > Dne 7.11.2014 v 12:43 Dominik Psenner dominik.psen...@topcontrol.it > [firebird-python] napsal(a): >> Apologies, I should have explained that better. :-) The good thing (or bad?) >> is, that the database can be any firebird database. It must not even raise >> events. Take any database you want and modify the first few variables in the >> __main__ section such that it can connect to a database you have on your >> computer. You could also adjust the timeout and minTimeout to smaller values >> to get the results faster. >> >> Cheers >> >> -----Ursprüngliche Nachricht----- >> Von: firebird-python@yahoogroups.com >> [mailto:firebird-python@yahoogroups.com] >> Gesendet: Freitag, 7. November 2014 12:28 >> An: firebird-python@yahoogroups.com >> Betreff: Re: AW: [firebird-python] Memory leak when using events >> >> Hi, >> >> Well, the script is nice but I can't use it without the database it >> uses :) >> >> best regards >> Pavel Cisar >> IBPhoenix >> >> Dne 7.11.2014 v 12:19 Dominik Psenner dominik.psen...@topcontrol.it >> [firebird-python] napsal(a): >>> Hi Pavel, >>> >>> Happy to read you. Sure thing, come up with a fix and I'll do my best. >>> However, I attach a little python script to this email that does reproduce >>> the issue within a few cycles. This should ease your job a little. :-) For >>> the records, I have reported this issue as bug #PYFB-43. >>> >>> Cheers and read you back soon, >>> Dominik >>> >>> -----Ursprüngliche Nachricht----- >>> Von: firebird-python@yahoogroups.com >>> [mailto:firebird-python@yahoogroups.com] >>> Gesendet: Freitag, 7. November 2014 12:05 >>> An: firebird-python@yahoogroups.com >>> Betreff: Re: [firebird-python] Memory leak when using events >>> >>> Hi, >>> >>> This is definitely a bug in FDB event handling, but tricky one. It seems to >>> me that root of this problem is that exception raised in >>> EventBlock.__wait_for_events is not properly handled in upper levels, which >>> in turn causes accumulation of blocks in queue. The circular reference you >>> pointed out just made it visible. So fixing the circular reference with >>> weakref.proxy will probably not fix the problem. I'm working on proper fix, >>> but it's hard to test. Can you help me with testing if I'll create a patch? >>> >>> best regards >>> Pavel Cisar >>> IBPhoenix >>> >>> Dne 7.11.2014 v 08:56 Dominik Psenner dominik.psen...@topcontrol.it >>> [firebird-python] napsal(a): >>>> Good morning, >>>> >>>> as written yesterday we have recently implemented a way to react on >>>> database events in our python based program. But we are facing several >>>> outages with that implementation. First we encountered at least three >>>> errors when invoking event_conduit(events), namely: >>>> >>>> * Error reading data from the connection. >>>> >>>> * Error writing data to the connection. >>>> >>>> * unknown ISC error 0 >>>> >>>> Then I had the suspicion that the private memory was increasing and never >>>> dropping, thus I investigated a little further and used the dummy program >>>> from yesterday and let it run overnight. It produced and consumed about >>>> 1.250.000 events and the process memory raised from 27mb to roughly 400mb. >>>> Luckily I was wise enough to start the test script with pdb and this is >>>> the outcome: >>>> >>>> A memory leak in the class EventBlock(object), defined in dbcore.py at >>>> line 1687 caused by the cyclic reference introduced in its constructor at >>>> lines 1705 and 1709: >>>> >>>> 1705: self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) >>>> 1709: self.__queue = queue >>>> >>>> The events issue is now officially promoted to be a blocker for us. >>>> Looking at what the code does, it looks mostly like a chicken and egg >>>> problem. Ideas / suggestions? Is this the right place to come up with such >>>> problems (i.e. do the maintainers of the fdb python package read this >>>> mailinglist)? >>>> >>>> Best regards, >>>> Dominik >>>> >>> >>> >>> ------------------------------------ >>> >>> ------------------------------------ >>> >>> >>> ------------------------------------ >>> >>> Yahoo Groups Links >>> >>> >>> >>> >>> >>> >>> ------------------------------------ >>> Posted by: Dominik Psenner <dominik.psen...@topcontrol.it> >>> ------------------------------------ >>> >>> >>> ------------------------------------ >>> >>> Yahoo Groups Links >>> >>> >>> >>> >> >> >> ------------------------------------ >> >> ------------------------------------ >> >> >> ------------------------------------ >> >> Yahoo Groups Links >> >> >> >> >> >> ------------------------------------ >> Posted by: Dominik Psenner <dominik.psen...@topcontrol.it> >> ------------------------------------ >> >> >> ------------------------------------ >> >> Yahoo Groups Links >> >> >> >> > > > ------------------------------------ > > ------------------------------------ > > > ------------------------------------ > > Yahoo Groups Links > > > > > > ------------------------------------ > Posted by: Dominik Psenner <dominik.psen...@topcontrol.it> > ------------------------------------ > > > ------------------------------------ > > Yahoo Groups Links > > > > ------------------------------------ ------------------------------------ ------------------------------------ Yahoo Groups Links