Re: [IronPython] IP v8 Hosting question...
You would need to do the following. engine.Sys.modules.Remove(myModuleName); Since engine.Sys.modules is a regular Python dict, it is indexed using strings, not SymbolIDs. Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE-11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 8:15 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Sweet, I'm naming the modules which probably explains where I'm running into issues... everything I use (that I care about collecting during runtime) are snippets... I don't imagine anything I have needs publishing in sys... incidentally to remove a module from sys, do I just need to do something like engine.Sys.modules.Remove(SymbolTable.StringToId(myModuleName)); ? -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 3:05 p.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We publish the ModuleScope to sys.modules if you do scope = new ModuleScope(some name); We don't publish the ModuleScope if you do scope = new ModuleScope(); Note that we are thinking of changing this to let the user explicitly chose whether the module is published to sys.modules. Code compiled by all the PythonEngine APIs should generate collectible code except for: If you pass in ExecutionOptions.EnableDebugging. If you use ExecuteFileOptimized (formerly known as RunFile) You should not need to worry about memory leaks. I will add a test case to ensure that this stays this way. Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Dino Viehland Sent: Tuesday, June 27, 2006 6:47 PM To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... Two things could be causing the increasing memory usage: 1. Instances of modules hanging around. If you're creating new modules those are getting published in sys.modules, then you'll keep these things around (you could manually remove them from sys.modules and they should be collected after that). 2. Code compiled in a module will not be collectible - it will get compiled into a type in-memory that won't get unloaded until app domain unload. Code compiled stand-alone (e.g. from Python doing exec or eval) will be collectible and go away when the GC kicks in. In general this should come down to RunFile* will not be collectible, and everything else is. We do have a mode, -X:GenerateAsSnippets, which forces everything to be generated in mode #2. But there are significant performance benefits to having the non-collectible code. Everything else should be collectible. From: [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 6:19 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2
Re: [IronPython] IP v8 Hosting question...
Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006 5:03 PM To: 'Discussion of IronPython' Subject: [IronPython] IP v8 Hosting question... I'm looking for the most appropriate way to scope variables for my expression to evaluate that come from an external context (ie. Outside of the python engine) - So far I have something working, using ModuleScope - but is this the way I should be doing it? PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope1.SetGlobal(id, hello); ModuleScope scope2 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Using this method I can evaluate around 2000 expressions a second, which is more the adequate for what I'm doing - but I'm concerned about memory consumption... Chez, - Alex ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] IP v8 Hosting question...
Two things could be causing the increasing memory usage: 1. Instances of modules hanging around. If you're creating new modules those are getting published in sys.modules, then you'll keep these things around (you could manually remove them from sys.modules and they should be collected after that). 2. Code compiled in a module will not be collectible - it will get compiled into a type in-memory that won't get unloaded until app domain unload. Code compiled stand-alone (e.g. from Python doing exec or eval) will be collectible and go away when the GC kicks in. In general this should come down to RunFile* will not be collectible, and everything else is. We do have a mode, -X:GenerateAsSnippets, which forces everything to be generated in mode #2. But there are significant performance benefits to having the non-collectible code. Everything else should be collectible. From: [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 6:19 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006 5:03 PM To: 'Discussion of IronPython' Subject: [IronPython] IP v8 Hosting question... I'm looking for the most appropriate way to scope variables for my expression to evaluate that come from an external context (ie. Outside of the python engine) - So far I have something working, using ModuleScope - but is this the way I should be doing it? PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope1.SetGlobal(id, hello); ModuleScope scope2 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Using this method I can evaluate around 2000 expressions a second, which is more the adequate for what I'm doing - but I'm concerned about memory consumption... Chez, - Alex ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users
Re: [IronPython] IP v8 Hosting question...
We publish the ModuleScope to sys.modules if you do scope = new ModuleScope(some name); We don't publish the ModuleScope if you do scope = new ModuleScope(); Note that we are thinking of changing this to let the user explicitly chose whether the module is published to sys.modules. Code compiled by all the PythonEngine APIs should generate collectible code except for: If you pass in ExecutionOptions.EnableDebugging. If you use ExecuteFileOptimized (formerly known as RunFile) You should not need to worry about memory leaks. I will add a test case to ensure that this stays this way. Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE-11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dino Viehland Sent: Tuesday, June 27, 2006 6:47 PM To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... Two things could be causing the increasing memory usage: 1. Instances of modules hanging around. If you're creating new modules those are getting published in sys.modules, then you'll keep these things around (you could manually remove them from sys.modules and they should be collected after that). 2. Code compiled in a module will not be collectible - it will get compiled into a type in-memory that won't get unloaded until app domain unload. Code compiled stand-alone (e.g. from Python doing exec or eval) will be collectible and go away when the GC kicks in. In general this should come down to RunFile* will not be collectible, and everything else is. We do have a mode, -X:GenerateAsSnippets, which forces everything to be generated in mode #2. But there are significant performance benefits to having the non-collectible code. Everything else should be collectible. From: [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 6:19 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006 5:03 PM To: 'Discussion of IronPython' Subject: [IronPython] IP v8 Hosting question... I'm looking for the most appropriate way to scope variables for my expression to evaluate that come from an external context (ie. Outside of the python engine) - So far I have something working, using ModuleScope - but is this the way I should be doing it? PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope1.SetGlobal(id, hello); ModuleScope scope2 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope2.SetGlobal(id, goodbye
Re: [IronPython] IP v8 Hosting question...
Sweet, I'm naming the modules which probably explains where I'm running into issues... everything I use (that I care about collecting during runtime) are snippets... I don't imagine anything I have needs publishing in sys... incidentally to remove a module from sys, do I just need to do something like engine.Sys.modules.Remove(SymbolTable.StringToId(myModuleName)); ? -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 3:05 p.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We publish the ModuleScope to sys.modules if you do scope = new ModuleScope(some name); We don't publish the ModuleScope if you do scope = new ModuleScope(); Note that we are thinking of changing this to let the user explicitly chose whether the module is published to sys.modules. Code compiled by all the PythonEngine APIs should generate collectible code except for: If you pass in ExecutionOptions.EnableDebugging. If you use ExecuteFileOptimized (formerly known as RunFile) You should not need to worry about memory leaks. I will add a test case to ensure that this stays this way. Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Dino Viehland Sent: Tuesday, June 27, 2006 6:47 PM To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... Two things could be causing the increasing memory usage: 1. Instances of modules hanging around. If you're creating new modules those are getting published in sys.modules, then you'll keep these things around (you could manually remove them from sys.modules and they should be collected after that). 2. Code compiled in a module will not be collectible - it will get compiled into a type in-memory that won't get unloaded until app domain unload. Code compiled stand-alone (e.g. from Python doing exec or eval) will be collectible and go away when the GC kicks in. In general this should come down to RunFile* will not be collectible, and everything else is. We do have a mode, -X:GenerateAsSnippets, which forces everything to be generated in mode #2. But there are significant performance benefits to having the non-collectible code. Everything else should be collectible. From: [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 6:19 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006 5
