Re: [IronPython] Speed test
I finally achieved some spare time to get MIT's ConceptNet 2.1 to run using IronPython. I decided just to modify some of the files included with the 2.1 distribution and copy them; placing them in another directory to acheive execution in IronPython. Just to get it to run the first time, I created a custom(and not exactly functional. It is actually an IronPython.Runtime.List type object called array) array.py, commented out some zlib lines used by MontyLingua (doesn't seem to effect any results from tests thus far). I ran it using IronPythonConsole.exe beta8 So it appears it takes IronPythonConsole.exe beta8 151.478704 seconds to load ConceptNet 2.1 and its predicate files. It takes IDLE 60.205843 seconds to execute the same calls. One of these days I'll have everything in a presentable manner, and hopefully a nice little MS VS project. Even though it's not as fast, I still believe it is a wonderful achievement the IronPython team has done! * Loading Common Sense Component... predicates_concise_nonkline.txt: Loaded 27000 predicates! predicates_concise_kline.txt: Loaded 174000 predicates! predicates_nonconcise_nonkline.txt: Loaded 348000 predicates! -- took 151.478704 seconds. -- Performing Optimizations... Optimised 129563 of 129563 nodes! ** - *IDLE: Loaded 342000 predicates! predicates_nonconcise_nonkline.txt: Loaded 345000 predicates! predicates_nonconcise_nonkline.txt: Loaded 348000 predicates! -- took 60.205843 seconds. -- -- Later, Joe ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] IP v8 Hosting question...
Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006 5:03 PM To: 'Discussion of IronPython' Subject: [IronPython] IP v8 Hosting question... I'm looking for the most appropriate way to scope variables for my expression to evaluate that come from an external context (ie. Outside of the python engine) - So far I have something working, using ModuleScope - but is this the way I should be doing it? PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope1.SetGlobal(id, hello); ModuleScope scope2 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Using this method I can evaluate around 2000 expressions a second, which is more the adequate for what I'm doing - but I'm concerned about memory consumption... Chez, - Alex ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] IP v8 Hosting question...
Two things could be causing the increasing memory usage: 1. Instances of modules hanging around. If you're creating new modules those are getting published in sys.modules, then you'll keep these things around (you could manually remove them from sys.modules and they should be collected after that). 2. Code compiled in a module will not be collectible - it will get compiled into a type in-memory that won't get unloaded until app domain unload. Code compiled stand-alone (e.g. from Python doing exec or eval) will be collectible and go away when the GC kicks in. In general this should come down to RunFile* will not be collectible, and everything else is. We do have a mode, -X:GenerateAsSnippets, which forces everything to be generated in mode #2. But there are significant performance benefits to having the non-collectible code. Everything else should be collectible. From: [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 6:19 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006 5:03 PM To: 'Discussion of IronPython' Subject: [IronPython] IP v8 Hosting question... I'm looking for the most appropriate way to scope variables for my expression to evaluate that come from an external context (ie. Outside of the python engine) - So far I have something working, using ModuleScope - but is this the way I should be doing it? PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope1.SetGlobal(id, hello); ModuleScope scope2 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Using this method I can evaluate around 2000 expressions a second, which is more the adequate for what I'm doing - but I'm concerned about memory consumption... Chez, - Alex ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ users mailing list users@lists.ironpython.com
Re: [IronPython] IP v8 Hosting question...
We publish the ModuleScope to sys.modules if you do scope = new ModuleScope(some name); We don't publish the ModuleScope if you do scope = new ModuleScope(); Note that we are thinking of changing this to let the user explicitly chose whether the module is published to sys.modules. Code compiled by all the PythonEngine APIs should generate collectible code except for: If you pass in ExecutionOptions.EnableDebugging. If you use ExecuteFileOptimized (formerly known as RunFile) You should not need to worry about memory leaks. I will add a test case to ensure that this stays this way. Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE-11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dino Viehland Sent: Tuesday, June 27, 2006 6:47 PM To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... Two things could be causing the increasing memory usage: 1. Instances of modules hanging around. If you're creating new modules those are getting published in sys.modules, then you'll keep these things around (you could manually remove them from sys.modules and they should be collected after that). 2. Code compiled in a module will not be collectible - it will get compiled into a type in-memory that won't get unloaded until app domain unload. Code compiled stand-alone (e.g. from Python doing exec or eval) will be collectible and go away when the GC kicks in. In general this should come down to RunFile* will not be collectible, and everything else is. We do have a mode, -X:GenerateAsSnippets, which forces everything to be generated in mode #2. But there are significant performance benefits to having the non-collectible code. Everything else should be collectible. From: [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 6:19 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006 5:03 PM To: 'Discussion of IronPython' Subject: [IronPython] IP v8 Hosting question... I'm looking for the most appropriate way to scope variables for my expression to evaluate that come from an external context (ie. Outside of the python engine) - So far I have something working, using ModuleScope - but is this the way I should be doing it? PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope1.SetGlobal(id, hello); ModuleScope scope2 = ModuleScope.MakeScopeForFunction(new PythonModule(Junk, new Dict(), engine.Sys)); scope2.SetGlobal(id, goodbye);
Re: [IronPython] IP v8 Hosting question...
Sweet, I'm naming the modules which probably explains where I'm running into issues... everything I use (that I care about collecting during runtime) are snippets... I don't imagine anything I have needs publishing in sys... incidentally to remove a module from sys, do I just need to do something like engine.Sys.modules.Remove(SymbolTable.StringToId(myModuleName)); ? -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 3:05 p.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We publish the ModuleScope to sys.modules if you do scope = new ModuleScope(some name); We don't publish the ModuleScope if you do scope = new ModuleScope(); Note that we are thinking of changing this to let the user explicitly chose whether the module is published to sys.modules. Code compiled by all the PythonEngine APIs should generate collectible code except for: If you pass in ExecutionOptions.EnableDebugging. If you use ExecuteFileOptimized (formerly known as RunFile) You should not need to worry about memory leaks. I will add a test case to ensure that this stays this way. Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Dino Viehland Sent: Tuesday, June 27, 2006 6:47 PM To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... Two things could be causing the increasing memory usage: 1. Instances of modules hanging around. If you're creating new modules those are getting published in sys.modules, then you'll keep these things around (you could manually remove them from sys.modules and they should be collected after that). 2. Code compiled in a module will not be collectible - it will get compiled into a type in-memory that won't get unloaded until app domain unload. Code compiled stand-alone (e.g. from Python doing exec or eval) will be collectible and go away when the GC kicks in. In general this should come down to RunFile* will not be collectible, and everything else is. We do have a mode, -X:GenerateAsSnippets, which forces everything to be generated in mode #2. But there are significant performance benefits to having the non-collectible code. Everything else should be collectible. From: [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Tuesday, June 27, 2006 6:19 PM To: 'Discussion of IronPython' Subject: Re: [IronPython] IP v8 Hosting question... Yeah, I arrived at this yesterday after some experimentation :) I've since dumped the idea of setting global variables (as it's obviously not thread safe for multiple threads evaluating the same expression) and now generate methods for the expression i.e. For an expression: Hello + Message I would end up generating def generated_method_1(Message): return Hello + Message Then execute that against a newly created module, then evaluate generated_method_1, casting it to a PythonFunction and Call that with the array of values that make up my context... works well now and suits my needs perfectly as I can wrap up iron python to work with our standard scripting interfaces. When I no longer need these methods I set their value to None, but I'm wondering if generating methods like this will incur overhead I can't get back (ie. will my apps memory consumption just keep growing, until the python engine itself is shut down?) It certainly seems a little like that when I've been unit testing, but perhaps that's a side effect of the NUnit environment? Chez, - Alex -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Shri Borde Sent: Wednesday, 28 June 2006 9:55 a.m. To: Discussion of IronPython Subject: Re: [IronPython] IP v8 Hosting question... We will are taking a look at the API and will be changing it a bit. The current way of doing this as follows: PythonEngine engine = new PythonEngine(); SymbolId id = SymbolTable.StringToId(Message); ModuleScope scope1 = new ModuleScope(Junk1); scope1.SetGlobal(id, hello); ModuleScope scope2 = new ModuleScope(Junk2); scope2.SetGlobal(id, goodbye); Assert.AreEqual(hello, engine.Evaluatestring(Message, scope1, ExecutionOptions.Default)); Assert.AreEqual(goodbye, engine.Evaluatestring(Message, scope2, ExecutionOptions.Default)); Do you want to help develop Dynamic languages on CLR? (http://members.microsoft.com/careers/search/details.aspx?JobID=6D4754DE- 11F0-45DF-8B78-DC1B43134038) -Original Message- From: [EMAIL PROTECTED] [mailto:users- [EMAIL PROTECTED] On Behalf Of Alex Henderson Sent: Monday, June 26, 2006
[IronPython] no module named win32api using PyWin32-208
I posted this on the comp.lang.python list yesterday but am not sure if that is the correct list to have posted on, so apologies for the cross-post but I would really appreciate some help on this. I have Python 2.4 installed on my local machine in c:\Python24. I have also downloaded the python for windows extensions installer pywin32-208.win32-py2.4.exe and installed this to C:\Python24\Lib\site-packages Trying to run a python script through a C# console app is causing me problems: the last line of code in the following block results in a no module named win32ap error. I'm not sure if this is because there is no win32api.py in the win32 folder off site-packages, just a win32api.pyc file. m_engine = new PythonEngine();m_engine.AddToPath(C:\\Python24\\DLLs);m_engine.AddToPath(C:\\Python24\\lib); m_engine.AddToPath(C:\\Python24\\lib\\plat-win);m_engine.AddToPath(C:\\Python24\\lib\\lib-tk); m_engine.AddToPath(C:\\Python24\\Lib\\site-packages\\pythonwin);m_engine.AddToPath(C:\\Python24);m_engine.AddToPath(C:\\Python24\\lib\\site-packages); m_engine.AddToPath(C:\\Python24\\lib\\site-packages\\win32); m_engine.AddToPath(C:\\Python24\\lib\\site-packages\\win32\\lib); m_engine.Execute(from win32api import win32api); I have added all the addtopaths to get the path to match the sys.path I see in a normal python console which can successfully import the module. Incidentally, I have tried making the last linem_engine.Execute(import win32api); with no luck. Can the win32 extensions handle compiled python modules? If not how can I get it to work? Thanks, Mike ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] no module named win32api using PyWin32-208
We don't support compiled Python modules (PYD's or PYC's, I'm not sure which one pywin32 is though I'd bet PYD) and isn't anything we'll support before 1.0. But if it's access to Win32 you want, there's a large amount of Win32 exposed through the .NET APIs that you can access directly by importing the appropriate namespaces (and adding references to the appropriate assemblies if it's not mscorlib or System). That may or may not work for you depending on what your goals are. From: [EMAIL PROTECTED] On Behalf Of Mike Raath Sent: Tuesday, June 27, 2006 10:21 PM To: users@lists.ironpython.com Subject: [IronPython] no module named win32api using PyWin32-208 I posted this on the comp.lang.python list yesterday but am not sure if that is the correct list to have posted on, so apologies for the cross-post but I would really appreciate some help on this. I have Python 2.4 installed on my local machine in c:\Python24. I have also downloaded the python for windows extensions installer pywin32-208.win32-py2.4.exe and installed this to C:\Python24\Lib\site-packages Trying to run a python script through a C# console app is causing me problems: the last line of code in the following block results in a no module named win32ap error. I'm not sure if this is because there is no win32api.py in the win32 folder off site-packages, just a win32api.pyc file. m_engine = new PythonEngine(); m_engine.AddToPath(C:\\Python24\\DLLs); m_engine.AddToPath(C:\\Python24\\lib); m_engine.AddToPath(C:\\Python24\\lib\\plat-win); m_engine.AddToPath(C:\\Python24\\lib\\lib-tk); m_engine.AddToPath(C:\\Python24\\Lib\\site-packages\\pythonwin); m_engine.AddToPath(C:\\Python24); m_engine.AddToPath(C:\\Python24\\lib\\site-packages); m_engine.AddToPath(C:\\Python24\\lib\\site-packages\\win32); m_engine.AddToPath(C:\\Python24\\lib\\site-packages\\win32\\lib); m_engine.Execute(from win32api import win32api); I have added all the addtopaths to get the path to match the sys.path I see in a normal python console which can successfully import the module. Incidentally, I have tried making the last line m_engine.Execute(import win32api); with no luck. Can the win32 extensions handle compiled python modules? If not how can I get it to work? Thanks, Mike ___ users mailing list users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
