Re: unloading a module created with imp.new_module
I am defining a single class with a destructor method that prints ‘__del__’,
and running that source code string using exec with the module’s dict like so:
import rtmidi
import sys
import types
import time
import gc
s =
class A:
def __del__(self):
print('__del__')
a = A()
m = types.ModuleType('mine')
exec(s, m.__dict__)
print('deleting...')
m = None
print('done')
and the output is:
deleting...
done
__del__
I the “__del__ to come between “deleting…” and “done”. This is not being run
from the interactive interpreter by via a .py file.
On Nov 23, 2014, at 12:56 AM, Ian Kelly ian.g.ke...@gmail.com wrote:
On Sun, Nov 23, 2014 at 2:48 AM, Ian Kelly ian.g.ke...@gmail.com wrote:
On Sat, Nov 22, 2014 at 11:49 PM, Patrick Stinson patrickk...@gmail.com
wrote:
If I create a module with imp.new_module(name), how can I unload it so that
all the references contained in it are set to zero and the module is
deleted? deleting the reference that is returned doesn’t seem to do the
job, and it’s not in sys.modules, so where is the dangling reference?
How are you determining that the module is not deleted?
From my testing, using Python 3.4:
for i in range(5): imp.new_module('spam')
...
module 'spam'
module 'spam'
module 'spam'
module 'spam'
module 'spam'
import gc, types
[m for m in gc.get_objects() if isinstance(m, types.ModuleType) and
m.__name__ == 'spam']
[module 'spam']
42
42
[m for m in gc.get_objects() if isinstance(m, types.ModuleType) and
m.__name__ == 'spam']
[]
In this case one of the created modules was hanging around because it
was still referenced by the special _ variable of the interactive
interpreter. Evaluating a new expression cleared that out and deleted
the module. Maybe you're seeing the same thing.
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
Basically, I have an app and am making it possible for the users to automate it by writing python scripts that run against the app's API. It includes PyQt5 widgets, a QScintilla editor, and a simple API for I/O. Apple’s app store is the plan. Right now I create a new module with types.ModuleType(‘something’), then exec the code in the module’s dict. Then when the user saves the script again in my editor, it deletes all top-level references in the module’s dict that don’t belong to the core list: ['__spec__', '__name__', '__loader__', '__package__', '__doc__', '__builtins__’] before running exec again. So far this seems to effectively “delete” the old module and re-compile it with the updated source. For example I can create Qt Widgets on the module level and they are safely deleted when the script is saved. On Nov 23, 2014, at 5:56 PM, Ned Batchelder n...@nedbatchelder.com wrote: On 11/23/14 1:49 AM, Patrick Stinson wrote: If I create a module with imp.new_module(name), how can I unload it so that all the references contained in it are set to zero and the module is deleted? deleting the reference that is returned doesn’t seem to do the job, and it’s not in sys.modules, so where is the dangling reference? Thanks! This sounds tricky, and possible very difficult to do properly. Do you mind if I ask what the larger problem is? Python might not be very good at having modules come and go as you want. -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
How does the __del__ method have a reference to the module’s globals dict?
because it references the print function?
Crazy. Is there any other way to comfort when a module is being deleted beside
defining a class object with a printing dtor?
On Nov 23, 2014, at 4:27 PM, Ian Kelly ian.g.ke...@gmail.com wrote:
On Nov 23, 2014 4:10 AM, Patrick Stinson patrickk...@gmail.com
mailto:patrickk...@gmail.com wrote:
m = types.ModuleType('mine')
exec(s, m.__dict__)
print('deleting...')
m = None
print('done')
and the output is:
deleting...
done
__del__
I the “__del__ to come between “deleting…” and “done”. This is not being
run from the interactive interpreter by via a .py file.
This suggests the presence of a reference cycle, since the object is deleted
afterward by the garbage collector. One cycle here is that the __del__
function has a reference to the module's globals dict, which has a reference
to the class instance, which has a reference to the class, which has a
reference back to the function. There may be other cycles here as well, but
it may also be that the module object itself isn't part of them.
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Mon, Nov 24, 2014 at 6:27 PM, Patrick Stinson patrickk...@gmail.com wrote: How does the __del__ method have a reference to the module’s globals dict? because it references the print function? Crazy. Is there any other way to comfort when a module is being deleted beside defining a class object with a printing dtor? Functions have references to their modules. That's how they know what globals to use. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On 11/24/14 2:25 AM, Patrick Stinson wrote: Basically, I have an app and am making it possible for the users to automate it by writing python scripts that run against the app's API. It includes PyQt5 widgets, a QScintilla editor, and a simple API for I/O. Apple’s app store is the plan. Right now I create a new module with types.ModuleType(‘something’), then exec the code in the module’s dict. Then when the user saves the script again in my editor, it deletes all top-level references in the module’s dict that don’t belong to the core list: ['__spec__', '__name__', '__loader__', '__package__', '__doc__', '__builtins__’] before running exec again. So far this seems to effectively “delete” the old module and re-compile it with the updated source. For example I can create Qt Widgets on the module level and they are safely deleted when the script is saved. (BTW: the convention in most groups like this one is to bottom-post: put your responses beneath the text you are responding to.) Keep in mind that Python object have a reference to their class. If your users are defining classes in their modules, and you instantiate the class to create objects, then even when you reload updated code, the old objects will still refer to the old classes. This is one of the things that makes reloading modules in Python so difficult. On Nov 23, 2014, at 5:56 PM, Ned Batchelder n...@nedbatchelder.com mailto:n...@nedbatchelder.com wrote: On 11/23/14 1:49 AM, Patrick Stinson wrote: If I create a module with imp.new_module(name), how can I unload it so that all the references contained in it are set to zero and the module is deleted? deleting the reference that is returned doesn’t seem to do the job, and it’s not in sys.modules, so where is the dangling reference? Thanks! This sounds tricky, and possible very difficult to do properly. Do you mind if I ask what the larger problem is? Python might not be very good at having modules come and go as you want. -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On 11/23/14 5:10 AM, Patrick Stinson wrote:
I am defining a single class with a destructor method that prints
‘__del__’, and running that source code string using exec with the
module’s dict like so:
import rtmidi
importsys
import types
importtime
importgc
s=
class A:
def __del__(self):
print('__del__')
a = A()
m = types.ModuleType('mine')
exec(s, m.__dict__)
print('deleting...')
m= None
print('done')
and the output is:
deleting...
done
__del__
I the “__del__ to come between “deleting…” and “done”. This is not
being run from the interactive interpreter by via a .py file.
Let's look at this another way: Why do you need the module to be
unloaded? Isn't it enough to have the new code loaded?
--
Ned Batchelder, http://nedbatchelder.com
--
https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Mon, Nov 24, 2014 at 10:21 PM, Ned Batchelder n...@nedbatchelder.com wrote: Keep in mind that Python object have a reference to their class. If your users are defining classes in their modules, and you instantiate the class to create objects, then even when you reload updated code, the old objects will still refer to the old classes. This is one of the things that makes reloading modules in Python so difficult. Yes, which is why I actually moved right away from that model, in two projects that are built heavily around code reloading. In both cases, code is all done as simple functions which run and then finish, and everything returns as quickly as possible to a main back-end loop. All persistent state is kept in code-free data structures, usually key-based mappings of various sorts, with instantiable objects being used only in places where it won't matter if they keep using the old code. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Nov 24, 2014 1:27 AM, Patrick Stinson patrickk...@gmail.com wrote: How does the __del__ method have a reference to the module’s globals dict? because it references the print function? The module's dict becomes the __globals__ dict used by the function for looking up globals and builtins. Crazy. Is there any other way to comfort when a module is being deleted beside defining a class object with a printing dtor? Modules don't strictly have to be ModuleType. You could substitute a class with a __del__ method for the module itself for testing this. Be careful with leaving this in your production code though, as the presence of __del__ methods can interfere with garbage collection and are not advisable prior to 3.4. You can also test whether an object is still in memory by looking for it in gc.get_objects() although be warned that is an expensive call. -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Tue, Nov 25, 2014 at 2:12 AM, Ian Kelly ian.g.ke...@gmail.com wrote: Crazy. Is there any other way to comfort when a module is being deleted beside defining a class object with a printing dtor? Modules don't strictly have to be ModuleType. You could substitute a class with a __del__ method for the module itself for testing this. And based on the original description of exec'ing code into the module's dictionary, the module is actually quite superfluous - just exec the code into an empty dict and you'll get a similar result. Though if you're doing a lot with code reloading, you may want to consider a language with stronger support for it, like Pike. Python can do it, but it's not really something most people do, so the language and stdlib are generally built on the assumption that, once loaded, code stays loaded. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Nov 24, 2014, at 6:12 AM, Ian Kelly ian.g.ke...@gmail.com wrote: On Nov 24, 2014 1:27 AM, Patrick Stinson patrickk...@gmail.com mailto:patrickk...@gmail.com wrote: How does the __del__ method have a reference to the module’s globals dict? because it references the print function? The module's dict becomes the __globals__ dict used by the function for looking up globals and builtins. Wow, that’s very helpful and good to know. Funny I never ran into that when doing this via an extensive CPython embedding project. Or maybe I ignored it. Crazy. Is there any other way to comfort when a module is being deleted beside defining a class object with a printing dtor? Modules don't strictly have to be ModuleType. You could substitute a class with a __del__ method for the module itself for testing this. Be careful with leaving this in your production code though, as the presence of __del__ methods can interfere with garbage collection and are not advisable prior to 3.4. You can also test whether an object is still in memory by looking for it in gc.get_objects() although be warned that is an expensive call. Interesting concept. One thing I am doing is allowing one script to import another by name. So in my gui I’ll have a panel for each script with a line edit to set the name, which then generates an appropriate slug. Is it still possible to import non-module objects? I guess this question is just for fun now :) -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Tue, Nov 25, 2014 at 5:48 PM, Patrick Stinson patrickk...@gmail.com wrote: Is it still possible to import non-module objects? I guess this question is just for fun now :) rosuav@sikorsky:~$ python3 Python 3.5.0a0 (default:23ab1197df0b, Nov 20 2014, 12:57:44) [GCC 4.7.2] on linux Type help, copyright, credits or license for more information. import sys sys.modules[hello]=lambda s: print(Hello, %s!%s) import hello hello(world) Hello, world! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Nov 24, 2014, at 2:44 AM, Ned Batchelder n...@nedbatchelder.com wrote:
On 11/23/14 5:10 AM, Patrick Stinson wrote:
I am defining a single class with a destructor method that prints
‘__del__’, and running that source code string using exec with the
module’s dict like so:
import rtmidi
importsys
import types
importtime
importgc
s=
class A:
def __del__(self):
print('__del__')
a = A()
m = types.ModuleType('mine')
exec(s, m.__dict__)
print('deleting...')
m= None
print('done')
and the output is:
deleting...
done
__del__
I the “__del__ to come between “deleting…” and “done”. This is not
being run from the interactive interpreter by via a .py file.
Let's look at this another way: Why do you need the module to be unloaded?
Isn't it enough to have the new code loaded?
I think that’s a good question. My original intention was to delete the module
and everything contained in it since that’s what best matches what the user
would expect by recompiling a script that they just created within this app.
But the more I read and think, this purist perspective doesn’t make much sense.
The solution that I’ve come up with (which seems to be working really well so
far), is to delete all objects in the module’s dict except the usual implicit
objects (__import__, __name__, etc), then re-compile the code into the module’s
dict. This even works well when I define QWidget gui objects, as they are
hidden and deleted when the last reference goes away.
The main drawbacks for this approach are that the scripts exist in the same
interpreter, and so can directly manipulate the app’s object hierarchy. But a
user would have to go out of their way to do that so it’s their own fault if
they break something :)
I would *love* to see code examples of how this sort of thing is done, where
you write a GUI app in python that allows it’s users to write plugins for it in
python using a built-in editor.
Thanks! Great thread
--
Ned Batchelder, http://nedbatchelder.com http://nedbatchelder.com/
--
https://mail.python.org/mailman/listinfo/python-list
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
unloading a module created with imp.new_module
If I create a module with imp.new_module(name), how can I unload it so that all the references contained in it are set to zero and the module is deleted? deleting the reference that is returned doesn’t seem to do the job, and it’s not in sys.modules, so where is the dangling reference? Thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Sat, Nov 22, 2014 at 11:49 PM, Patrick Stinson patrickk...@gmail.com wrote: If I create a module with imp.new_module(name), how can I unload it so that all the references contained in it are set to zero and the module is deleted? deleting the reference that is returned doesn’t seem to do the job, and it’s not in sys.modules, so where is the dangling reference? How are you determining that the module is not deleted? -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Sun, Nov 23, 2014 at 2:48 AM, Ian Kelly ian.g.ke...@gmail.com wrote:
On Sat, Nov 22, 2014 at 11:49 PM, Patrick Stinson patrickk...@gmail.com
wrote:
If I create a module with imp.new_module(name), how can I unload it so that
all the references contained in it are set to zero and the module is
deleted? deleting the reference that is returned doesn’t seem to do the job,
and it’s not in sys.modules, so where is the dangling reference?
How are you determining that the module is not deleted?
From my testing, using Python 3.4:
for i in range(5): imp.new_module('spam')
...
module 'spam'
module 'spam'
module 'spam'
module 'spam'
module 'spam'
import gc, types
[m for m in gc.get_objects() if isinstance(m, types.ModuleType) and
m.__name__ == 'spam']
[module 'spam']
42
42
[m for m in gc.get_objects() if isinstance(m, types.ModuleType) and
m.__name__ == 'spam']
[]
In this case one of the created modules was hanging around because it
was still referenced by the special _ variable of the interactive
interpreter. Evaluating a new expression cleared that out and deleted
the module. Maybe you're seeing the same thing.
--
https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Nov 23, 2014 4:10 AM, Patrick Stinson patrickk...@gmail.com wrote:
m = types.ModuleType('mine')
exec(s, m.__dict__)
print('deleting...')
m = None
print('done')
and the output is:
deleting...
done
__del__
I the “__del__ to come between “deleting…” and “done”. This is not being
run from the interactive interpreter by via a .py file.
This suggests the presence of a reference cycle, since the object is
deleted afterward by the garbage collector. One cycle here is that the
__del__ function has a reference to the module's globals dict, which has a
reference to the class instance, which has a reference to the class, which
has a reference back to the function. There may be other cycles here as
well, but it may also be that the module object itself isn't part of them.
--
https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On 11/23/14 1:49 AM, Patrick Stinson wrote: If I create a module with imp.new_module(name), how can I unload it so that all the references contained in it are set to zero and the module is deleted? deleting the reference that is returned doesn’t seem to do the job, and it’s not in sys.modules, so where is the dangling reference? Thanks! This sounds tricky, and possible very difficult to do properly. Do you mind if I ask what the larger problem is? Python might not be very good at having modules come and go as you want. -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: unloading a module created with imp.new_module
On Sat, 22 Nov 2014 21:49:32 -0900, Patrick Stinson wrote: If I create a module with imp.new_module(name), how can I unload it so that all the references contained in it are set to zero and the module is deleted? deleting the reference that is returned doesn’t seem to do the job, and it’s not in sys.modules, so where is the dangling reference? It would be very surprising (although not impossible) if you have found a bug in the garbage collector. More likely you've just misinterpreted what you're seeing. Can you please reply to the list with the simplest example of code that behaves in the way you are describing? (If you reply to me privately, I may not get to see it for days or weeks, and when I do, my response will be to ask you to resend it to the list. So save all of us some time by replying directly to the list.) If you're experimenting in the interactive interpreter, be aware that it holds onto a reference to the last few objects displayed. IPython can hold on to an unlimited number of such references, but the standard Python REPL only holds onto one. That can confound experimentation: py import imp, gc, types py mymodule = imp.new_module(mymodule) py [o for o in gc.get_objects() if isinstance(o, types.ModuleType) ... and o.__name__ == 'mymodule'] [module 'mymodule' (built-in)] py del mymodule py [o for o in gc.get_objects() if isinstance(o, types.ModuleType) ... and o.__name__ == 'mymodule'] [module 'mymodule' (built-in)] It appears that the module isn't being garbage collected. But that's because the list [mymodule] is being held in the interactive interpreter's special _ variable. If we get rid of that by displaying something else, the module is garbage collected: py 23 23 py [o for o in gc.get_objects() if isinstance(o, types.ModuleType) ... and o.__name__ == 'mymodule'] [] -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
In article 78388a7a-b148-499a-8894-34e55721e...@k19g2000pro.googlegroups.com, lordofcode ajay@gmail.com wrote: Thanks you all for your replies ,cleared quiet a few doubts about importing modules and namespace references . Currently am going with static importing of all modules. But it may not be efficient in future as the number of interchangeable modules that I have to import may run in 30-40's.(Basically this Python code is part of larger GSM Testing project where each module represents one model of GSM mobile phone and this number keeps growing). So changing the design will be a last resort. Worry when you have 30-40 hundreds of modules. ;-) Seriously, it's just memory; Python's efficient dicts make lots of modules not a problem. If/when it's time to worry, there are other techniques you can use. -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ If you think it's expensive to hire a professional to do the job, wait until you hire an amateur. --Red Adair -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
Steven D'Aprano wrote: On Wed, 23 Dec 2009 15:31:53 +0100, Jean-Michel Pichavant wrote: Steven D'Aprano wrote: On Wed, 23 Dec 2009 13:37:06 +0100, Jean-Michel Pichavant wrote: But believe me, you don't want to mess up with the python import mechanism. Unless you understand how it works. Let me quote the OP: 'Not an expert in Python' I was just answering the OP question, without refering to something he could do if he was someone else. But what he *can* do is to learn how importing works. I'm not sure it's terribly helpful to tell somebody all the things they can't do instead of what they can. Hacking python imports would not be in the top of the list of things I would teach to some python newcomer, that was my point. And to be a little more constructive, let's give the OP a litle bit of help. How can you unload module in python ? That I don't know, anyone knows ? JM -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
On 12/24/2009 11:51 PM, Jean-Michel Pichavant wrote: But what he *can* do is to learn how importing works. I'm not sure it's terribly helpful to tell somebody all the things they can't do instead of what they can. Hacking python imports would not be in the top of the list of things I would teach to some python newcomer, that was my point. And to be a little more constructive, let's give the OP a litle bit of help. How can you unload module in python ? That I don't know, anyone knows ? Given that there is no way to ensure deconstruction of an object instance (`del` only removes the reference and __del__ can only remove the resources used by the instance but not the instance itself), why would anyone expect it is possible to unload a module explicitly (which is a deconstruction of the module) -- http://mail.python.org/mailman/listinfo/python-list
Problem with Dynamically unloading a module
Hi All Not an expert in Python, so sorry if this sounds like a silly question. I went through other few threads in the mailing list but they are not helping me much. I have run into a problem related to dynamically loading and unloading a module. I need to dynamically load a module and unload it and load another module. For example I have many files(All files in Python are modules right?) like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder which contains classes and methods with same name but different functionality.(am afraid I cannot change this structure as these files are generated randomly by the user) So initially when my program starts I have to load a default module. I do this as follows: ## MODULE_name = mobile_1 exec from +MODULE_name+ import * ## And use the methods defined in mobile_1.py file Now as the application continues , I may have to use the methods defined in mobile_2.py or mobile_3.py etc instead of the previously loaded module,which I incorrectly try to do as below: MODULE_name = mobile_2 exec from +MODULE_name+ import * # The above import does not have any impact and the methods called from my application still pertain to mobile_1.py as its still in the current namespace(?). I tried below code with del(), reload() etc but could not figure it out. ###Code to unload a dll del sys.modules[MODULE_name]#== does not delete the reference in namespace 1)How do I unload a module dynamically and completely remove the references in the module so a new module with same name references can be loaded? 2)Are there any alternative way to do the above requirement? Currently I am working around by restarting the whole initial setup for each new module which is unnecessary waste.Can I avoid this reset? -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
On 12/23/2009 8:41 PM, lordofcode wrote:
Hi All
Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.
For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)
So initially when my program starts I have to load a default module. I
do this as follows:
##
MODULE_name = mobile_1
exec from +MODULE_name+ import *
##
And use the methods defined in mobile_1.py file
You probably shouldn't use from module import *, use the __import__
function:
name = __import__('one')
name.foo() # defined in `one`
name = __import__('two')
name.bar() # defined in `two`
Now as the application continues , I may have to use the methods
defined in mobile_2.py or mobile_3.py etc instead of the
previously loaded module,which I incorrectly try to do as below:
MODULE_name = mobile_2
exec from +MODULE_name+ import *
#
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).
that will just imports mobile_2's function into the current namespace,
possibly overriding the previously defined names from mobile_1 but it
won't delete anything that doesn't get overriden.
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll
del sys.modules[MODULE_name]#== does not delete the reference in namespace
1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
reset?
You can start a python subprocess for each new module you're loading.
That restarts the whole thing, but python loads fairly quickly and it
ensures that there won't be any leftover side effects caused by the
execution of the previous module.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
lordofcode wrote:
Hi All
Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.
For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)
So initially when my program starts I have to load a default module. I
do this as follows:
##
MODULE_name = mobile_1
exec from +MODULE_name+ import *
##
And use the methods defined in mobile_1.py file
Now as the application continues , I may have to use the methods
defined in mobile_2.py or mobile_3.py etc instead of the
previously loaded module,which I incorrectly try to do as below:
MODULE_name = mobile_2
exec from +MODULE_name+ import *
#
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll
del sys.modules[MODULE_name]#== does not delete the reference in namespace
1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
reset?
1/ Do not use from x import *
2/ How many modules would you like to import in the end ? If this number
is bound I would suggest to use a static approach:
import mod1
import mod2
import mod3
mod4 = __import__('mod4') # equivalent to import mod4
for mod in [mod1, mod2, mod3, mod4]:
# will call func1 on each modules
print now using module %s functions % mod.__file__
mod.func1()
mod.func2()
With this approach it's pretty much easy to call the correct
methods/attributes, as every symbols is scopped by module namespace. You
will have less hard time debugging your code.
3/ if you really need to unload the previous module, it's a little bit
tedious.
import mod1
del mod1
sys.modules['mod1'] = None # will unload mod1 assuming mod1 was the only
reference to that module.
But believe me, you don't want to mess up with the python import mechanism.
JM
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
Jean-Michel Pichavant wrote:
lordofcode wrote:
Hi All
Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.
For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)
So initially when my program starts I have to load a default module. I
do this as follows:
##
MODULE_name = mobile_1
exec from +MODULE_name+ import *
##
And use the methods defined in mobile_1.py file
Now as the application continues , I may have to use the methods
defined in mobile_2.py or mobile_3.py etc instead of the
previously loaded module,which I incorrectly try to do as below:
MODULE_name = mobile_2
exec from +MODULE_name+ import *
#
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll
del sys.modules[MODULE_name]#== does not delete the reference
in namespace
1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
reset?
1/ Do not use from x import *
2/ How many modules would you like to import in the end ? If this
number is bound I would suggest to use a static approach:
import mod1
import mod2
import mod3
mod4 = __import__('mod4') # equivalent to import mod4
for mod in [mod1, mod2, mod3, mod4]:
# will call func1 on each modules
print now using module %s functions % mod.__file__
mod.func1()
mod.func2()
With this approach it's pretty much easy to call the correct
methods/attributes, as every symbols is scopped by module namespace.
You will have less hard time debugging your code.
3/ if you really need to unload the previous module, it's a little bit
tedious.
import mod1
del mod1
sys.modules['mod1'] = None # will unload mod1 assuming mod1 was the
only reference to that module.
But believe me, you don't want to mess up with the python import
mechanism.
JM
You may also want to look at the imp module,
http://docs.python.org/library/imp.html
It could help you in your task.
JM
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
On Wed, 23 Dec 2009 13:37:06 +0100, Jean-Michel Pichavant wrote: 3/ if you really need to unload the previous module, it's a little bit tedious. import mod1 del mod1 sys.modules['mod1'] = None Assigning sys.modules[name] to None is not the same as deleting the entry. None has special meaning to imports from packages, and for modules it is interpreted as meaning that the module doesn't exist. import math del math sys.modules['math'] = None import math Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named math # will unload mod1 assuming mod1 was the only reference to that module. Which is highly unlikely. Any classes or functions from the module will keep the module alive. But believe me, you don't want to mess up with the python import mechanism. Unless you understand how it works. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
On Wed, 23 Dec 2009 01:41:27 -0800, lordofcode wrote: I need to dynamically load a module and unload it and load another module. Why bother unloading it? Unless you're programming for an embedded device, it's highly unlikely that you're so strapped for memory that a module will make any real difference. Just stop using the module and start using another. (See below for example.) For example I have many files(All files in Python are modules right?) like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder which contains classes and methods with same name but different functionality.(am afraid I cannot change this structure as these files are generated randomly by the user) So initially when my program starts I have to load a default module. I do this as follows: ## MODULE_name = mobile_1 exec from +MODULE_name+ import * ## And use the methods defined in mobile_1.py file Now as the application continues , I may have to use the methods defined in mobile_2.py or mobile_3.py etc instead of the previously loaded module, Sounds like an absolute horrible design. I recommend that you re-think the design and do something less fragile and difficult. The obvious way would be something like this: # Import the first module and use it. user_module = __import__(mobile_1) x = user_module.function(123) y = user_module.SomeClass(123) # Now change modules. user_module = __import__(mobile_2) x = user_module.function(456) y = user_module.SomeClass(456) Note that: (1) The function __import__ loads the module given a name which is only known at runtime instead of compile time. (2) Don't use the form from module import name form. That injects the objects into the current namespace, which generally leads to more trouble than benefit. (3) After replacing the module, you have to get rid of the old classes that refer to the old module, and create a new objects. Classes don't magically change their behaviour just because the name of the class points to something different: class C: ... def method(self): ... return Something! ... instance = C() instance.method() 'Something!' class C: # replace the old class ... def method(self): ... return Something different ... instance.method() 'Something!' I'm betting that once you understand that behaviour, you will understand why your approach is failing. which I incorrectly try to do as below: MODULE_name = mobile_2 exec from +MODULE_name+ import * # The above import does not have any impact and the methods called from my application still pertain to mobile_1.py as its still in the current namespace(?). The most likely problem is that you have misunderstood how Python's object model works. Suppose you do this: from module_1 import MyClass x = MyClass(1, 2, 3) # create a class instance x.method() The object called x is now an instance of MyClass from module_1. So far so good. Suppose that you do this: from module_2 import MyClass x.method() I'm guessing that you expect that this will call the method from MyClass in module_2. But that's not what happens: the object x is still the same object it was before, and it still calls the method from module_1. You have to throw away that object and create a new one: from module_2 import MyClass x = MyClass(1, 2, 3) x.method() (This, BTW, is exactly the same situation as the same code above with class C and instance.method, except an import is added to the mix.) Now, it's not *impossible* to play tricks like this: from module_2 import MyClass # replace the reference to the old MyClass with the new MyClass: x.__class__ = MyClass x.method() which *may* work, for some classes. But better to avoid this unless you can control the classes yourself, which you can't. Or find a better design that doesn't rely on dynamically replacing modules. I tried below code with del(), reload() etc but could not figure it out. ###Code to unload a dll This has nothing to do with DLLs. Python modules are Python modules, not DLLs, and they don't work the same way. del sys.modules[MODULE_name]#== does not delete the reference in namespace Deleting the module from the cache does nothing except slow Python down the next time you import the module. 1)How do I unload a module dynamically and completely remove the references in the module so a new module with same name references can be loaded? You don't have to. You just have to understand how Python actually works, and work *with* it instead of *against* it. 2)Are there any alternative way to do the above requirement? Currently I am working around by restarting the whole initial setup for each new module which is unnecessary waste. What makes you think it's unnecessary? Can I avoid this reset? I don't know, because I don't fully understand what restarting the whole initial
Re: Problem with Dynamically unloading a module
Steven D'Aprano wrote: On Wed, 23 Dec 2009 13:37:06 +0100, Jean-Michel Pichavant wrote: But believe me, you don't want to mess up with the python import mechanism. Unless you understand how it works. Let me quote the OP: 'Not an expert in Python' I was just answering the OP question, without refering to something he could do if he was someone else. JM -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
Steven D'Aprano wrote:
On Wed, 23 Dec 2009 13:37:06 +0100, Jean-Michel Pichavant wrote:
3/ if you really need to unload the previous module, it's a little bit
tedious.
import mod1
del mod1
sys.modules['mod1'] = None
Assigning sys.modules[name] to None is not the same as deleting the
entry. None has special meaning to imports from packages, and for modules
it is interpreted as meaning that the module doesn't exist.
did'nt know that.
sys.modules.pop('mod1')
should then do the trick.
# will unload mod1 assuming mod1 was the only
reference to that module.
Which is highly unlikely. Any classes or functions from the module will
keep the module alive.
Yep.
This is why I strongly suggested the OP to use a static approach,
unloading/reloading module in python isn't really the most obvious thing
ever done. I would even dare to say that if it was possible, someone
would have already written the module for it.
JM
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
On Wed, 23 Dec 2009 15:31:53 +0100, Jean-Michel Pichavant wrote: Steven D'Aprano wrote: On Wed, 23 Dec 2009 13:37:06 +0100, Jean-Michel Pichavant wrote: But believe me, you don't want to mess up with the python import mechanism. Unless you understand how it works. Let me quote the OP: 'Not an expert in Python' I was just answering the OP question, without refering to something he could do if he was someone else. But what he *can* do is to learn how importing works. I'm not sure it's terribly helpful to tell somebody all the things they can't do instead of what they can. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
On Dec 23, 7:40 am, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: On Wed, 23 Dec 2009 13:37:06 +0100, Jean-Michel Pichavant wrote: 3/ if you really need to unload the previous module, it's a little bit tedious. import mod1 del mod1 sys.modules['mod1'] = None Assigning sys.modules[name] to None is not the same as deleting the entry. None has special meaning to imports from packages, and for modules it is interpreted as meaning that the module doesn't exist. import math del math sys.modules['math'] = None import math Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named math # will unload mod1 assuming mod1 was the only reference to that module. Which is highly unlikely. Any classes or functions from the module will keep the module alive. Actually, they won't. Neither classes nor functions directly reference their module; classes know their module only by name, and functions only hold references to the module's namespace, not to the module itself. So if any references to functions defined in the module remain, the module dict will stick around, but the module itself may be collected. But believe me, you don't want to mess up with the python import mechanism. Unless you understand how it works. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
Hi All, Thanks you all for your replies ,cleared quiet a few doubts about importing modules and namespace references . Currently am going with static importing of all modules. But it may not be efficient in future as the number of interchangeable modules that I have to import may run in 30-40's.(Basically this Python code is part of larger GSM Testing project where each module represents one model of GSM mobile phone and this number keeps growing). So changing the design will be a last resort. But am trying to reduce the number of objects(classes,methods etc) in each module so importing all 30-40 modules do not impact much. Anyway am trying out all the mentioned tricks here and get back to you with any doubts. Thanks again Ajay Baragur -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
On Wed, 23 Dec 2009 15:18:10 -0800, Carl Banks wrote: # will unload mod1 assuming mod1 was the only reference to that module. Which is highly unlikely. Any classes or functions from the module will keep the module alive. Actually, they won't. Neither classes nor functions directly reference their module; classes know their module only by name, and functions only hold references to the module's namespace, not to the module itself. So if any references to functions defined in the module remain, the module dict will stick around, but the module itself may be collected. Hmmm... so it seems. Well spotted, thanks. It makes sense to do it that way, as it prevents circular references (module contains a class which contains the module which...). -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with Dynamically unloading a module
lordofcode wrote: Hi All Not an expert in Python, so sorry if this sounds like a silly question. I went through other few threads in the mailing list but they are not helping me much. I have run into a problem related to dynamically loading and unloading a module. I need to dynamically load a module and unload it and load another module. For example I have many files(All files in Python are modules right?) like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder which contains classes and methods with same name but different functionality.(am afraid I cannot change this structure as these files are generated randomly by the user) So initially when my program starts I have to load a default module. I do this as follows: ## MODULE_name = mobile_1 exec from +MODULE_name+ import * ## And use the methods defined in mobile_1.py file Now as the application continues , I may have to use the methods defined in mobile_2.py or mobile_3.py etc instead of the previously loaded module,which I incorrectly try to do as below: MODULE_name = mobile_2 exec from +MODULE_name+ import * # The above import does not have any impact and the methods called from my application still pertain to mobile_1.py as its still in the current namespace(?). I tried below code with del(), reload() etc but could not figure it out. ###Code to unload a dll del sys.modules[MODULE_name]#== does not delete the reference in namespace 1)How do I unload a module dynamically and completely remove the references in the module so a new module with same name references can be loaded? 2)Are there any alternative way to do the above requirement? Currently I am working around by restarting the whole initial setup for each new module which is unnecessary waste.Can I avoid this reset? I'd argue that you don't want to unload the old code before loading the new module, and that what you really need is an understanding that if mod1 and mod2 both define a function called f then you can perfectly happily reference the two functions as mod1.f and mod2.f Indeed, if x is some object that needs processing by functions f, g and h from module mod1 then it's perfectly valid to write mod = mod1 result = mod.f(mod.g(mod.h(x))) for example. So you can write code that uses the appropriate module for each input. It's possible, of course, that I am misunderstanding your requirements, but I hope that helps. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS:http://holdenweb.eventbrite.com/ -- http://mail.python.org/mailman/listinfo/python-list
Unloading a module
Hello Group, If a reference to an imported module reaches zero will Python cleanup everything related to that module and unload the compiled code, etc, etc...? For example: import sys m = [__import__(str(x)) for x in xrange(1,4)] del sys.modules['1'] del m[0] print m Is module['1'] really unloaded or just it is not reachable but still loaded? Thanks, Elias -- http://mail.python.org/mailman/listinfo/python-list
Re: Unloading a module
En Thu, 22 Oct 2009 11:59:58 -0300, lallous lall...@lgwm.org escribió: If a reference to an imported module reaches zero will Python cleanup everything related to that module and unload the compiled code, etc, etc...? The module object itself (as any other object whose reference count reaches zero) will be destroyed. This in turn will decrement the reference count of its namespace (its __dict__), and when that reaches zero it will decrement the reference count of any object referenced (classes defined in the module, global variables...). If there are no other references to them, they will be destroyed too, as with any other object. It's always the same story. For example: import sys m = [__import__(str(x)) for x in xrange(1,4)] del sys.modules['1'] del m[0] print m Is module['1'] really unloaded or just it is not reachable but still loaded? Try with sys.getrefcount(the_module); when it reaches 2 you know the_module is the last reference to the object [remember that getrefcount always returns one more than the actual reference count] In your example above, after the __import__ line, there are two references to the module '1': one in sys.modules, and one in the m list. Once you remove them (with del, as in your example), the module object itself is destroyed, yes. -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
