Importing a class without knowing the module
Hello, I'm developing a small XML marshaller and I'm facing an annoying issue. Here's some sample code: ### My test application class Foo(object): #The class I'd like to serialize pass import myMarshaller foo = Foo() s = myMarshaller.dumps(foo) #works fine, spits something like another_foo = loads(s) #fails, see below ### My marshaller (in its own module) def loads(s): #First, get class name (here "Foo") klass = eval(className) #fails because "Foo" is not in the marshaller's namespace ! How could I tell the marshaller to locate the Foo class and any other class I try to deserialize ? I've tried to pass my test application's globals() to the marshaller, it works but it's dirty IMHO... I've tried also to locate the class (here "Foo") somewhere in sys.modules in the "loads" method, but it was heavy and unsuccessful. Thanks a lot for your help ! Franck -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
Franck PEREZ <[EMAIL PROTECTED]> writes: > ### My test application > class Foo(object): > #The class I'd like to serialize > pass > > import myMarshaller > foo = Foo() > s = myMarshaller.dumps(foo) #works fine, spits something like class = "Foo"...> > another_foo = loads(s) #fails, see below > > ### My marshaller (in its own module) > def loads(s): > #First, get class name (here "Foo") > klass = eval(className) #fails because "Foo" is not in the > marshaller's namespace ! > > How could I tell the marshaller to locate the Foo class and any other > class I try to deserialize ? How about adding Foo.__file__ to the serialized data? http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
I thought about it, but it would make the XML file depend on the machine... no more portability... On 11/18/05, Mike Meyer <[EMAIL PROTECTED]> wrote: > Franck PEREZ <[EMAIL PROTECTED]> writes: > > ### My test application > > class Foo(object): > > #The class I'd like to serialize > > pass > > > > import myMarshaller > > foo = Foo() > > s = myMarshaller.dumps(foo) #works fine, spits something like > class = "Foo"...> > > another_foo = loads(s) #fails, see below > > > > ### My marshaller (in its own module) > > def loads(s): > > #First, get class name (here "Foo") > > klass = eval(className) #fails because "Foo" is not in the > > marshaller's namespace ! > > > > How could I tell the marshaller to locate the Foo class and any other > > class I try to deserialize ? > > How about adding Foo.__file__ to the serialized data? > > -- > Mike Meyer <[EMAIL PROTECTED]> > http://www.mired.org/home/mwm/ > Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
Franck PEREZ <[EMAIL PROTECTED]> wrote: > I thought about it, but it would make the XML file depend on the > machine... no more portability... ... > > How about adding Foo.__file__ to the serialized data? Your top-posting makes this discourse weird (why put your comments BEFORE the text you're commenting on?!), but anyway I think that using the class's __module__ rather than __file__ should be what you want. Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
[Format recovered from top posting.] Franck PEREZ <[EMAIL PROTECTED]> writes: > On 11/18/05, Mike Meyer <[EMAIL PROTECTED]> wrote: >> Franck PEREZ <[EMAIL PROTECTED]> writes: >> > ### My test application >> > class Foo(object): >> > #The class I'd like to serialize >> > pass >> > >> > import myMarshaller >> > foo = Foo() >> > s = myMarshaller.dumps(foo) #works fine, spits something like > > class = "Foo"...> >> > another_foo = loads(s) #fails, see below >> > >> > ### My marshaller (in its own module) >> > def loads(s): >> > #First, get class name (here "Foo") >> > klass = eval(className) #fails because "Foo" is not in the >> > marshaller's namespace ! >> > >> > How could I tell the marshaller to locate the Foo class and any other >> > class I try to deserialize ? >> >> How about adding Foo.__file__ to the serialized data? > I thought about it, but it would make the XML file depend on the > machine... no more portability... They already depend on the machine. You can't take them to an arbitary machine and reconstruct them: it has to have the classes the XML file depends on somewhere on it. You can use the module name if you have it available. If not, deriving the module name from the file name is about the best you can do. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
Mike Meyer <[EMAIL PROTECTED]> wrote: ... > >> How about adding Foo.__file__ to the serialized data? > > I thought about it, but it would make the XML file depend on the > > machine... no more portability... > > They already depend on the machine. You can't take them to an arbitary > machine and reconstruct them: it has to have the classes the XML file > depends on somewhere on it. You can use the module name if you have it > available. If not, deriving the module name from the file name is > about the best you can do. I disagree with the last sentence. From a filepath of '/zip/zap/zop/zup.py', it's VERY hard to say whether the module name is to be zup, zop.zup, or zap.zop.zup -- it depends on which directories are on sys.path and which have __init__.py, which is impossible to tell at unmarshaling time. If you use Foo.__module__, you should get the *module* name correctly, independently from sys.path or __init__.py's, or .pth files for that matter -- so, I do NOT agree that using __file__ is "about the best you can do" for this use case. Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
[EMAIL PROTECTED] (Alex Martelli) writes: > Mike Meyer <[EMAIL PROTECTED]> wrote: >> >> How about adding Foo.__file__ to the serialized data? >> > I thought about it, but it would make the XML file depend on the >> > machine... no more portability... >> They already depend on the machine. You can't take them to an arbitary >> machine and reconstruct them: it has to have the classes the XML file >> depends on somewhere on it. You can use the module name if you have it >> available. If not, deriving the module name from the file name is >> about the best you can do. > I disagree with the last sentence. From a filepath of > '/zip/zap/zop/zup.py', it's VERY hard to say whether the module name is > to be zup, zop.zup, or zap.zop.zup -- it depends on which directories > are on sys.path and which have __init__.py, which is impossible to tell > at unmarshaling time. If you use Foo.__module__, you should get the > *module* name correctly, independently from sys.path or __init__.py's, > or .pth files for that matter -- so, I do NOT agree that using __file__ > is "about the best you can do" for this use case. You should read the next-to-last sentence, which says to use the module name if you have it. The last sentence starts "If not" - meaning you don't have the module name. *That's* the case for which the file name is about the best you can do. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
Mike Meyer <[EMAIL PROTECTED]> wrote: ... > >> >> How about adding Foo.__file__ to the serialized data? ... > >> depends on somewhere on it. You can use the module name if you have it > >> available. If not, deriving the module name from the file name is > >> about the best you can do. > > I disagree with the last sentence. From a filepath of ... > You should read the next-to-last sentence, which says to use the > module name if you have it. The last sentence starts "If not" - > meaning you don't have the module name. *That's* the case for which > the file name is about the best you can do. I see! Thanks for clarifying. Could you please give me an example of a Foo class which has a Foo.__file__ attribute but not a Foo.__module__ attribute? Sorry, must be some moment of weakness on my mind's part (quite possible, since I am recovering from recent surgery), but I cannot think of a situation where that would be the case (while classes with __module__ and w/o __file__ are the normal situation). Were there no case in which, given a class, you can learn about its file (by a __file__ attribute) but not about its module (by a __module__ attribute), I would of course hope that my inability to parse that sentence of yours, which would under such hypothetical circumstaces be an absurd hypothesis, might be more forgivable. Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
[EMAIL PROTECTED] (Alex Martelli) writes: > Mike Meyer <[EMAIL PROTECTED]> wrote: >... >> >> >> How about adding Foo.__file__ to the serialized data? >... >> >> depends on somewhere on it. You can use the module name if you have it >> >> available. If not, deriving the module name from the file name is >> >> about the best you can do. >> > I disagree with the last sentence. From a filepath of >... >> You should read the next-to-last sentence, which says to use the >> module name if you have it. The last sentence starts "If not" - >> meaning you don't have the module name. *That's* the case for which >> the file name is about the best you can do. > I see! Thanks for clarifying. Could you please give me an example of a > Foo class which has a Foo.__file__ attribute but not a Foo.__module__ > attribute? Sorry, must be some moment of weakness on my mind's part > (quite possible, since I am recovering from recent surgery), but I > cannot think of a situation where that would be the case (while classes > with __module__ and w/o __file__ are the normal situation). Were there > no case in which, given a class, you can learn about its file (by a > __file__ attribute) but not about its module (by a __module__ > attribute), I would of course hope that my inability to parse that > sentence of yours, which would under such hypothetical circumstaces be > an absurd hypothesis, might be more forgivable. A classes __module__ attribute doesn't always tell you the name of the module - or at least, not a name that would be usefull for the the OPs use case. That's the case where you don't have the module name. The reference to a classes __file__ attribute was meant to be to the modules __file__ attribute - I'm surprised that no one picked up on that. Again, assuming that the module has an __file__ attribute at all. Getting the __file__ attribute to a module you don't know the name of is a bit tricky, but possible. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
On 11/18/05, Mike Meyer <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] (Alex Martelli) writes: > > Mike Meyer <[EMAIL PROTECTED]> wrote: > >... > >> >> >> How about adding Foo.__file__ to the serialized data? > >... > >> >> depends on somewhere on it. You can use the module name if you have it > >> >> available. If not, deriving the module name from the file name is > >> >> about the best you can do. > >> > I disagree with the last sentence. From a filepath of > >... > >> You should read the next-to-last sentence, which says to use the > >> module name if you have it. The last sentence starts "If not" - > >> meaning you don't have the module name. *That's* the case for which > >> the file name is about the best you can do. > > I see! Thanks for clarifying. Could you please give me an example of a > > Foo class which has a Foo.__file__ attribute but not a Foo.__module__ > > attribute? Sorry, must be some moment of weakness on my mind's part > > (quite possible, since I am recovering from recent surgery), but I > > cannot think of a situation where that would be the case (while classes > > with __module__ and w/o __file__ are the normal situation). Were there > > no case in which, given a class, you can learn about its file (by a > > __file__ attribute) but not about its module (by a __module__ > > attribute), I would of course hope that my inability to parse that > > sentence of yours, which would under such hypothetical circumstaces be > > an absurd hypothesis, might be more forgivable. > > A classes __module__ attribute doesn't always tell you the name of the > module - or at least, not a name that would be usefull for the the OPs > use case. That's the case where you don't have the module name. The > reference to a classes __file__ attribute was meant to be to the > modules __file__ attribute - I'm surprised that no one picked up on > that. Again, assuming that the module has an __file__ attribute at > all. Getting the __file__ attribute to a module you don't know the > name of is a bit tricky, but possible. > > -- > Mike Meyer <[EMAIL PROTECTED]> > http://www.mired.org/home/mwm/ > Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. > -- > http://mail.python.org/mailman/listinfo/python-list > Thanks for your answers. And btw, sorry for top-posting, I'll be more careful next time. -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
Mike Meyer <[EMAIL PROTECTED]> wrote: ... > A classes __module__ attribute doesn't always tell you the name of the > module - or at least, not a name that would be usefull for the the OPs > use case. That's the case where you don't have the module name. The How do you arrange a module so that its classes' __module__ attributes don't tell you the name of the module "that would be useful", yet the module's __file__ DOES give you information that you can usefully process, heuristically I assume, to infer a module name that IS useful? I just don't see it, which of course doesn't mean it can't be done, with sufficient evil ingenuity. But since you're the one arguing that this case is important enough to be worth dwelling on, I'd rather see some specific examples from you, to understand whether my gut reaction "if possible at all, this has gotta be such a totally CORNER, ARTIFICIAL case, that it's absurd to bend over backwards to cover it" is justified, or whether I'm misjudging instead. > reference to a classes __file__ attribute was meant to be to the > modules __file__ attribute - I'm surprised that no one picked up on > that. Again, assuming that the module has an __file__ attribute at > all. Getting the __file__ attribute to a module you don't know the > name of is a bit tricky, but possible. If you have the module object, getting its __file__ isn't hard -- but then, neither is getting its module name... m.__file__ and m.__name__ are just about as accessible as each other. Of the two, the one more likely to be useless would be the __file__ -- import hooks (zipimport or more deviously sophisticated ones) might mean that string is not a file path at all; __name__ is supposed to be the string key of that module object within sys.modules, and it is, I think, far less likely that any tricks will have been played wrt that -- plus, you can easily doublecheck by scouring sys.modules. I just cannot see ANY case where I would want to try heuristics on some __file__ attribute (hopefully but not necessarily a filename) to try and recover a __name__ that appears to be missing, mangled, or incorrect; my instinct would be to raise exceptions informing the user that what they're trying to marshal is too weird and strange for their own good, and let them deal with the situation. But as I said, that may depend on a failure of the imagination -- if you can show me compelling use cases in which heuristics on __file__ prove perfectly satisfactory where just dealing with __name__ wouldn't, I'm quite ready to learn! Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
[EMAIL PROTECTED] (Alex Martelli) writes: > Mike Meyer <[EMAIL PROTECTED]> wrote: >... >> A classes __module__ attribute doesn't always tell you the name of the >> module - or at least, not a name that would be usefull for the the OPs >> use case. That's the case where you don't have the module name. The > How do you arrange a module so that its classes' __module__ attributes > don't tell you the name of the module "that would be useful", yet the > module's __file__ DOES give you information that you can usefully > process, heuristically I assume, to infer a module name that IS useful? So what module name do you import if C.__module__ is __main__? > I just don't see it, which of course doesn't mean it can't be done, with > sufficient evil ingenuity. But since you're the one arguing that this > case is important enough to be worth dwelling on, I'm not dwelling on it, you are. All I did was recommend using the module name if you had one, and if not then the file name was your best bet. You chose to ignore part of my statement to focus on the part you disagreed with, because you thought what I had suggested in the first place was a better idea. I pointed out this oversight on your part, and you've been dwelling on it ever since. Frankly, I thought you were brighter than that, have no idea why you're acting like this, and frankly don't think you need a tutor. I figured this out by fooling around with the interpreter before posting in the first place, you certainly know enough to do the same. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
Mike Meyer <[EMAIL PROTECTED]> wrote: ... > > How do you arrange a module so that its classes' __module__ attributes > > don't tell you the name of the module "that would be useful", yet the > > module's __file__ DOES give you information that you can usefully > > process, heuristically I assume, to infer a module name that IS useful? > > So what module name do you import if C.__module__ is __main__? In this case, I would strongly suggest that raising an exception upon the attempt to marshal is better than trying to guess that the file holding the main script could later be innocuously imported from a different script, or, alternatively, that the same main script would be running when a future unmarshaling attempt is done. "In the face of ambiguity, refuse the temptation to guess" is a sound principle. > > I just don't see it, which of course doesn't mean it can't be done, with > > sufficient evil ingenuity. But since you're the one arguing that this > > case is important enough to be worth dwelling on, > > I'm not dwelling on it, you are. All I did was recommend using the > module name if you had one, and if not then the file name was your > best bet. You chose to ignore part of my statement to focus on the > part you disagreed with, because you thought what I had suggested in > the first place was a better idea. I pointed out this oversight on > your part, and you've been dwelling on it ever since. I don't see the history of this thread the same way as you do, apparently. I first posted to this thread in a post identified as: """ Date: Thu, 17 Nov 2005 17:28:56 -0800 Message-ID: <[EMAIL PROTECTED]> """ and later replied to a post of yours identified as: """ Date: Thu, 17 Nov 2005 23:27:45 -0500 Message-ID: <[EMAIL PROTECTED]> """ i.e., your post, to which I was replying, was about 3 hours later than my original one. My original post said: > BEFORE the text you're commenting on?!), but anyway I think that using > the class's __module__ rather than __file__ should be what you want. so it seems incorrect to say that you had "suggested in the first place" the idea of using __module__; on the contrary, in a previous post of yours, the first one of yours on this thread, specifically: """ Date: Thu, 17 Nov 2005 19:10:39 -0500 Message-ID: <[EMAIL PROTECTED]> """ your ENTIRE suggestion -- what you "suggested in the first place" -- was the entirely wrong: > How about adding Foo.__file__ to the serialized data? So, it seems to me that you're the one dwelling on the issue, because you made the original wrong suggestion, and after I corrected it, repeated and are now defending the gist of it (though not the obviously wrong idea of using a non-existent __file__ attribute of the Foo class). > Frankly, I thought you were brighter than that, have no idea why > you're acting like this, and frankly don't think you need a tutor. I > figured this out by fooling around with the interpreter before posting > in the first place, you certainly know enough to do the same. "Before posting in the first place" (your post at 19:10 EST) you clearly had not tested your suggestion (nor had you claimed to, until now). In any case, if your specific idea is that mucking around with the __file__ attribute of the __main__ module is a reasonable way to solve the conumdrum of marshaling instances of classes defined in __main__ (which, as per the first paragraph of this post, I dispute) it would have been vastly preferable to say so in the first place, rather than expressing this particular case as just an "If not" (wrt "You can use the module name if you have it available") -- particularly since you DO have the module name available for classes defined in __main__, it's just suspicious and risky to use it in marshaling. Incidentally, pickle DOES just use the module name in marshaling even in this case -- the opposite tack to what you suggest, and, in its own special way, almost as wrong, IMHO (just guessing the other way, in the presence of obvious ambiguity). Still, pickle does take ONE precaution worth noticing: it checks that the class it's marshaling is, in fact, the same object that's available under that class's name from the module it belongs to -- otherwise (e.g., for classes dynamically defined inside a function), it raises a PicklingError. As a general point, refusing to marshal data that strongly smells like it will be hard or unlikely to unmarshal later is commendable prudence, and I would recommend this general, prudent approach strongly. Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
[EMAIL PROTECTED] (Alex Martelli) writes: > Mike Meyer <[EMAIL PROTECTED]> wrote: >... >> > How do you arrange a module so that its classes' __module__ attributes >> > don't tell you the name of the module "that would be useful", yet the >> > module's __file__ DOES give you information that you can usefully >> > process, heuristically I assume, to infer a module name that IS useful? >> So what module name do you import if C.__module__ is __main__? > In this case, I would strongly suggest that raising an exception upon > the attempt to marshal is better than trying to guess that the file > holding the main script could later be innocuously imported from a > different script, or, alternatively, that the same main script would be > running when a future unmarshaling attempt is done. "In the face of > ambiguity, refuse the temptation to guess" is a sound principle. Once you move the data to another system, pretty much anything you do to try and figure out how to load the code that defines a class is ambiguous. After all, it may not be present at all. The only way to avoid ambiguity is to serialize the code. >> > I just don't see it, which of course doesn't mean it can't be done, with >> > sufficient evil ingenuity. But since you're the one arguing that this >> > case is important enough to be worth dwelling on, >> I'm not dwelling on it, you are. All I did was recommend using the >> module name if you had one, and if not then the file name was your >> best bet. You chose to ignore part of my statement to focus on the >> part you disagreed with, because you thought what I had suggested in >> the first place was a better idea. I pointed out this oversight on >> your part, and you've been dwelling on it ever since. > I don't see the history of this thread the same way as you do, > apparently. Given that this is netnews, that's not at *all* surprising. > i.e., your post, to which I was replying, was about 3 hours later than > my original one. My original post said: Ok, so you suggested it first. Congratulations. I didn't see them in that order. I didn't reply to that post of yours. I replied to the OP pointing out that the problem with __file__ being system-dependent. He never suggested using module. > so it seems incorrect to say that you had "suggested in the first place" > the idea of using __module__; on the contrary, in a previous post of > yours, the first one of yours on this thread, specifically: >> How about adding Foo.__file__ to the serialized data? > So, it seems to me that you're the one dwelling on the issue, because > you made the original wrong suggestion, and after I corrected it, > repeated and are now defending the gist of it (though not the obviously > wrong idea of using a non-existent __file__ attribute of the Foo class). I never did anything to defend it. As soon as the OP pointed out that he'd didn't like __file__ because of the loss of system independence, I suggested using the module, and mentioned that he might not have a better choice than to use __file__ anyway. You overlooked the mention of __module__ to complain that __file__ wasn't his best choice, because he could use __module__. All I did was point out what you overlooked, and have been answering your questions about it ever since. If you don't think __file__ is such a good idea, you should have said so in the first place. Instead, you asked questions about it. Trying to help you out, I answered them. If trying to help someone who seems to be having trouble understanding an issue is "dwelling on the issue", well, I'm glad we have a newsgroup full of people who dwell on issues. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
Mike Meyer <[EMAIL PROTECTED]> wrote: ... > I suggested using the module, and mentioned that he might not have a > better choice than to use __file__ anyway. You overlooked the mention > of __module__ to complain that __file__ wasn't his best choice, > because he could use __module__. All I did was point out what you > overlooked, and have been answering your questions about it ever Sorry to have given the impression that I overlooked something -- I didn't KNOW whether I had, which is why I checked back with you about it, but apparently I hadn't. > since. If you don't think __file__ is such a good idea, you should > have said so in the first place. And I did! My exact words in my first post to this thread were "I think that using the class's __module__ rather than __file__ should be what you want" -- a pretty clear indication that I don't think using __file__ is a hot idea, don't you think? > Instead, you asked questions about About what would be the cases where the module's name was not available, as you had claimed it could be; turns out that you haven't given a single example yet, just one (__main__) where the module name is perfectly available, though it may not be *useful* (and there's no special reason to guess that the *file*'s name would be any use either). > it. Trying to help you out, I answered them. If trying to help someone > who seems to be having trouble understanding an issue is "dwelling on > the issue", well, I'm glad we have a newsgroup full of people who > dwell on issues. You did help me to better understand some of the roots of your many mistaken assertions in this thread, from your first "How about adding Foo.__file__" proposal onwards, yes -- thank you. However, I am now reasonably convinced that your attempt to *defend*, at some level, those assertions, appear to have no sound technical basis, just a defensiveness that (in my mind) justifies the use of the word "dwelling". But we do fully agree that trying to be helpful, per se, is never inherently blameworthy, whether one succeeds or fails in the attempt -- so, thanks for your many (and mostly better based than this one) attempts to be of help to c.l.py posters. Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Importing a class without knowing the module
[EMAIL PROTECTED] (Alex Martelli) writes: > You did help me to better understand some of the roots of your many > mistaken assertions in this thread, from your first "How about adding > Foo.__file__" proposal onwards, yes -- thank you. I claim I haven't made a single mistaken assertion in this thread. Referencing class.__file__ instead of module.__file__ was a mistake, but not an assertion. In fact, when I did that, I asked the OP what was wrong with it. You have read into some of the things I've written assertions which simply weren't there. For instance, I said "you may not have the module name", which led you to ask for an example of a class without an __module__ attribute. I never asserted that you could get a class without a __module__ attribute, merely that you could get classes where the module name wasn't useful. if you had spent your time reading the text a bit more carefully, you might have saved us both some time. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list