Re: Inner imports in core code
On Sep 24, 2008, at 18:27, Alex Myodov wrote: > Being curious, had tried several benchmarks as well. > Indeed, the function-level imports have some overhead. But the > overhead becomes less and less important, as long as the function > complexity increases. If a function does nothing except import (as in > your case), the performance hit is several hundred percents. If a > function does at least a crypto hash calculation, the performance loss > is about 25% (~7.3 seconds versus 5.4 seconds for 1 million > iterations). If a function does three different hash calculations, the > performance loss is mere 6%: 26.5 seconds vs 24.8 (with the same > conditions). > So yes, the in-function imports have their cost. On the other hand, if > different functions in a module import different other modules, there > will be drawbacks (for moving the imports to the head of the module) > too: with the function-level imports, we import only the modules we > need, and only when we need them. With the module-level imports, we > import everything (any module function ever needs) when calling any > function from the module. Light startup/heavy function calls in the > first case; heavy startup/light function calls in the second one. > So, neither approach is a silver bullet. The consideration is needed > for any module/function to decide what is better for them. For the case where the imports are all at module level (impossible), it will mean that all of Django will be imported at once - which is much more desirable in production. But as you say, the actual performance hit is negligible for either case. The one actual benefit of module-level imports is that it results in a kind of startup code validation - as everything is imported early, errors will surface early, not during say a URL reverse operation. -- Ludvig --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Inner imports in core code
Being curious, had tried several benchmarks as well.
Indeed, the function-level imports have some overhead. But the
overhead becomes less and less important, as long as the function
complexity increases. If a function does nothing except import (as in
your case), the performance hit is several hundred percents. If a
function does at least a crypto hash calculation, the performance loss
is about 25% (~7.3 seconds versus 5.4 seconds for 1 million
iterations). If a function does three different hash calculations, the
performance loss is mere 6%: 26.5 seconds vs 24.8 (with the same
conditions).
So yes, the in-function imports have their cost. On the other hand, if
different functions in a module import different other modules, there
will be drawbacks (for moving the imports to the head of the module)
too: with the function-level imports, we import only the modules we
need, and only when we need them. With the module-level imports, we
import everything (any module function ever needs) when calling any
function from the module. Light startup/heavy function calls in the
first case; heavy startup/light function calls in the second one.
So, neither approach is a silver bullet. The consideration is needed
for any module/function to decide what is better for them.
On 24 сент, 11:37, David Cramer <[EMAIL PROTECTED]> wrote:
> I was digging through some code today, and I noticed imports are
> happening within a lot of functions. It was my knowledge that it works
> like so:
>
> import in a function is the same as ruby's load or php's include --
> its executed everytime the function is
>
> import in a module outside a function is like ruby's require or php's
> require_once -- its loaded and cached in memory
>
> If this is the case, then this seems like an unneeded performance hits
> in many situations.
>
> An example off hand, is the get_hexdigest() function of the auth
> module:http://code.djangoproject.com/browser/django/trunk/django/contrib/aut...
>
> It waits to import the hashing methods, which if you're in a pooled
> environment, it could save a lot by onlying importing it once,
> correct?
>
> Here's my benchmark script:
>
> import timeit
> def without_import():
> pass
> def with_import():
> import random
>
> def main():
> print "Without an inner import:",
> print timeit.Timer("module_loading.without_import()", "import
> module_loading, random").timeit()
> print "With an inner import:",
> print timeit.Timer("module_loading.with_import()", "import
> module_loading").timeit()
>
> if __name__ == '__main__':
> main()
>
> And the result:
>
> Without an inner import: 1.28283500671
> With an inner import: 5.402146101
>
> If everyone's in agreement, I'd like to approach correcting this in
> Django
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Inner imports in core code
On Sep 24, 2008, at 14:52, Simon Willison wrote:
> Not entirely sure how I managed to miss your benchmark figures when I
> read your mail, but I'm wrong here - there's clearly a performance
> overhead involved in importing inside a function. It would be
> interesting to see how much this affects Django performance as a whole
> - I guess this is the kind of question the proposed metronome
> benchmark will help answer.
As you said yourself, it is correct that Python does cache imported
modules.
However, the Python VM still has to run the bytecode produced, which is:
>>> import dis
>>> def x():
... from sys import stdout
... print stdout,
...
>>> dis.dis(x)
2 0 LOAD_CONST 1 (-1)
3 LOAD_CONST 2 (('stdout',))
6 IMPORT_NAME 0 (sys)
9 IMPORT_FROM 1 (stdout)
12 STORE_FAST 0 (stdout)
15 POP_TOP
3 16 LOAD_FAST0 (stdout)
19 PRINT_ITEM
20 LOAD_CONST 0 (None)
23 RETURN_VALUE
As you see, -1 and a tuple are pushed onto the stack, then two bytecodes
execute: IMPORT_NAME: This is Python importing sys - in Django's case,
this'll mean at the very least a dict lookup, in sys.modules. Thence,
IMPORT_FROM: retrieve a name from the module on the stack.
These two take care of the object fetching, then a STORE_FAST is run to
bind the object to a name (which, since no "as foo" was given, coincides
with the object's name in the module.)
What's a little interesting is this snippet, though - didn't expect it:
12 STORE_FAST 0 (stdout)
15 POP_TOP
3 16 LOAD_FAST0 (stdout)
It is exactly equivalent to:
12 STORE_FAST 0 (stdout)
(The last three are the print and implicit "return None".)
So yes, inline imports do have their overhead as Python has to fetch and
bind the name every time.
-- Ludvig
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Inner imports in core code
On Sep 24, 1:40 pm, Simon Willison <[EMAIL PROTECTED]> wrote: > As a result the performance overhead from having imports inside > functions as opposed to at module level should be virtually non- > existent. Not entirely sure how I managed to miss your benchmark figures when I read your mail, but I'm wrong here - there's clearly a performance overhead involved in importing inside a function. It would be interesting to see how much this affects Django performance as a whole - I guess this is the kind of question the proposed metronome benchmark will help answer. Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Inner imports in core code
On Sep 24, 8:37 am, David Cramer <[EMAIL PROTECTED]> wrote: > I was digging through some code today, and I noticed imports are > happening within a lot of functions. It was my knowledge that it works > like so: > > import in a function is the same as ruby's load or php's include -- > its executed everytime the function is > > import in a module outside a function is like ruby's require or php's > require_once -- its loaded and cached in memory That's not the case. Try the following: # importme.py print "running importme" def three(): print "3 - from importme" # test1.py import importme importme.three() # test2.py def one(): import importme print "1" importme.three() def two(): import importme print "2" importme.three() one() two() $ python test1.py running importme 3 - from importme $ python test2.py running importme 1 3 - from importme 2 3 - from importme As you can see, the initial print statement in importme.py only executes once, even when "import importme" is run twice (from inside functions). As I understand it, Python's "import" statement checks sys.modules to see if something has been imported already. If it hasn't, the module is loaded for the first time and executed. If it has already been imported (and hence is in sys.modules) then no extra code is loaded or executed - the import statement merely ensures that a reference to that module is available in the function's scope. As a result the performance overhead from having imports inside functions as opposed to at module level should be virtually non- existent. I don't think we need to worry about this - and like Jeremy said, usually there's a reason for doing it (avoiding importing something unless it's actually going to be needed, or avoiding circular imports). Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Inner imports in core code
Note that in some cases, dynamic imports are done to avoid loading a
Django subsystem, large external library, or optional dependency until
it is strictly necessary.
There are certainly still examples where none of those good reasons
apply.
On Sep 24, 2008, at 2:37 AM, David Cramer <[EMAIL PROTECTED]> wrote:
>
> I was digging through some code today, and I noticed imports are
> happening within a lot of functions. It was my knowledge that it works
> like so:
>
> import in a function is the same as ruby's load or php's include --
> its executed everytime the function is
>
> import in a module outside a function is like ruby's require or php's
> require_once -- its loaded and cached in memory
>
> If this is the case, then this seems like an unneeded performance hits
> in many situations.
>
> An example off hand, is the get_hexdigest() function of the auth
> module:
> http://code.djangoproject.com/browser/django/trunk/django/contrib/auth/models.py
>
> It waits to import the hashing methods, which if you're in a pooled
> environment, it could save a lot by onlying importing it once,
> correct?
>
> Here's my benchmark script:
>
> import timeit
> def without_import():
>pass
> def with_import():
>import random
>
>
> def main():
>print "Without an inner import:",
>print timeit.Timer("module_loading.without_import()", "import
> module_loading, random").timeit()
>print "With an inner import:",
>print timeit.Timer("module_loading.with_import()", "import
> module_loading").timeit()
>
> if __name__ == '__main__':
>main()
>
> And the result:
>
> Without an inner import: 1.28283500671
> With an inner import: 5.402146101
>
> If everyone's in agreement, I'd like to approach correcting this in
> Django
> >
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
