> import cPickle
>
> for i in range(0, 20):
> with open("forest%d.pkl" % (i), 'r') as f:
> start = datetime.now()
> a = cPickle.load(f)
> print 'loaded ', i, datetime.now() - start
>
> produce these run-time results
>
> loaded 0 0:00:14.952436
> loaded 1 0:00:15.759927
> loaded 2 0:00:15.839598
> loaded 3 0:00:14.505774
> loaded 4 0:00:15.703471
> loaded 5 0:00:15.492304
> loaded 6 0:00:16.379292
> loaded 7 0:00:17.276785
> loaded 8 0:00:17.725532
> loaded 9 0:00:16.245370
> loaded 10 0:00:12.884921
> loaded 11 0:00:15.775455
> loaded 12 0:00:14.682209
> loaded 13 0:00:16.039402
> loaded 14 0:00:19.444111
> loaded 15 0:00:14.574627
> loaded 16 0:00:16.927921
> loaded 17 0:00:18.554036
> loaded 18 0:00:13.532662
> loaded 19 0:00:18.664413
In this case, the reference to `a` is lost after each iteration, which
means that the forest can be garbage-collected and that memory can be
reused.
>
> and this code
>
> import cPickle
>
> classifier_bank = []
> for i in range(0, 20):
> with open("forest%d.pkl" % (i), 'r') as f:
> start = datetime.now()
> a = cPickle.load(f)
> classifier_bank.append(a)
> print 'loaded ', i, datetime.now() - start
>
> produce these results?
>
> loaded 0 0:00:16.561096
> loaded 1 0:00:28.319847
> loaded 2 0:00:37.514201
> loaded 3 0:00:47.548183
> loaded 4 0:00:56.997077
> loaded 5 0:01:06.473708
> loaded 6 0:01:20.373356
> loaded 7 0:01:33.540237
> loaded 8 0:01:42.579691
> loaded 9 0:01:46.615368
> loaded 10 0:01:44.872446
> loaded 11 0:02:03.572577
> loaded 12 0:02:16.641806
> loaded 13 0:02:32.068945
> loaded 14 0:02:59.249750
> loaded 15 0:02:34.015673
> loaded 16 0:03:02.650718
> loaded 17 0:03:38.124911
> loaded 18 0:03:00.707291
> loaded 19 0:03:55.910640
In contrast, in this case, forests can no longer be garbage-collected
and new memory need to be allocated at each iteration, the private heap
need to be extended and so on. In the process, I suspect that objects
are moved to one place to another, which may be the reason why it
slows down (since the number of objects in memory keeps increasing, it
takes longer and longer to move them).
Well this is just an hypothesis. Maybe I am wrong.
Gilles
------------------------------------------------------------------------------
The demand for IT networking professionals continues to grow, and the
demand for specialized networking skills is growing even more rapidly.
Take a complimentary Learning@Cisco Self-Assessment and learn
about Cisco certifications, training, and career opportunities.
http://p.sf.net/sfu/cisco-dev2dev
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
