[
https://issues.apache.org/jira/browse/LIBCLOUD-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492283#comment-13492283
]
Mahendra M commented on LIBCLOUD-254:
-------------------------------------
I have not added any test cases for the iterator_*** api because it is taken
care of by the list_*** apis. Let me know if I have to add special test cases.
> Provide generator based iteration instead of LazyList
> -----------------------------------------------------
>
> Key: LIBCLOUD-254
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-254
> Project: Libcloud
> Issue Type: Improvement
> Components: Core, Storage
> Reporter: Mahendra M
> Priority: Minor
> Attachments: libcloud-254.patch
>
>
> LazyList was implemented by issue (LIBCLOUD-78) for more efficient iteration
> over objects stored in a container (S3, CloudFiles etc. limit the maximum
> number of objects returned in a single call).
> LazyList solved this problem, but I think it might have the following issues
> while handling containers with large number of objects
> 1) It loads the entire list to memory
> 2) caller has to wait for the entire list to be loaded in memory before any
> operation can be done
> 3) The api invocation using get_more() and value_list is a bit complex
> (as-in, it can be simplified)
> By using python generators, the above problems can be alleviated. Results can
> be returned to the caller as and when it is returned from the server.
> The following changes were done to the libcloud apis
> 1) A new api called - iterate_container_objects() was introduced. The storage
> drivers need to implement this instead of list_container_objects(). This API
> now returns a generator. Usage of this API will alleviate the above three
> problems.
> 2) list_container_objects() will simply do -
> list(self.iterate_container_objects(container)) - this is maintained for
> backwards compatibility. It would be better if users can start using
> iterate_**() api instead.
> 3) The same changes have been made for the DNS base class also.
> 4) LazyList() can be removed from libcloud if it is OK with everyone.
> 5) The generator based interface can be used (WIP) for providing paginated
> access to objects - This will be useful for webpages/apps where the user has
> to paginate through the results. The same can be implemented by providing
> "start_key" and "count" parameters (similar to CouchDB) instead of generating
> the entire list and then doing an offset. This will be more
> performance/memory efficient than generating the entire list for every
> request.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
