[
https://issues.apache.org/jira/browse/LIBCLOUD-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924312#comment-13924312
]
Russell Alexander Spitzer commented on LIBCLOUD-527:
----------------------------------------------------
Np :) I didn't notice it until I called list_nodes when I had 100 instances up.
My solution in the branch above is to remove the setting of
extra['boot_volume'] from to_node and move it into list_nodes, and ex_get_node.
In get node we do an additional call for that volume's info and attach it so
the behavior is the same. For list_nodes we call list_volumes and match each
boot volume from that request with it's instance.
For list_volumes I noticed that we actually have the zone info before calling
_to_storage_volume so I just apply the zone info without the additional
request. When no zone is supplied we make one request for each zone present as
opposed to once per volume.
> GCE Libcloud Driver list_nodes and list_volumes Scaling Improvement
> -------------------------------------------------------------------
>
> Key: LIBCLOUD-527
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-527
> Project: Libcloud
> Issue Type: Improvement
> Affects Versions: 0.14.0-beta3
> Reporter: Russell Alexander Spitzer
>
> The current code for the GCE Driver has two significant bottle necks in the
> list_nodes and list_volumes code. In both cases the slowdown can be traced to
> the response to object methods _to_node and _to_storage_volume. These methods
> are called in serial from both list_node and list_volume methods and each
> call to one of these methods requires an additional http request. For
> list_nodes the price is doubled since each _to_node request requires a
> _to_storage_volume request.
> Basically, list_nodes and list_volumes require 1 + 2 * Nodes and 1 + Volumes
> requests to complete. I've found that at large numbers of nodes this is a
> considerable pain point.
> I've been working on a branch at
> https://github.com/RussellSpitzer/libcloud/tree/GCE_perf_to_node
> that I would be happy to contribute upstream if there is any desire for it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
