Re: [google-appengine] Re: Google App Engine is slow to deploy, hangs on "Updating service [someproject]..."

[Shiva Wu]

So I guess no update on the load balancer update speed, huh? It has been 
more than a year...

On Wednesday, 5 April 2017 16:28:11 UTC-7, Nick (Cloud Platform Support) 
wrote:
>
> Hey Stanislas,
>
> So, I can confirm that my prior investigation and intuitions were on the 
> right track. I can confirm that a majority of deployment lag comes from 
> programming the Google Cloud Load Balancers (GCLB). This is what we've seen 
> in this case. Updates have to go out across the entire infrastructure while 
> still respecting certain locks used to keep configurations consistent. 
> These Load Balancers are not visible to users in the Console and not 
> user-configurable as they're infrastructural and meant to be exactly as 
> they are, else users may try to modify or accidentally delete them, which 
> would cause a lot of issues.
>
> We're devoting a lot of energy to decreasing GCLB configuration push 
> times, so rest assured that our efforts in that direction should pay off 
> going forward.
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
> On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
>>
>> Great, thank you so much for your help, I'll be very interested in the 
>> details you'll get from your investigation.
>>
>> On Tue, Mar 21, 2017 at 9:22 PM 'Nick (Cloud Platform Support)' via 
>> Google App Engine > wrote:
>>
>>> Hey Stanislas,
>>>
>>> The exact explanation speculated on in my last post shouldn't be taken 
>>> as any description of what's necessarily going on, however it was an 
>>> estimate of what might be happening based on the logs observed. I'm 
>>> corresponding with experts in this area to get a more clear answer at the 
>>> moment. 
>>>
>>> You could look into deploying on Container Engine 
>>> , which would mean that the 
>>> front-end management done by the App Engine Flexible Environment 
>>> infrastructure wouldn't be happening, rather it would be the responsibility 
>>> of the resources you deploy on Container Engine (a managed service based 
>>> pretty transparently on Kubernetes ). Surely 
>>> deploying new container images to your pool of instances in a cluster (or 
>>> multiple clusters) would be quite fast, since the master sitting in front 
>>> of your clusters and the nodes in the cluster are not as massively 
>>> distributed as the App Engine Flexible Environment serving infrastructure, 
>>> hence updating their routing rules would be relatively fast. This is 
>>> something to look into and experiment with if you don't want to wait on the 
>>> more detailed word from experts, but don't rush to that if you're not all 
>>> that curious.
>>>
>>> I'll get back to this thread with more details when they're forthcoming 
>>> in our investigation.
>>>
>>>
>>> Cheers,
>>>
>>> Nick
>>> Cloud Platform Community Support 
>>>
>>> On Tuesday, March 21, 2017 at 3:50:34 PM UTC-4, Stanislas Marion wrote:

 Hi Nick,
 Thanks a lot for the lengthy explanation. 
 In this light, is there anything I can do to speed things up? Like for 
 instance take care of the load-balancer myself? Indeed I don't see a 
 reason 
 why it should need to be changed. Could I do this with GAE or would I have 
 to move to G Container/Compute E?
 Cheers,

 On Tue, Mar 21, 2017 at 8:42 PM 'Nick (Cloud Platform Support)' via 
 Google App Engine > wrote:

> Hey Stanislas,
>
> My initial hunch was that the issue was the deployment of other 
> resources necessary to support the containers running. My analysis of 
> deployment-related logs appears to confirm this:
>
> I created a simple NodeJS app using your dockerfile and default.yaml. 
> I then pushed the docker image to gcr.io and ran "gcloud app deploy 
> --image-url ..."
>
> After about 1 minute of waiting, all resources associated with the 
> deployment had apparently completed, but the command had not returned yet:
>
> ```
> $ gcloud deployment-manager resources list --deployment 
> aef-default-20170321t185300
>
> NAME   TYPE   
>   STATE  ERRORS  INTENT
> aef-default-20170321t185300-00 
> compute.beta.regionInstanceGroupManager  COMPLETED  []
> aef-default-20170321t185300-00ahs  compute.v1.httpsHealthCheck 
>  COMPLETED  []
> aef-default-20170321t185300-00it   compute.v1.instanceTemplate 
>  COMPLETED  []
> aef-default-20170321t185300-bs compute.v1.backendService   
>  COMPLETED  []
> aef-default-20170321t185300-hcfw   compute.v1.firewall 
>  COMPLETED  []
> aef-default-20170321t185300-hcscompute.v1.httpsHealthCheck 
>  COMPLETED  []
> ```
>
> At around the same time that the last of the above completed, I see 
> the following in the Console logs

Re: [google-appengine] Re: Google App Engine is slow to deploy, hangs on "Updating service [someproject]..."

[Shiva Wu]

So I guess no update on the GCLB deployment speed, huh? It has been more 
than a year...

On Wednesday, 5 April 2017 16:28:11 UTC-7, Nick (Cloud Platform Support) 
wrote:
>
> Hey Stanislas,
>
> So, I can confirm that my prior investigation and intuitions were on the 
> right track. I can confirm that a majority of deployment lag comes from 
> programming the Google Cloud Load Balancers (GCLB). This is what we've seen 
> in this case. Updates have to go out across the entire infrastructure while 
> still respecting certain locks used to keep configurations consistent. 
> These Load Balancers are not visible to users in the Console and not 
> user-configurable as they're infrastructural and meant to be exactly as 
> they are, else users may try to modify or accidentally delete them, which 
> would cause a lot of issues.
>
> We're devoting a lot of energy to decreasing GCLB configuration push 
> times, so rest assured that our efforts in that direction should pay off 
> going forward.
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
> On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote:
>>
>> Great, thank you so much for your help, I'll be very interested in the 
>> details you'll get from your investigation.
>>
>> On Tue, Mar 21, 2017 at 9:22 PM 'Nick (Cloud Platform Support)' via 
>> Google App Engine > wrote:
>>
>>> Hey Stanislas,
>>>
>>> The exact explanation speculated on in my last post shouldn't be taken 
>>> as any description of what's necessarily going on, however it was an 
>>> estimate of what might be happening based on the logs observed. I'm 
>>> corresponding with experts in this area to get a more clear answer at the 
>>> moment. 
>>>
>>> You could look into deploying on Container Engine 
>>> , which would mean that the 
>>> front-end management done by the App Engine Flexible Environment 
>>> infrastructure wouldn't be happening, rather it would be the responsibility 
>>> of the resources you deploy on Container Engine (a managed service based 
>>> pretty transparently on Kubernetes ). Surely 
>>> deploying new container images to your pool of instances in a cluster (or 
>>> multiple clusters) would be quite fast, since the master sitting in front 
>>> of your clusters and the nodes in the cluster are not as massively 
>>> distributed as the App Engine Flexible Environment serving infrastructure, 
>>> hence updating their routing rules would be relatively fast. This is 
>>> something to look into and experiment with if you don't want to wait on the 
>>> more detailed word from experts, but don't rush to that if you're not all 
>>> that curious.
>>>
>>> I'll get back to this thread with more details when they're forthcoming 
>>> in our investigation.
>>>
>>>
>>> Cheers,
>>>
>>> Nick
>>> Cloud Platform Community Support 
>>>
>>> On Tuesday, March 21, 2017 at 3:50:34 PM UTC-4, Stanislas Marion wrote:

 Hi Nick,
 Thanks a lot for the lengthy explanation. 
 In this light, is there anything I can do to speed things up? Like for 
 instance take care of the load-balancer myself? Indeed I don't see a 
 reason 
 why it should need to be changed. Could I do this with GAE or would I have 
 to move to G Container/Compute E?
 Cheers,

 On Tue, Mar 21, 2017 at 8:42 PM 'Nick (Cloud Platform Support)' via 
 Google App Engine > wrote:

> Hey Stanislas,
>
> My initial hunch was that the issue was the deployment of other 
> resources necessary to support the containers running. My analysis of 
> deployment-related logs appears to confirm this:
>
> I created a simple NodeJS app using your dockerfile and default.yaml. 
> I then pushed the docker image to gcr.io and ran "gcloud app deploy 
> --image-url ..."
>
> After about 1 minute of waiting, all resources associated with the 
> deployment had apparently completed, but the command had not returned yet:
>
> ```
> $ gcloud deployment-manager resources list --deployment 
> aef-default-20170321t185300
>
> NAME   TYPE   
>   STATE  ERRORS  INTENT
> aef-default-20170321t185300-00 
> compute.beta.regionInstanceGroupManager  COMPLETED  []
> aef-default-20170321t185300-00ahs  compute.v1.httpsHealthCheck 
>  COMPLETED  []
> aef-default-20170321t185300-00it   compute.v1.instanceTemplate 
>  COMPLETED  []
> aef-default-20170321t185300-bs compute.v1.backendService   
>  COMPLETED  []
> aef-default-20170321t185300-hcfw   compute.v1.firewall 
>  COMPLETED  []
> aef-default-20170321t185300-hcscompute.v1.httpsHealthCheck 
>  COMPLETED  []
> ```
>
> At around the same time that the last of the above completed, I see 
> the following in the Console logs when