Hey Nick! How are these devoted efforts coming along? I just started a brand new GAE project with a simple Node application. Deployment takes 8-10 mins, the major part of which is still spent waiting for "Updating service". I'm not particularly impressed.
Cheers, Magnus Den torsdag 6 april 2017 kl. 01:28:11 UTC+2 skrev Nick (Cloud Platform Support): > > We're devoting a lot of energy to decreasing GCLB configuration push > times, so rest assured that our efforts in that direction should pay off > going forward. > > Cheers, > > Nick > Cloud Platform Community Support > > On Tuesday, March 21, 2017 at 6:41:52 PM UTC-4, Stanislas Marion wrote: >> >> Great, thank you so much for your help, I'll be very interested in the >> details you'll get from your investigation. >> >> On Tue, Mar 21, 2017 at 9:22 PM 'Nick (Cloud Platform Support)' via >> Google App Engine <google-a...@googlegroups.com <javascript:>> wrote: >> >>> Hey Stanislas, >>> >>> The exact explanation speculated on in my last post shouldn't be taken >>> as any description of what's necessarily going on, however it was an >>> estimate of what might be happening based on the logs observed. I'm >>> corresponding with experts in this area to get a more clear answer at the >>> moment. >>> >>> You could look into deploying on Container Engine >>> <https://cloud.google.com/container-engine/>, which would mean that the >>> front-end management done by the App Engine Flexible Environment >>> infrastructure wouldn't be happening, rather it would be the responsibility >>> of the resources you deploy on Container Engine (a managed service based >>> pretty transparently on Kubernetes <https://kubernetes.io/>). Surely >>> deploying new container images to your pool of instances in a cluster (or >>> multiple clusters) would be quite fast, since the master sitting in front >>> of your clusters and the nodes in the cluster are not as massively >>> distributed as the App Engine Flexible Environment serving infrastructure, >>> hence updating their routing rules would be relatively fast. This is >>> something to look into and experiment with if you don't want to wait on the >>> more detailed word from experts, but don't rush to that if you're not all >>> that curious. >>> >>> I'll get back to this thread with more details when they're forthcoming >>> in our investigation. >>> >>> >>> Cheers, >>> >>> Nick >>> Cloud Platform Community Support >>> >>> On Tuesday, March 21, 2017 at 3:50:34 PM UTC-4, Stanislas Marion wrote: >>>> >>>> Hi Nick, >>>> Thanks a lot for the lengthy explanation. >>>> In this light, is there anything I can do to speed things up? Like for >>>> instance take care of the load-balancer myself? Indeed I don't see a >>>> reason >>>> why it should need to be changed. Could I do this with GAE or would I have >>>> to move to G Container/Compute E? >>>> Cheers, >>>> >>>> On Tue, Mar 21, 2017 at 8:42 PM 'Nick (Cloud Platform Support)' via >>>> Google App Engine <google-a...@googlegroups.com <javascript:>> wrote: >>>> >>>>> Hey Stanislas, >>>>> >>>>> My initial hunch was that the issue was the deployment of other >>>>> resources necessary to support the containers running. My analysis of >>>>> deployment-related logs appears to confirm this: >>>>> >>>>> I created a simple NodeJS app using your dockerfile and default.yaml. >>>>> I then pushed the docker image to gcr.io and ran "gcloud app deploy >>>>> --image-url ..." >>>>> >>>>> After about 1 minute of waiting, all resources associated with the >>>>> deployment had apparently completed, but the command had not returned yet: >>>>> >>>>> ``` >>>>> $ gcloud deployment-manager resources list --deployment >>>>> aef-default-20170321t185300 >>>>> >>>>> NAME TYPE >>>>> STATE ERRORS INTENT >>>>> aef-default-20170321t185300-00 >>>>> compute.beta.regionInstanceGroupManager COMPLETED [] >>>>> aef-default-20170321t185300-00ahs compute.v1.httpsHealthCheck >>>>> COMPLETED [] >>>>> aef-default-20170321t185300-00it compute.v1.instanceTemplate >>>>> COMPLETED [] >>>>> aef-default-20170321t185300-bs compute.v1.backendService >>>>> COMPLETED [] >>>>> aef-default-20170321t185300-hcfw compute.v1.firewall >>>>> COMPLETED [] >>>>> aef-default-20170321t185300-hcs compute.v1.httpsHealthCheck >>>>> COMPLETED [] >>>>> ``` >>>>> >>>>> At around the same time that the last of the above completed, I see >>>>> the following in the Console logs when selecting the "Deployment" logs >>>>> source: >>>>> >>>>> ``` >>>>> { >>>>> protoPayload: {…} >>>>> insertId: "54B422B11D921.AE9070D.A80D9821" >>>>> resource: { >>>>> type: "deployment" >>>>> labels: { >>>>> project_id: "<PROJECTID>" >>>>> name: "<PROJECTID>-gclb" >>>>> } >>>>> } >>>>> timestamp: "2017-03-21T18:54:05.999Z" >>>>> severity: "ERROR" >>>>> logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com >>>>> %2Factivity" >>>>> } >>>>> ``` >>>>> >>>>> I haven't expanded the protoPayload, but it shows that this was an >>>>> attempt to delete the deployment "<PROJECTID>-gclb" if it exists. It's >>>>> near-immediately followed by a log representing an update on the >>>>> deployment >>>>> "<PROJECTID>-gclb" with createPolicy: "CREATE_OR_ACQUIRE". I believe it's >>>>> reasonable to assume gclb stands for "Google Cloud Load Balancer", but I >>>>> could be wrong here. >>>>> >>>>> About six minutes later, the deployment command finally completed. At >>>>> about the same time, the following shows up in the logs: >>>>> >>>>> ``` >>>>> 18:59:35.632 >>>>> {"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":404,"message":"The >>>>> >>>>> object 'projects/<PROJECTID>/global/deployments/<PROJECTID>-gclb' is not >>>>> found."},"authenticationInfo":{"principalEmail":" >>>>> num...@cloudservices.gserviceaccount.com <javascript:> >>>>> "},"requestMetadata":{"calle... >>>>> { >>>>> protoPayload: {…} >>>>> insertId: "54B423EB72139.A6A3558.C2071C50" >>>>> resource: { >>>>> type: "deployment" >>>>> labels: { >>>>> name: "<PROJECTID>-gclb" >>>>> project_id: "<PROJECTID>" >>>>> } >>>>> } >>>>> timestamp: "2017-03-21T18:59:35.632Z" >>>>> severity: "ERROR" >>>>> logName: "projects/<PROJECTID>/logs/cloudaudit.googleapis.com >>>>> %2Factivity" >>>>> } >>>>> ``` >>>>> >>>>> This represents the deployment infrastructure checking whether the >>>>> deployment "<PROJECTID>-gclb" has been created yet, whether the create >>>>> operation scheduled above has finished yet. However, after another 25 >>>>> seconds, approximately, we see the following: >>>>> >>>>> ``` >>>>> 19:00:00.122 >>>>> {"@type":"type.googleapis.com/google.cloud.audit.AuditLog >>>>> ","authenticationInfo":{"principalEmail":" >>>>> num...@cloudservices.gserviceaccount.com >>>>> <javascript:>"},"requestMetadata":{"callerIp":"10.131.45.3","callerSuppliedUserAgent":"App >>>>> >>>>> Engine Flex Google-API-Java-Client >>>>> Google-HTTP-Java-Client/1.22.0-SNAPSHOT >>>>> (... >>>>> { >>>>> protoPayload: { >>>>> @type: "type.googleapis.com/google.cloud.audit.AuditLog" >>>>> authenticationInfo: { >>>>> principalEmail: "num...@cloudservices.gserviceaccount.com >>>>> <javascript:>" >>>>> } >>>>> requestMetadata: { >>>>> callerIp: "10.131.45.3" >>>>> callerSuppliedUserAgent: "App Engine Flex Google-API-Java-Client >>>>> Google-HTTP-Java-Client/1.22.0-SNAPSHOT (gzip)" >>>>> } >>>>> serviceName: "deploymentmanager.googleapis.com" >>>>> methodName: "v2.deploymentmanager.deployments.update" >>>>> resourceName: "projects/<PROJECTID>/global/deployments/aef-gclb" >>>>> } >>>>> insertId: "54B42402DA050.AE94741.97003020" >>>>> resource: { >>>>> type: "deployment" >>>>> labels: { >>>>> name: "aef-gclb" >>>>> project_id: "<PROJECTID>" >>>>> } >>>>> } >>>>> timestamp: "2017-03-21T19:00:00.122Z" >>>>> severity: "NOTICE" >>>>> logName: >>>>> "projects/<PROJECTID>/logs/cloudaudit.googleapis.com%2Factivity" >>>>> >>>>> operation: { >>>>> id: "operation-1490122775787-54b423eb8f5f9-5492ae56-bcfc4ec6" >>>>> producer: "deploymentmanager.googleapis.com" >>>>> last: true >>>>> } >>>>> } >>>>> ``` >>>>> >>>>> Notice "type: deployment" and "last: true". This appears to signal the >>>>> completion of a deployment "aef-gclb" at around the same time that the >>>>> command line returned news that the deployment was ready to serve. >>>>> >>>>> This all seems to point to the fact that there are some resources >>>>> which need to be deployed before the app can serve, quite apart from >>>>> fetching the container image from gcr.io. It appears that creating a >>>>> load balancer / updating its rules is an important part of this which >>>>> naturally will take a long time since a whole array of distributed >>>>> machines >>>>> in the routing infrastructure will need to sign off that they contain the >>>>> new rules before the user can be confident the deployment has completed. >>>>> >>>>> I hope this helps clarify things for you. As we continue to improve >>>>> the infrastructure, doubtless times involved in these steps will >>>>> decrease. >>>>> We definitely analyze the data related to deployments and all kinds of >>>>> other operations to determine how to streamline our systems. If you ever >>>>> notice something in future that needs explanation, feel free to post in >>>>> this group to inquire. >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Nick >>>>> Cloud Platform Community Support >>>>> >>>>> On Tuesday, March 21, 2017 at 6:00:11 AM UTC-4, Stanislas Marion wrote: >>>>>> >>>>>> Hi Nick, >>>>>> My app is based in us-central. >>>>>> My app is a very simple nodejs app, that runs an expressjs server. It >>>>>> starts in a few seconds (at most) with the command "node app.js --config >>>>>> config.json" >>>>>> I'm attaching my Dockerfile and default.yaml. >>>>>> Keep in mind that I deploy using the "gcloud app deploy default.yaml >>>>>> --image-url gcr.io/my-project/default" >>>>>> Cheers, >>>>>> >>>>>> On Mon, Mar 20, 2017 at 9:27 PM 'Nick (Cloud Platform Support)' via >>>>>> Google App Engine <google-a...@googlegroups.com <javascript:>> wrote: >>>>>> >>>>>>> Ah, apologies for sending that last message too quick - I'll also >>>>>>> need to know roughly the details of your application's contents. How >>>>>>> long >>>>>>> does it take your code to start up (you can check this by running the >>>>>>> container on your own machine, assuming it has stats comparable to the >>>>>>> deployment machine specs). Along with this information, could you share >>>>>>> your Dockerfile contents with me in response to the email I've just >>>>>>> sent >>>>>>> you from esup...@google.com <javascript:>? >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Nick >>>>>>> >>>>>>> Cloud Platform Community Support >>>>>>> >>>>>>> On Saturday, March 18, 2017 at 12:03:46 PM UTC-4, Stanislas Marion >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Nick, >>>>>>>> I have tweaked my build so that all the image building is done >>>>>>>> outside of GAE. >>>>>>>> After I have pushed my docker image to Google (using gcloud docker >>>>>>>> -- push gcr.io/...) which takes very little time, it seems like >>>>>>>> the "Updating service" step takes forever (7-8min) whereas all that is >>>>>>>> needed is a simple variant of 'docker run new_image' > `migrate >>>>>>>> traffic to >>>>>>>> new_image'. Why does that last 7-8 minutes ??? Is there any way to >>>>>>>> speed it >>>>>>>> up? It is very painful when developing. >>>>>>>> Best, >>>>>>>> >>>>>>>> On Monday, October 3, 2016 at 10:39:50 PM UTC+2, Nick (Cloud >>>>>>>> Platform Support) wrote: >>>>>>>>> >>>>>>>>> Hey Kevin, >>>>>>>>> >>>>>>>>> The speed of deployment can be changed by limiting the size of the >>>>>>>>> uploaded app, limiting the complexity of the build necessary in the >>>>>>>>> Dockerfile, if present, and by ensuring a fast and reliable internet >>>>>>>>> connection. Other than these variables, it's hard to comment on what >>>>>>>>> specifically might be making deployment exhibit the timing that it >>>>>>>>> does on >>>>>>>>> your system. >>>>>>>>> >>>>>>>>> I hope this is helpful - let me know if you have any further >>>>>>>>> questions! I'll be happy to help! >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> >>>>>>>>> Nick >>>>>>>>> Cloud Platform Community Suppor >>>>>>>>> >>>>>>>>> On Thursday, September 29, 2016 at 3:48:26 PM UTC-4, Kevin Lau >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Is there a faster way to deploy an app? Google App Engine is slow >>>>>>>>>> to deploy, hangs on "Updating service [someproject]..." I am using a >>>>>>>>>> flexible environment with PHP. >>>>>>>>>> >>>>>>>>> >>>>>>>> *"This email and any files transmitted with it are confidential and >>>>>>>> intended solely for the use of the individual or entity to whom they >>>>>>>> are >>>>>>>> addressed. If you have received this email in error please notify the >>>>>>>> sender immediately"* >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "Google App Engine" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> google-a...@googlegroups.com <javascript:>. >>>>>>> To post to this group, send email to google-a...@googlegroups.com >>>>>>> <javascript:>. >>>>>>> Visit this group at https://groups.google.com/group/google-appengine >>>>>>> . >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/google-appengine/5aa7088c-a33e-4135-974b-232adc1c4c08%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/google-appengine/5aa7088c-a33e-4135-974b-232adc1c4c08%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> *"This email and any files transmitted with it are confidential and >>>>>> intended solely for the use of the individual or entity to whom they are >>>>>> addressed. If you have received this email in error please notify the >>>>>> sender immediately"* >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Google App Engine" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> google-a...@googlegroups.com <javascript:>. >>>>> To post to this group, send email to google-a...@googlegroups.com >>>>> <javascript:>. >>>>> Visit this group at https://groups.google.com/group/google-appengine. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/google-appengine/dcfd42f7-f1da-4c0f-8adb-9e3cfdd361ca%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/google-appengine/dcfd42f7-f1da-4c0f-8adb-9e3cfdd361ca%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> *"This email and any files transmitted with it are confidential and >>>> intended solely for the use of the individual or entity to whom they are >>>> addressed. If you have received this email in error please notify the >>>> sender immediately"* >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Google App Engine" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> google-a...@googlegroups.com <javascript:>. >>> To post to this group, send email to google-a...@googlegroups.com >>> <javascript:>. >>> Visit this group at https://groups.google.com/group/google-appengine. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/google-appengine/01edf030-61cd-4bed-b4c5-3564749c05fe%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/google-appengine/01edf030-61cd-4bed-b4c5-3564749c05fe%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> *"This email and any files transmitted with it are confidential and >> intended solely for the use of the individual or entity to whom they are >> addressed. If you have received this email in error please notify the >> sender immediately"* >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/80269306-1ab5-43ba-a559-ab9fa9ba5863%40googlegroups.com.
