Hi Claudio, I'm working with David and lead up our provisioning team. We did eventually finish our migration of the 84,000 groups, but the rate dropped off again on day 4 to 1 transaction per 20 seconds per thread. We had to increase from 5 drivers/threads to 10 in order to finish in the nick of time. A week later we attempted to use the provisioning API's to move 4500 users from one sub-org to another to support their move to Google mail and calendar and migration of their existing data. The response times from the Google API servers were once again abissmal. We were processing 1 transaction per 5 seconds per thread. We see delays of 1.5-2 seconds in response times to simple API calls like retrieveCustomerId, and 4 or more seconds to updateOrganizationUser. Some sanitized log file examples below to show the gaps in our timestamps as our process sits idle waiting for a response from the Google API servers:
[05/10/12 04:47:58.294]:GA1 ST:appsForYourDomain.retrieveCustomerId: sDomain = umich.edu [05/10/12 04:48:00.452]:GA1 ST:appsForYourDomain.updateOrganizationUser: CustomerId = ********************* [05/10/12 04:48:00.453]:GA1 ST:appsForYourDomain.updateOrganizationUser: NewOrgUnitPath = All Services [05/10/12 04:48:04.551]:GA1 ST:addHandler: attr-name == 'GivenName' How can we verify that this was due to overloaded servers or being caused by something else, perhaps something we can correct or compensate for locally? If we do verify it was overloaded servers, for 5 days, what can be done to shore up Google's provisioning API infrastructure and is that being done? We have other large migrations to complete before we are fully deployed and we need to find a way of predicting and protecting the throughput of our provisioning processes or our deployments will continue to be plagued with these issues and Google's image will suffer here at the University of Michigan. Thank you for any help you can provide us. Luke Tracy Technical Manager Identity and Access Management Universtiy of Michigan On Friday, May 4, 2012 2:36:43 PM UTC-4, Claudio Cherubino wrote: > > I'll need to double check, but in general Google Data API requests' limits > are per user, so using different admins should allow to work around them. > > Claudio > > On Fri, May 4, 2012 at 11:31 AM, David Spangler <[email protected]>wrote: > >> Also, does it matter if the threads are using different admin IDs? Should >> we be using different admin IDs for each thread? >> >> >> On Friday, May 4, 2012 1:24:49 PM UTC-5, wrote: >>> >>> There were no errors reported in the driver. >>> >>> On Friday, May 4, 2012 1:21:31 PM UTC-5, Claudio Cherubino wrote: >>>> >>>> I don't think we were specifically throttling your app, but instead it >>>> might have been that our servers were overloaded. >>>> Did you get any specific error message? We'll send you 503s if it is >>>> your app that needs to slow down the requests. >>>> >>>> Claudio >>>> >>>> On Fri, May 4, 2012 at 11:19 AM, wrote: >>>> >>>>> There was a period where the 5 drivers were working at the 1 >>>>> transaction per second (for 4 hours), then there was a drop-off in speed >>>>> (by half). Does this indicate that we reached some limit? If we add >>>>> more >>>>> threads will it also work properly for some time until another limit is >>>>> reached? >>>>> >>>>> >>>>> On Friday, May 4, 2012 1:06:43 PM UTC-5, Claudio Cherubino wrote: >>>>> >>>>>> Hi David, >>>>>> >>>>>> All limits are per thread so adding more threads should definitely >>>>>> increase the throughput. >>>>>> Just be prepared to slow down using exponential backoff in case you >>>>>> get a 503 response code. >>>>>> For more details, check our Usage Limits page in the docs: >>>>>> >>>>>> https://developers.google.com/****google-apps/provisioning/**limit**s<https://developers.google.com/google-apps/provisioning/limits> >>>>>> >>>>>> Claudio >>>>>> >>>>>> >>>>>> In addition, we are interested in whether or not adding another 5 >>>>>>> threads will increase the overall rates, i.e. is the process for >>>>>>> provisioning Groups using the same API actions as "regular" user >>>>>>> provisioning, hitting the same services that might be affecting the >>>>>>> limits? >>>>>>> >>>>>>> In addition to the existing 5 threads, there are 5 threads that are >>>>>>> dedicated to user provisioning and 1 for password sync. Are these >>>>>>> additive? In other words, we would be going from 11 to 16 threads >>>>>>> overall, >>>>>>> would this help? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Friday, May 4, 2012 10:30:14 AM UTC-5, David Spangler wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I have a situation where we are attempting to populate around >>>>>>>> 90,000 groups using 5 different threads against the Provisioning API. >>>>>>>> When >>>>>>>> we started at 5pm last night, we had "expected" throughput of around >>>>>>>> 200-300 users per minute processed. After 4 hours, the rates dropped >>>>>>>> more >>>>>>>> than in half. Did we reach an API limit? If so, is it possible to >>>>>>>> request >>>>>>>> a lift of this limit temporarily until all the Groups are provisioned? >>>>>>>> >>>>>>>> Attached is a graph of what last night looked like. >>>>>>>> >>>>>>>> Thanks for looking at this. >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Google Apps Domain Information and Management APIs" group. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/**ms**g/google-apps-mgmt-apis/-/**MSD9** >>>>>>> rGLQ6AcJ<https://groups.google.com/d/msg/google-apps-mgmt-apis/-/MSD9rGLQ6AcJ> >>>>>>> . >>>>>>> >>>>>>> To post to this group, send email to google-apps-mgmt-apis@** >>>>>>> googlegr**oups.com <[email protected]>. >>>>>>> To unsubscribe from this group, send email to google-apps-mgmt-apis+ >>>>>>> **unsubscr**[email protected]<google-apps-mgmt-apis%[email protected]> >>>>>>> . >>>>>>> For more options, visit this group at http://groups.google.com/** >>>>>>> group**/google-apps-mgmt-apis?**hl=en<http://groups.google.com/group/google-apps-mgmt-apis?hl=en> >>>>>>> . >>>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google Apps Domain Information and Management APIs" group. >>>>> To view this discussion on the web visit https://groups.google.com/d/* >>>>> *msg/google-apps-mgmt-apis/-/**lYs7PleFp4EJ<https://groups.google.com/d/msg/google-apps-mgmt-apis/-/lYs7PleFp4EJ> >>>>> . >>>>> >>>>> To post to this group, send email to google-apps-mgmt-apis@** >>>>> googlegroups.com <[email protected]>. >>>>> To unsubscribe from this group, send email to google-apps-mgmt-apis+** >>>>> [email protected]<google-apps-mgmt-apis%[email protected]> >>>>> . >>>>> For more options, visit this group at http://groups.google.com/** >>>>> group/google-apps-mgmt-apis?**hl=en<http://groups.google.com/group/google-apps-mgmt-apis?hl=en> >>>>> . >>>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Google Apps Domain Information and Management APIs" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-apps-mgmt-apis/-/IU4gwh0GW_4J. >> >> To post to this group, send email to >> [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-apps-mgmt-apis?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/ux7ov0LJkfAJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-apps-mgmt-apis?hl=en.
