Even on mutates that don't make it to our DEADLINE threshold we're seeing a
massive slowdown in responsiveness in mutate calls. Below, hour 20
corresponds to 4pm EST on the 22nd of June.
day hour avg(unix_timestamp(date_finished)-unix_timestamp(date_started))
22 12 10.4448
22 13 10.3481
22 14 9.4694
22 15 13.9076
22 16 13.1181
22 17 20.0910
22 18 16.3803
22 19 19.1051
22 20 38.2070
22 21 480.0347
22 22 61.2185
22 23 227.2379
On Tuesday, June 22, 2021 at 9:49:41 PM UTC-4 Pete Lavetsky (AdWords API
Guru) wrote:
> We're now seeing this on other calls as well. These are across multiple
> customerIds.
>
> 2021-06-23 01:19:33,532 WARN RequestLogger - FAILURE REQUEST SUMMARY.
> Method: google.ads.googleads.v7.services.AdService/MutateAds, Endpoint:
> googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null,
> ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after
> 3599.998632475s. [buffered_nanos=67392207, buffered_nanos=93613556,
> remote_addr=googleads.googleapis.com/142.250.191.138:443].
>
> 2021-06-23 01:22:48,579 WARN RequestLogger - FAILURE REQUEST SUMMARY.
> Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds,
> Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID:
> null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after
> 3599.985958315s. [buffered_nanos=65378596, buffered_nanos=67500777,
> remote_addr=googleads.googleapis.com/172.217.8.202:443].
>
> 20212021-06-23 01:30:08,166 WARN RequestLogger - FAILURE REQUEST SUMMARY.
> Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds,
> Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID:
> null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after
> 3599.998821696s. [buffered_nanos=66057372, buffered_nanos=77619715,
> remote_addr=googleads.googleapis.com/142.250.191.202:443].
>
> On Tuesday, June 22, 2021 at 8:23:06 PM UTC-4 Pete Lavetsky (AdWords API
> Guru) wrote:
>
>> We put V8 into production earlier today. Everything was fine for about
>> 6.5 hours and then we started seeing an increased rate of DEADLINE_EXCEEDED
>>
>> Our GrpcCallContext is configured for a 5m timeout and that's been fine
>> for months
>>
>> I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the
>> last 20 minutes but I'm not sure if that proves anything
>>
>> Is it perhaps a transient issue on Google's end for maintenance work
>> around increased capacity for V8?
>>
>> Here's the traceroute ... we've been asked to provide this before when we
>> witnessed a much higher rate of DEADLINE_EXCEEDED
>>
>> traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60
>> byte packets
>>
>> 1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104
>> ms 0.096 ms 0.102 ms
>>
>> 2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985
>> ms ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 ms
>> ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms
>>
>> 3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16)
>> 6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms
>>
>> 4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144
>> (100.66.12.144) 3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms
>>
>> 5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202
>> (100.66.14.202) 22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms
>>
>> 6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131)
>> 50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms
>>
>> 7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117)
>> 31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms
>>
>> 8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161
>> (100.65.13.161) 0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms
>>
>> 9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161
>> (15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms
>>
>> 10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10)
>> 2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms
>>
>> 11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820
>> ms 52.93.239.50 (52.93.239.50) 1.313 ms
>>
>> 12 100.92.53.124 (100.92.53.124) 12.442 ms 100.92.53.76 (100.92.53.76)
>> 14.483 ms 100.92.53.72 (100.92.53.72) 12.509 ms
>>
>> 13 100.92.43.34 (100.92.43.34) 12.711 ms 100.92.48.100 (100.92.48.100)
>> 16.661 ms 100.92.48.10 (100.92.48.10) 13.001 ms
>>
>> 14 100.92.43.85 (100.92.43.85) 12.143 ms 100.92.48.139 (100.92.48.139)
>> 12.547 ms 100.92.48.135 (100.92.48.135) 12.539 ms
>>
>> 15 100.92.49.66 (100.92.49.66) 12.591 ms 100.92.44.90 (100.92.44.90)
>> 18.647 ms 18.368 ms
>>
>> 16 100.92.49.7 (100.92.49.7) 12.986 ms 100.92.44.9 (100.92.44.9)
>> 12.990 ms 100.92.44.123 (100.92.44.123) 18.647 ms
>>
>> 17 52.93.132.62 (52.93.132.62) 12.399 ms 52.93.130.134 (52.93.130.134)
>> 12.463 ms 52.93.133.56 (52.93.133.56) 12.744 ms
>>
>> 18 100.91.163.4 (100.91.163.4) 13.681 ms 100.91.167.8 (100.91.167.8)
>> 18.976 ms 100.91.163.96 (100.91.163.96) 13.129 ms
>>
>> 19 100.91.163.89 (100.91.163.89) 12.558 ms 100.91.168.43
>> (100.91.168.43) 13.010 ms 100.91.167.17 (100.91.167.17) 13.632 ms
>>
>> 20 100.91.164.108 (100.91.164.108) 12.366 ms 100.91.159.64
>> (100.91.159.64) 17.061 ms 100.91.164.60 (100.91.164.60) 12.693 ms
>>
>> 21 100.91.159.73 (100.91.159.73) 17.008 ms 100.91.164.105
>> (100.91.164.105) 13.024 ms 100.91.165.129 (100.91.165.129) 13.239 ms
>>
>> 22 100.91.177.41 (100.91.177.41) 11.699 ms 100.91.177.55
>> (100.91.177.55) 11.945 ms 100.91.176.241 (100.91.176.241) 11.779 ms
>>
>> 23 100.100.6.21 (100.100.6.21) 12.162 ms 100.100.8.11 (100.100.8.11)
>> 14.884 ms 100.100.8.117 (100.100.8.117) 12.359 ms
>>
>> 24 100.100.93.8 (100.100.93.8) 12.228 ms 100.100.77.198
>> (100.100.77.198) 12.200 ms 100.100.76.134 (100.100.76.134) 12.463 ms
>>
>> 25 100.100.93.195 (100.100.93.195) 12.901 ms 100.100.73.221
>> (100.100.73.221) 13.835 ms 100.100.85.221 (100.100.85.221) 12.385 ms
>>
>> 26 100.100.28.124 (100.100.28.124) 12.296 ms 100.100.34.92
>> (100.100.34.92) 12.247 ms 100.100.2.36 (100.100.2.36) 12.940 ms
>>
>> 27 72.14.203.158 (72.14.203.158) 12.939 ms 99.82.181.23 (99.82.181.23)
>> 12.316 ms 72.14.203.158 (72.14.203.158) 12.906 ms
>>
>> 28 108.170.246.66 (108.170.246.66) 12.874 ms * 12.818 ms
>>
>> 29 108.170.246.33 (108.170.246.33) 13.149 ms 72.14.203.158
>> (72.14.203.158) 17.185 ms 99.82.181.23 (99.82.181.23) 15.395 ms
>>
>> 30 108.170.246.66 (108.170.246.66) 12.841 ms * 108.170.240.98
>> (108.170.240.98) 12.831 ms
>>
>> 31 * * *
>>
>> 32 * * *
>>
>> 33 * * *
>>
>> 34 * * *
>>
>> 35 * * *
>>
>> 36 * * *
>>
>> 37 * * *
>>
>> 38 * * *
>>
>> 39 * * *
>>
>> 40 * * *
>>
>> 41 * * *
>>
>> 42 * * *
>>
>> 43 * * *
>>
>> 44 * * *
>>
>> 45 * * *
>>
>> 46 * * *
>>
>> 47 * * *
>>
>> 48 * * *
>>
>> 49 * * *
>>
>> 50 * * *
>>
>> Looking closer it looks like almost all of the DEADLINE_EXCEEDED were on
>>
>> google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
>> calls
>>
>> Request
>>
>> -------
>>
>> MethodName:
>> google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
>>
>> Endpoint: googleads.googleapis.com:443
>>
>> Headers: {developer-token=REDACTED, login-customer-id=5741853629,
>> x-goog-api-client=gl-java/1.8.0_201 gapic/ gax/1.65.1 grpc/1.37.0}
>>
>> Body: customer_id: "REDACTED"
>>
>> operations {
>>
>> update {
>>
>> resource_name: "customers/REDACTED/campaignBudgets/7001096614"
>>
>> status: ENABLED
>>
>> delivery_method: STANDARD
>>
>> id: 7001096614
>>
>> amount_micros: 10000
>>
>> explicitly_shared: false
>>
>> }
>>
>> update_mask {
>>
>> paths: "resource_name"
>>
>> paths: "id"
>>
>> paths: "amount_micros"
>>
>> paths: "status"
>>
>> paths: "delivery_method"
>>
>> paths: "explicitly_shared"
>>
>> }
>>
>> }
>>
>> partial_failure: true
>>
>>
>>
>> Response
>>
>> --------
>>
>> Headers: null
>>
>> Body: null
>>
>> Failure message: null
>>
>> Status: Status{code=DEADLINE_EXCEEDED, description=deadline exceeded
>> after 3599.999166595s. [buffered_nanos=65828859, buffered_nanos=107502844,
>> remote_addr=googleads.googleapis.com/142.250.190.74:443], cause=null}.
>>
>> 2021-06-22 23:49:42,443 INFO RequestLogger - FAILURE REQUEST DETAIL.
>>
>> There's no request-id generated / given on a DEADLINE_EXCEEDED so I can't
>> provide one.
>>
>> Pete
>>
>
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to adwords-api+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/adwords-api/f95dadf3-6ea9-4ae2-875f-cd2437645ad7n%40googlegroups.com.
