Migrated to V8 : Started Seeing Increased Rate of DEADLINE_EXCEEDED
We put V8 into production earlier today. Everything was fine for about 6.5
hours and then we started seeing an increased rate of DEADLINE_EXCEEDED
Our GrpcCallContext is configured for a 5m timeout and that's been fine for
months
I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the last
20 minutes but I'm not sure if that proves anything
Is it perhaps a transient issue on Google's end for maintenance work around
increased capacity for V8?
Here's the traceroute ... we've been asked to provide this before when we
witnessed a much higher rate of DEADLINE_EXCEEDED
traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60 byte
packets
1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104 ms
0.096 ms 0.102 ms
2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985 ms
ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 ms
ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms
3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16)
6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms
4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144 (100.66.12.144)
3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms
5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202 (100.66.14.202)
22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms
6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131)
50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms
7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117)
31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms
8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161 (100.65.13.161)
0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms
9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161
(15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms
10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10)
2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms
11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820 ms
52.93.239.50 (52.93.239.50) 1.313 ms
12 100.92.53.124 (100.92.53.124) 12.442 ms 100.92.53.76 (100.92.53.76)
14.483 ms 100.92.53.72 (100.92.53.72) 12.509 ms
13 100.92.43.34 (100.92.43.34) 12.711 ms 100.92.48.100 (100.92.48.100)
16.661 ms 100.92.48.10 (100.92.48.10) 13.001 ms
14 100.92.43.85 (100.92.43.85) 12.143 ms 100.92.48.139 (100.92.48.139)
12.547 ms 100.92.48.135 (100.92.48.135) 12.539 ms
15 100.92.49.66 (100.92.49.66) 12.591 ms 100.92.44.90 (100.92.44.90)
18.647 ms 18.368 ms
16 100.92.49.7 (100.92.49.7) 12.986 ms 100.92.44.9 (100.92.44.9) 12.990
ms 100.92.44.123 (100.92.44.123) 18.647 ms
17 52.93.132.62 (52.93.132.62) 12.399 ms 52.93.130.134 (52.93.130.134)
12.463 ms 52.93.133.56 (52.93.133.56) 12.744 ms
18 100.91.163.4 (100.91.163.4) 13.681 ms 100.91.167.8 (100.91.167.8)
18.976 ms 100.91.163.96 (100.91.163.96) 13.129 ms
19 100.91.163.89 (100.91.163.89) 12.558 ms 100.91.168.43 (100.91.168.43)
13.010 ms 100.91.167.17 (100.91.167.17) 13.632 ms
20 100.91.164.108 (100.91.164.108) 12.366 ms 100.91.159.64
(100.91.159.64) 17.061 ms 100.91.164.60 (100.91.164.60) 12.693 ms
21 100.91.159.73 (100.91.159.73) 17.008 ms 100.91.164.105
(100.91.164.105) 13.024 ms 100.91.165.129 (100.91.165.129) 13.239 ms
22 100.91.177.41 (100.91.177.41) 11.699 ms 100.91.177.55 (100.91.177.55)
11.945 ms 100.91.176.241 (100.91.176.241) 11.779 ms
23 100.100.6.21 (100.100.6.21) 12.162 ms 100.100.8.11 (100.100.8.11)
14.884 ms 100.100.8.117 (100.100.8.117) 12.359 ms
24 100.100.93.8 (100.100.93.8) 12.228 ms 100.100.77.198 (100.100.77.198)
12.200 ms 100.100.76.134 (100.100.76.134) 12.463 ms
25 100.100.93.195 (100.100.93.195) 12.901 ms 100.100.73.221
(100.100.73.221) 13.835 ms 100.100.85.221 (100.100.85.221) 12.385 ms
26 100.100.28.124 (100.100.28.124) 12.296 ms 100.100.34.92
(100.100.34.92) 12.247 ms 100.100.2.36 (100.100.2.36) 12.940 ms
27 72.14.203.158 (72.14.203.158) 12.939 ms 99.82.181.23 (99.82.181.23)
12.316 ms 72.14.203.158 (72.14.203.158) 12.906 ms
28 108.170.246.66 (108.170.246.66) 12.874 ms * 12.818 ms
29 108.170.246.33 (108.170.246.33) 13.149 ms 72.14.203.158
(72.14.203.158) 17.185 ms 99.82.181.23 (99.82.181.23) 15.395 ms
30 108.170.246.66 (108.170.246.66) 12.841 ms * 108.170.240.98
(108.170.240.98) 12.831 ms
31 * * *
32 * * *
33 * * *
34 * * *
35 * * *
36 * * *
37 * * *
38 * * *
39 * * *
40 * * *
41 * * *
42 * * *
43 * * *
44 * * *
45 * * *
46 * * *
47 * * *
48 * * *
49 * * *
50 * * *
Looking closer it looks like almost all of the DEADLINE_EXCEEDED were on
google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
calls
Request
---
MethodName:
google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=5741853629,
x-g
Re: Migrated to V8 : Started Seeing Increased Rate of DEADLINE_EXCEEDED
We're now seeing this on other calls as well. These are across multiple customerIds. 2021-06-23 01:19:33,532 WARN RequestLogger - FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v7.services.AdService/MutateAds, Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after 3599.998632475s. [buffered_nanos=67392207, buffered_nanos=93613556, remote_addr=googleads.googleapis.com/142.250.191.138:443]. 2021-06-23 01:22:48,579 WARN RequestLogger - FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after 3599.985958315s. [buffered_nanos=65378596, buffered_nanos=67500777, remote_addr=googleads.googleapis.com/172.217.8.202:443]. 20212021-06-23 01:30:08,166 WARN RequestLogger - FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after 3599.998821696s. [buffered_nanos=66057372, buffered_nanos=77619715, remote_addr=googleads.googleapis.com/142.250.191.202:443]. On Tuesday, June 22, 2021 at 8:23:06 PM UTC-4 Pete Lavetsky (AdWords API Guru) wrote: > We put V8 into production earlier today. Everything was fine for about 6.5 > hours and then we started seeing an increased rate of DEADLINE_EXCEEDED > > Our GrpcCallContext is configured for a 5m timeout and that's been fine > for months > > I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the last > 20 minutes but I'm not sure if that proves anything > > Is it perhaps a transient issue on Google's end for maintenance work > around increased capacity for V8? > > Here's the traceroute ... we've been asked to provide this before when we > witnessed a much higher rate of DEADLINE_EXCEEDED > > traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60 > byte packets > > 1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104 ms > 0.096 ms 0.102 ms > > 2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985 ms > ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 ms > ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms > > 3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16) > 6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms > > 4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144 (100.66.12.144) > 3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms > > 5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202 > (100.66.14.202) 22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms > > 6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131) > 50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms > > 7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117) > 31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms > > 8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161 (100.65.13.161) > 0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms > > 9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161 > (15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms > > 10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10) > 2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms > > 11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820 > ms 52.93.239.50 (52.93.239.50) 1.313 ms > > 12 100.92.53.124 (100.92.53.124) 12.442 ms 100.92.53.76 (100.92.53.76) > 14.483 ms 100.92.53.72 (100.92.53.72) 12.509 ms > > 13 100.92.43.34 (100.92.43.34) 12.711 ms 100.92.48.100 (100.92.48.100) > 16.661 ms 100.92.48.10 (100.92.48.10) 13.001 ms > > 14 100.92.43.85 (100.92.43.85) 12.143 ms 100.92.48.139 (100.92.48.139) > 12.547 ms 100.92.48.135 (100.92.48.135) 12.539 ms > > 15 100.92.49.66 (100.92.49.66) 12.591 ms 100.92.44.90 (100.92.44.90) > 18.647 ms 18.368 ms > > 16 100.92.49.7 (100.92.49.7) 12.986 ms 100.92.44.9 (100.92.44.9) 12.990 > ms 100.92.44.123 (100.92.44.123) 18.647 ms > > 17 52.93.132.62 (52.93.132.62) 12.399 ms 52.93.130.134 (52.93.130.134) > 12.463 ms 52.93.133.56 (52.93.133.56) 12.744 ms > > 18 100.91.163.4 (100.91.163.4) 13.681 ms 100.91.167.8 (100.91.167.8) > 18.976 ms 100.91.163.96 (100.91.163.96) 13.129 ms > > 19 100.91.163.89 (100.91.163.89) 12.558 ms 100.91.168.43 > (100.91.168.43) 13.010 ms 100.91.167.17 (100.91.167.17) 13.632 ms > > 20 100.91.164.108 (100.91.164.108) 12.366 ms 100.91.159.64 > (100.91.159.64) 17.061 ms 100.91.164.60 (100.91.164.60) 12.693 ms > > 21 100.91.159.73 (100.91.159.73) 17.008 ms 100.91.164.105 > (100.91.164.105) 13.024 ms 100.91.165.129 (100.91.165.129) 13.239 ms > > 22 100.91.177.41 (100.91.177.41) 11.699 ms 100.91.177.55 > (100.91.177.55) 11.945
Re: Migrated to V8 : Started Seeing Increased Rate of DEADLINE_EXCEEDED
Even on mutates that don't make it to our DEADLINE threshold we're seeing a massive slowdown in responsiveness in mutate calls. Below, hour 20 corresponds to 4pm EST on the 22nd of June. day hour avg(unix_timestamp(date_finished)-unix_timestamp(date_started)) 22 12 10.4448 22 13 10.3481 22 14 9.4694 22 15 13.9076 22 16 13.1181 22 17 20.0910 22 18 16.3803 22 19 19.1051 22 20 38.2070 22 21 480.0347 22 22 61.2185 22 23 227.2379 On Tuesday, June 22, 2021 at 9:49:41 PM UTC-4 Pete Lavetsky (AdWords API Guru) wrote: > We're now seeing this on other calls as well. These are across multiple > customerIds. > > 2021-06-23 01:19:33,532 WARN RequestLogger - FAILURE REQUEST SUMMARY. > Method: google.ads.googleads.v7.services.AdService/MutateAds, Endpoint: > googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, > ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after > 3599.998632475s. [buffered_nanos=67392207, buffered_nanos=93613556, > remote_addr=googleads.googleapis.com/142.250.191.138:443]. > > 2021-06-23 01:22:48,579 WARN RequestLogger - FAILURE REQUEST SUMMARY. > Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, > Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: > null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after > 3599.985958315s. [buffered_nanos=65378596, buffered_nanos=67500777, > remote_addr=googleads.googleapis.com/172.217.8.202:443]. > > 20212021-06-23 01:30:08,166 WARN RequestLogger - FAILURE REQUEST SUMMARY. > Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, > Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: > null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after > 3599.998821696s. [buffered_nanos=66057372, buffered_nanos=77619715, > remote_addr=googleads.googleapis.com/142.250.191.202:443]. > > On Tuesday, June 22, 2021 at 8:23:06 PM UTC-4 Pete Lavetsky (AdWords API > Guru) wrote: > >> We put V8 into production earlier today. Everything was fine for about >> 6.5 hours and then we started seeing an increased rate of DEADLINE_EXCEEDED >> >> Our GrpcCallContext is configured for a 5m timeout and that's been fine >> for months >> >> I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the >> last 20 minutes but I'm not sure if that proves anything >> >> Is it perhaps a transient issue on Google's end for maintenance work >> around increased capacity for V8? >> >> Here's the traceroute ... we've been asked to provide this before when we >> witnessed a much higher rate of DEADLINE_EXCEEDED >> >> traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60 >> byte packets >> >> 1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104 >> ms 0.096 ms 0.102 ms >> >> 2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985 >> ms ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 ms >> ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms >> >> 3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16) >> 6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms >> >> 4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144 >> (100.66.12.144) 3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms >> >> 5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202 >> (100.66.14.202) 22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms >> >> 6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131) >> 50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms >> >> 7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117) >> 31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms >> >> 8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161 >> (100.65.13.161) 0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms >> >> 9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161 >> (15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms >> >> 10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10) >> 2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms >> >> 11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820 >> ms 52.93.239.50 (52.93.239.50) 1.313 ms >> >> 12 100.92.53.124 (100.92.53.124) 12.442 ms 100.92.53.76 (100.92.53.76) >> 14.483 ms 100.92.53.72 (100.92.53.72) 12.509 ms >> >> 13 100.92.43.34 (100.92.43.34) 12.711 ms 100.92.48.100 (100.92.48.100) >> 16.661 ms 100.92.48.10 (100.92.48.10) 13.001 ms >> >> 14 100.92.43.85 (100.92.43.85) 12.143 ms 100.92.48.139 (100.92.48.139) >> 12.547 ms 100.92.48.135 (100.92.48.135) 12.539 ms >> >> 15 100.92.49.66 (100.92.49.66) 12.591 ms 100.92.44.90 (100.92.44.90) >> 18.647 ms 18.368 ms >> >> 16 100.92.49.7 (100.92.49.7) 12.986 ms 100.92.44.9 (100.92.44.9) >> 12.990 ms 100.92.44.123 (100.92.44.123) 18.647 ms >> >> 17 52.93.132.62 (52.93.132.62) 12.399 ms 52.93.130.134 (52.93.130.134) >> 12.463 ms 52.93.133.56 (52.93.133.56
Re: Migrated to V8 : Started Seeing Increased Rate of DEADLINE_EXCEEDED
Hi there, we seem to be in the same boat: [image: Bildschirmfoto vom 2021-06-23 07-16-40.png] Regards Mat Pete Lavetsky (AdWords API Guru) schrieb am Mittwoch, 23. Juni 2021 um 04:22:52 UTC+2: > Even on mutates that don't make it to our DEADLINE threshold we're seeing > a massive slowdown in responsiveness in mutate calls. Below, hour 20 > corresponds to 4pm EST on the 22nd of June. > > day hour avg(unix_timestamp(date_finished)-unix_timestamp(date_started)) > 22 12 10.4448 > 22 13 10.3481 > 22 14 9.4694 > 22 15 13.9076 > 22 16 13.1181 > 22 17 20.0910 > 22 18 16.3803 > 22 19 19.1051 > 22 20 38.2070 > 22 21 480.0347 > 22 22 61.2185 > 22 23 227.2379 > > On Tuesday, June 22, 2021 at 9:49:41 PM UTC-4 Pete Lavetsky (AdWords API > Guru) wrote: > >> We're now seeing this on other calls as well. These are across multiple >> customerIds. >> >> 2021-06-23 01:19:33,532 WARN RequestLogger - FAILURE REQUEST SUMMARY. >> Method: google.ads.googleads.v7.services.AdService/MutateAds, Endpoint: >> googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, >> ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after >> 3599.998632475s. [buffered_nanos=67392207, buffered_nanos=93613556, >> remote_addr=googleads.googleapis.com/142.250.191.138:443]. >> >> 2021-06-23 01:22:48,579 WARN RequestLogger - FAILURE REQUEST SUMMARY. >> Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, >> Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: >> null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after >> 3599.985958315s. [buffered_nanos=65378596, buffered_nanos=67500777, >> remote_addr=googleads.googleapis.com/172.217.8.202:443]. >> >> 20212021-06-23 01:30:08,166 WARN RequestLogger - FAILURE REQUEST >> SUMMARY. Method: >> google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, >> Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: >> null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after >> 3599.998821696s. [buffered_nanos=66057372, buffered_nanos=77619715, >> remote_addr=googleads.googleapis.com/142.250.191.202:443]. >> >> On Tuesday, June 22, 2021 at 8:23:06 PM UTC-4 Pete Lavetsky (AdWords API >> Guru) wrote: >> >>> We put V8 into production earlier today. Everything was fine for about >>> 6.5 hours and then we started seeing an increased rate of DEADLINE_EXCEEDED >>> >>> Our GrpcCallContext is configured for a 5m timeout and that's been fine >>> for months >>> >>> I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the >>> last 20 minutes but I'm not sure if that proves anything >>> >>> Is it perhaps a transient issue on Google's end for maintenance work >>> around increased capacity for V8? >>> >>> Here's the traceroute ... we've been asked to provide this before when >>> we witnessed a much higher rate of DEADLINE_EXCEEDED >>> >>> traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60 >>> byte packets >>> >>> 1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104 >>> ms 0.096 ms 0.102 ms >>> >>> 2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985 >>> ms ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 >>> ms ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms >>> >>> 3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16) >>> 6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms >>> >>> 4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144 >>> (100.66.12.144) 3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms >>> >>> 5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202 >>> (100.66.14.202) 22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms >>> >>> 6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131) >>> 50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms >>> >>> 7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117) >>> 31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms >>> >>> 8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161 >>> (100.65.13.161) 0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms >>> >>> 9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161 >>> (15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms >>> >>> 10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10) >>> 2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms >>> >>> 11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820 >>> ms 52.93.239.50 (52.93.239.50) 1.313 ms >>> >>> 12 100.92.53.124 (100.92.53.124) 12.442 ms 100.92.53.76 >>> (100.92.53.76) 14.483 ms 100.92.53.72 (100.92.53.72) 12.509 ms >>> >>> 13 100.92.43.34 (100.92.43.34) 12.711 ms 100.92.48.100 >>> (100.92.48.100) 16.661 ms 100.92.48.10 (100.92.48.10) 13.001 ms >>> >>> 14 100.92.43.85 (100.92.43.85) 12.143 ms 100.92.48.139 >>> (100.92.48.139) 12.547 ms 100.92.48.135 (100.92.48.135) 12.539 ms >>> >>> 15 100.92.49.
Re: Migrated to V8 : Started Seeing Increased Rate of DEADLINE_EXCEEDED
For us, this cleared up overnight: day hour avg_job_time 23 0 31.2911 23 1 101.0072 23 2 246.6130 23 3 1827.2152 23 4 296.0434 23 5 7.2689 23 6 19.2890 23 7 6.0186 23 8 11.5061 23 9 13.0516 On Wednesday, June 23, 2021 at 1:20:09 AM UTC-4 Mat wrote: > Hi there, > > we seem to be in the same boat: > > [image: Bildschirmfoto vom 2021-06-23 07-16-40.png] > > > Regards > Mat > > Pete Lavetsky (AdWords API Guru) schrieb am Mittwoch, 23. Juni 2021 um > 04:22:52 UTC+2: > >> Even on mutates that don't make it to our DEADLINE threshold we're seeing >> a massive slowdown in responsiveness in mutate calls. Below, hour 20 >> corresponds to 4pm EST on the 22nd of June. >> >> day hour avg(unix_timestamp(date_finished)-unix_timestamp(date_started)) >> 22 12 10.4448 >> 22 13 10.3481 >> 22 14 9.4694 >> 22 15 13.9076 >> 22 16 13.1181 >> 22 17 20.0910 >> 22 18 16.3803 >> 22 19 19.1051 >> 22 20 38.2070 >> 22 21 480.0347 >> 22 22 61.2185 >> 22 23 227.2379 >> >> On Tuesday, June 22, 2021 at 9:49:41 PM UTC-4 Pete Lavetsky (AdWords API >> Guru) wrote: >> >>> We're now seeing this on other calls as well. These are across multiple >>> customerIds. >>> >>> 2021-06-23 01:19:33,532 WARN RequestLogger - FAILURE REQUEST SUMMARY. >>> Method: google.ads.googleads.v7.services.AdService/MutateAds, Endpoint: >>> googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, >>> ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after >>> 3599.998632475s. [buffered_nanos=67392207, buffered_nanos=93613556, >>> remote_addr=googleads.googleapis.com/142.250.191.138:443]. >>> >>> 2021-06-23 01:22:48,579 WARN RequestLogger - FAILURE REQUEST SUMMARY. >>> Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, >>> Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, >>> RequestID: null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline >>> exceeded after 3599.985958315s. [buffered_nanos=65378596, >>> buffered_nanos=67500777, remote_addr= >>> googleads.googleapis.com/172.217.8.202:443]. >>> >>> 20212021-06-23 01:30:08,166 WARN RequestLogger - FAILURE REQUEST >>> SUMMARY. Method: >>> google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, >>> Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, >>> RequestID: null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline >>> exceeded after 3599.998821696s. [buffered_nanos=66057372, >>> buffered_nanos=77619715, remote_addr= >>> googleads.googleapis.com/142.250.191.202:443]. >>> >>> On Tuesday, June 22, 2021 at 8:23:06 PM UTC-4 Pete Lavetsky (AdWords API >>> Guru) wrote: >>> We put V8 into production earlier today. Everything was fine for about 6.5 hours and then we started seeing an increased rate of DEADLINE_EXCEEDED Our GrpcCallContext is configured for a 5m timeout and that's been fine for months I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the last 20 minutes but I'm not sure if that proves anything Is it perhaps a transient issue on Google's end for maintenance work around increased capacity for V8? Here's the traceroute ... we've been asked to provide this before when we witnessed a much higher rate of DEADLINE_EXCEEDED traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60 byte packets 1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104 ms 0.096 ms 0.102 ms 2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985 ms ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 ms ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms 3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16) 6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms 4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144 (100.66.12.144) 3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms 5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202 (100.66.14.202) 22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms 6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131) 50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms 7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117) 31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms 8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161 (100.65.13.161) 0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms 9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161 (15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms 10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10) 2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms 11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820 ms 52.93.239.50 (52.93.239.50) 1.313 ms 12 100.92.53.124 (100.92.53
RE: Migrated to V8 : Started Seeing Increased Rate of DEADLINE_EXCEEDED
Hi Pete and Mat, Thank you for the logs and dashboard pictures. Based on Pete's last response it seems that the issue is transient and it cleared up. If DEADLINE_EXCEEDED occurs again please get back to us. Regards, Aryeh Baker Google Ads API Team ref:_00D1U1174p._5004Q2IxwR1:ref -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/xNoir0QV822300rRYpemb_Tee0ZH1iH1pHRg%40sfdc.net.
