Hi Stefan, Thanks for reporting this. I personally experienced some longer than usual running batch jobs last night as well. We're looking into what caused this last spike in execution time.
Cheers, Nadine, AdWords API Team On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine Stegemann wrote: > > Hi, > > sorry for hijacking this thread but since yesterday morning, we also > started to have significant troubles with batch job service. Jobs remained > active for a very long time. The following jobs have been started yesterday > morning, german time (see start time is noted for each job) and were still > active this morning: > > Job 445516765 | 2017-02-14T03:34 > Job 445553000 | 2017-02-14T03:20 > Job 445532407 | 2017-02-14T03:58 > Job 445532515 | 2017-02-14T03:58 > Job 445693959 | 2017-02-14T03:58 > Job 445694061 | 2017-02-14T03:59 > Job 445694595 | 2017-02-14T03:59 > > We've canceled the jobs this morning (after they are still reported to be > active, how can that be?), but now they are in canceling state for about an > hour (at the time of this writing). Furthermore, the situation got worse > this morning, with 30 batch jobs being active for a couple of hours now. > The ids of these jobs are attached to the end of this post. We will cancel > these jobs as well because we do not expect them to finish, based on > yesterday's experience. > > As an additional information, the problems seem to affect accounts with > campaigns with a large to huge number or product partitions / keywords. The > problematic jobs were "spreaded" over three adwords accounts yesterday and > three accounts today. Two of the accounts affected yesterday are also > affected today. We do use these batch jobs to set cpc bids for keywords and > product partitions and use a maximum of 40.000 operations per job. We > experimented with this number in the past and found that to be a "suitable" > balance between performance and reliability of batch jobs. All jobs contain > only one type of operation. While multiple jobs might affect the same > campaign / adgroups, they do not affect the same keywords / product > partitions. I was told on the past API workshop that this is ok and will > not result in concurrency issues. In addition to concrete information on > the issue, we would be happy to know if it makes sense to "play" with the > limit of 40.000 operations per job again. > > As a general note, batch job continously tend to keep us busy since we > switched to them last year. The runtime performance seems to be rather > unpredictable. Especially on test accounts, batch jobs can run for a very > long time (up to 30 minutes) which is painful if you want to run automated > tests. Combined with the unclear advise on the number of operations to put > in a batch job and best practices in general, that leaves us with a very > uncertain feeling. Given the reliabilty issues we experienced in the past > half year, we're currently thinking about ditching batch jobs entirely and > implement batching on our site which would give us at least more control. > We understand that the intend of batch jobs is like "fire and forget" > (yeah, not really forget but check results etc.), we offload operations to > AdWords and it takes care of it. Currently its more like "fire and pray". > While we do not want to bring back the past, but we never had such troubles > with mutate job service. I apologize for this bit of ranting, but it's > constantly keeping us busy for a while now. > > Finally, here are the job ids of the still-active (soon to be canceled) > jobs from this morning. > > 445954525 > 445954540 > 445989185 > 446102085 > 446102088 > 445952002 > 445986527 > 445986596 > 445986662 > 445986857 > 445987025 > 445987106 > 445987562 > 445987577 > 445987715 > 446107815 > 446107821 > 446108097 > 446108442 > 446108472 > 446108553 > 446108598 > 446108796 > 446108910 > 446108958 > 45948708 > 445983389 > 445983824 > 446105442 > 446105460 > > Thanks for taking care of this and regards, > Stefan > > > Am Montag, 13. Februar 2017 22:08:05 UTC+1 schrieb AdWordsApiUser: >> >> Thanks Nadine - appreciate your quick reply. We'll keep an eye out and >> let you know if things get out of hand. >> >> On Monday, February 13, 2017 at 6:19:50 PM UTC+1, Nadine Sundquist >> (AdWords API Team) wrote: >>> >>> Hi MG, >>> >>> Thanks for sending over the batch job IDs. I took a look at the time it >>> took for the jobs to run for the slow running jobs. I didn't see any that >>> exceeded 15 minutes and most were completing within 5 minutes. There is a >>> bit of load balancing going on with a lot of job requests, so it's expected >>> that while some jobs may only take 2 minutes, there will be a few that may >>> exceed 10 minutes. Luckily, the jobs don't appear to be getting stuck >>> indefinitely, and they're not hanging for hours on end. I'll keep an eye on >>> things as I did see a spike in processing time this morning, but it looks >>> like things are returning to normal again. Please reach out if you see the >>> times increasing even more. >>> >>> Thanks again, >>> Nadine, AdWords API Team >>> >>> On Monday, February 13, 2017 at 9:39:03 AM UTC-5, AdWordsApiUser wrote: >>>> >>>> Hello team, >>>> >>>> The issue seems to happen again, though not as bad as last time, so I >>>> thought I'd let you know before it becomes worse. Some jobs are taking a >>>> long time to process, while others process very fast. Last time it >>>> happened, over 2-3 days, the jobs became slower, and a higher proportion >>>> of >>>> jobs started getting stuck till you guys did some magic to make everything >>>> fast again: >>>> >>>> stuck/took very long: >>>> 445491084 >>>> 445522515 >>>> 445523004 >>>> >>>> not stuck/ran fast: >>>> 445491588 >>>> 445352074 >>>> >>>> Cheers, >>>> MG >>>> >>>> On Thursday, February 2, 2017 at 6:51:38 PM UTC+1, AdWordsApiUser wrote: >>>>> >>>>> Thanks Nadine! You and your team saved us a costly rewrite that we had >>>>> started planning for (without using the bulk job service). We'll keep an >>>>> eye out and report back if we see issues. Thanks team - you're the most >>>>> responsive team at Google I've interacted with ;) >>>>> >>>>> On Thursday, February 2, 2017 at 6:46:03 PM UTC+1, Nadine Sundquist >>>>> (AdWords API Team) wrote: >>>>>> >>>>>> Hello Everyone, >>>>>> >>>>>> I have good news! We found the root cause, and the issue has been >>>>>> fixed. Your batch job processing times should be back to normal now. >>>>>> Thanks >>>>>> again for reporting this! >>>>>> >>>>>> Take care, >>>>>> Nadine, AdWords API Team >>>>>> >>>>>> On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine Sundquist >>>>>> (AdWords API Team) wrote: >>>>>>> >>>>>>> Greetings Fellow AdWordsians, >>>>>>> >>>>>>> Thanks for the batch job IDs. They are a BIG help. I've taken a look >>>>>>> at the history of those batch jobs in our internal systems, and I've >>>>>>> found >>>>>>> a pattern. I need to run it by a few teammates here to see if my >>>>>>> observations result in finding a root cause. I'll update you when I've >>>>>>> made >>>>>>> a bit more progress. >>>>>>> >>>>>>> Cheers, >>>>>>> Nadine, AdWords API Team >>>>>>> >>>>>>> On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, AdWordsApiUser >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Vishal, >>>>>>>> >>>>>>>> Thanks for your reply. However, we have had 100% of the jobs we've >>>>>>>> submitted in the last 6 hours take over 15 mintues to complete. >>>>>>>> >>>>>>>> Regards, >>>>>>>> MG >>>>>>>> >>>>>>>> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak >>>>>>>> (Adwords API Team) wrote: >>>>>>>>> >>>>>>>>> Hi MG, >>>>>>>>> >>>>>>>>> Issues related to batch job performance tend to be intermittent >>>>>>>>> some of the times. In such cases, cancelling the job and sending a >>>>>>>>> new >>>>>>>>> request solves the issue. Could you please try cancelling >>>>>>>>> <https://developers.google.com/adwords/api/docs/guides/batch-jobs#canceling_a_batch_job> >>>>>>>>> all >>>>>>>>> current jobs with the status >>>>>>>>> <https://developers.google.com/adwords/api/docs/reference/v201609/BatchJobService.BatchJob#status> >>>>>>>>> AWAITING_FILE >>>>>>>>> or ACTIVE and retry? If that doesn't help, please feel free to revert >>>>>>>>> and >>>>>>>>> I'll check with my team to see if we can dig into this issue further. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Vishal, AdWords API Team >>>>>>>>> >>>>>>>> -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog and Google+: https://googleadsdeveloper.blogspot.com/ https://plus.google.com/+GoogleAdsDevelopers/posts =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. Visit this group at https://groups.google.com/group/adwords-api. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/73ea077c-d352-4079-abaf-73a7f29ad519%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
