Re: Batch job service is not efficient, stucked

[Stefano Fontana]

See this issue https://github.com/googleads/googleads-python-lib/issues/118

On Tuesday, September 27, 2016 at 8:10:05 PM UTC+2, Vishal Vinayak (Adwords 
API Team) wrote:
>
> Hi Mike,
>
> The advantage of batch jobs is in the fact that you can group many 
> operations of various kinds into a single request and the API would 
> automatically handle retrying and rate limiting logic for you. It may, 
> however, take some time to complete the batch job if there are hundreds of 
> thousands of operations in a single request. If you follow the best 
> practices guide (which I shared in my previous response), you should be 
> able to achieve optimum performance. Since AdWords API uses SOAP protocol 
> in the back-end, unfortunately I cannot think of a workaround to the way 
> the jobs are sent via a SOAP XML. If you have more questions related to the 
> Python client library in general, I would recommend opening an issue here 
> . 
>
> Regards,
> Vishal, AdWords API Team
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/4b156431-df30-411c-8b9e-ce189b5bcc8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Batch job service is not efficient, stucked

['Vishal Vinayak (Adwords API Team)' via AdWords API Forum]

Hi Mike,

The advantage of batch jobs is in the fact that you can group many 
operations of various kinds into a single request and the API would 
automatically handle retrying and rate limiting logic for you. It may, 
however, take some time to complete the batch job if there are hundreds of 
thousands of operations in a single request. If you follow the best 
practices guide (which I shared in my previous response), you should be 
able to achieve optimum performance. Since AdWords API uses SOAP protocol 
in the back-end, unfortunately I cannot think of a workaround to the way 
the jobs are sent via a SOAP XML. If you have more questions related to the 
Python client library in general, I would recommend opening an issue here 
. 

Regards,
Vishal, AdWords API Team

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/45db2e67-2572-49e9-9e47-0fd99e606eff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Batch job service is not efficient, stucked

[mike]

Thanks for the quick response!
The problem we see is not the job in google. Its when building the SOAP xml 
to upload. The function that do this is job_helper.UploadOperations() from 
the adwprds library in python.
We checked and it takes a very long time before it sends to google 
(creating ~10 keywords).
Do you a better way to do this? Maybe a workaround or another protocol.

Thanks!



On Thursday, September 22, 2016 at 11:31:07 PM UTC+3, Vishal Vinayak 
(Adwords API Team) wrote:
>
> Hi Mike,
>
> Looks like you are referring to our python example 
> 
>  to 
> add keywords using a batch job, which is definitely recommended. Please 
> find the rest of my answers inline:
>
> Im creating adgroups with in total ~10 keywords and it takes a lot of 
> time(sometimes also stucked).
> *Could you please provide SOAP XML request and response logs for the job 
> that you feel got stuck? If not, if you have the batch job ID, that would 
> help me in debugging the issue.*
>
> 1. There a efficient way to create this batch jobs?
> *The recommended way to create a batch job is explained in our Batch 
> Processing Guide 
> .
>  
> Although the process is explained in Java, the same guidelines are referred 
> to when creating the Python library examples. *
>
> 2. There is a limit of operations?
> *You may see a RateExceededError 
> 
>  if 
> you perform more than 10,000 operations in a day. Please refer to our Rate 
> Sheet  to 
> understand what counts as an operation. In this case, the only way to 
> increase the operational limit for your account would be to apply for 
> a Standard Access 
> . You can, 
> however, also see this error if you send too many requests to the API in a 
> short amount of time (few minutes for example). This usually happens if you 
> do not group your operations into a batch job 
>  and 
> instead prefer to send one operation with each API request. This rate limit 
> is in place to stop malicious software from crashing our servers. Rate 
> limits 
>  (not to 
> be confused with System Limits 
> ) are 
> transient and expire after a period of time.The retryAfterSeconds 
> 
>  property 
> of the RateExceededError would tell you the appropriate time interval after 
> which it is safe to retry. Also, there are no penalties associated with 
> hitting the rate limit.*
>
> 3. How can we improve it(best practices)?
> *You can find a list of best practices for batch jobs here 
> .*
>
> Regards,
> Vishal, AdWords API Team
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/707dab04-1ba1-498d-a9ab-d066c896ad47%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Batch job service is not efficient, stucked

['Vishal Vinayak (Adwords API Team)' via AdWords API Forum]

Hi Mike,

Looks like you are referring to our python example 

 to 
add keywords using a batch job, which is definitely recommended. Please 
find the rest of my answers inline:

Im creating adgroups with in total ~10 keywords and it takes a lot of 
time(sometimes also stucked).
*Could you please provide SOAP XML request and response logs for the job 
that you feel got stuck? If not, if you have the batch job ID, that would 
help me in debugging the issue.*

1. There a efficient way to create this batch jobs?
*The recommended way to create a batch job is explained in our Batch 
Processing Guide 
.
 
Although the process is explained in Java, the same guidelines are referred 
to when creating the Python library examples. *

2. There is a limit of operations?
*You may see a RateExceededError 

 if 
you perform more than 10,000 operations in a day. Please refer to our Rate 
Sheet  to 
understand what counts as an operation. In this case, the only way to 
increase the operational limit for your account would be to apply for 
a Standard Access 
. You can, 
however, also see this error if you send too many requests to the API in a 
short amount of time (few minutes for example). This usually happens if you 
do not group your operations into a batch job 
 and 
instead prefer to send one operation with each API request. This rate limit 
is in place to stop malicious software from crashing our servers. Rate 
limits 
 (not to 
be confused with System Limits 
) are 
transient and expire after a period of time.The retryAfterSeconds 

 property 
of the RateExceededError would tell you the appropriate time interval after 
which it is safe to retry. Also, there are no penalties associated with 
hitting the rate limit.*

3. How can we improve it(best practices)?
*You can find a list of best practices for batch jobs here 
.*

Regards,
Vishal, AdWords API Team

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/c2e541d1-b9f7-4dc8-8491-2bee3e35f38d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Batch job service is not efficient, stucked

[mike]

Hello,
Im trying to use the BatchJob service in python.
Im creating adgroups with in total ~10 keywords and it takes a lot of 
time(sometimes also stucked).
I have read the code in the api documentation.

My question is:
1. There a efficient way to create this batch jobs?\
2. There is a limit of operations?
3. How can we improve it(best practices)?

This is my code(took from internet):

def run_operations_as_batch_job(account_id, wait_for_response=False, 
*operations):
set_adwords_client(account_id)
job_helper = adwords_client.GetBatchJobHelper()
job = create_batch_job()
upload_url = job['uploadUrl']['url']
batch_job_id = job['id']

job_helper.UploadOperations(upload_url, *operations)
while wait_for_response:
try:
sleep(5)
url = GetBatchJobDownloadUrlWhenReady(batch_job_id)
response = urllib2.urlopen(url).read()
return job_helper.ParseResponse(response)
except:
pass
return batch_job_id


def GetBatchJobDownloadUrlWhenReady(batch_job_id, max_poll_attempts=10):
batch_job = get_batch_job(batch_job_id)
poll_attempt = 0
while poll_attempt < max_poll_attempts and batch_job['status'] in ('DONE', 
'ACTIVE', 'AWAITING_FILE', 'CANCELING'):
if 'downloadUrl' in batch_job:
url = batch_job['downloadUrl']['url']
print ('Batch Job with Id "%s", Status "%s", and DownloadUrl "%s" 
ready.'
   % (batch_job['id'], batch_job['status'], url))
return url
sleep_interval = (30 * (2 ** poll_attempt) +
  (random.randint(0, 1) / 1000))
print 'Batch Job not ready, sleeping for %s seconds.' % sleep_interval
sleep(sleep_interval)
batch_job = get_batch_job(batch_job_id)
poll_attempt += 1

raise Exception('Batch Job not finished downloading. Try checking later.')


def get_batch_job(batch_job_id):
batch_job_service = get_service(Services.BATCH_JOB_SERVICE)
selector = {
  'fields': ['Id', 'Status', 'DownloadUrl'],
  'predicates': [{'field': 'Id', 'operator': 'EQUALS', 'values': 
[batch_job_id]}]
}
return batch_job_service.get(selector)['entries'][0]

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/14f4924f-62c8-4c1a-96af-c61e1fca0582%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.