Re: CONCURRENT_MODIFICATION while populate new users to audience group
Greetings! I did an analysis of your requests on our servers, and I found a pattern that appears to be causing these errors. I'll cover my observations followed by some best practices, so that these errors are not as frequent. Observations I noticed that there was a pattern in how the OfflineUserDataJobService was being called. The pattern was similar to this: Create the offline data job for the user list. Immediately add operations with one request. Run the job. Immediately create another data job for the same user list. Immediately add operations with one request. Run the job. And so on... When running offline user data jobs in this way, the concurrent modification exception is one I would expect. Heres's why. Every time a request is made in reference to an object like a user list, the user list resource is locked down so two threads don't accidentally cause a race condition. In the pattern I'm seeing, a new job is created for a user list before the previous job has completed. I saw repeatedly that the new job was created in the same second or the second after run job method was called on the same user list. This is where the concurrent modification exception comes in. Due to the user list still being locked down by the previous job adding data to it, the following requests can sometimes can locked out. I can give you some best practices next to explain how to structure your jobs to make that less of a chance. Best practices There are two best practices here that will help stop this error from happening. Best Practice #1 Do not have multiple jobs running in parallel for the same user list. What would need to change in your code would be to wait until the job has finished running for a user list by polling the job periodically until it's done. Then, create a new job for that user list. What I've seen our community do is have multiple offline data jobs running, but each offline data job is updating a different user list. So, they have one thread per user list running with multiple threads in parallel. That way they can still get great throughput on multiple user lists being updated without causing the error. Best Practice #2 Create larger jobs. What most people do is that for a single user list, they will create an offline job, send an add operations request, then another add operations request with more operations, then another operations create and so on, and then call run job. This creates a larger job. What this does is it means that you call create job and run job less often. A single job may take longer, but because you don't have to call create job and run job so often, you actually save time over a longer period. This is especially the case when you need to upload large amounts of data to a single user list. Please give those best practices a try. That should clear up those errors that you're seeing. Regards, Nadine Wang Google Ads API Team ref:_00D1U1174p._5004Q2TkoQ6:ref -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/5-5190R4YBQZ00K4Zw2EqMR22ro-elxs3Klg%40sfdc.net.
Re: CONCURRENT_MODIFICATION while populate new users to audience group
Hi Konstantin, Please allow me to share this with our team and we will provide an update shortly. Aside: I see you are using v7 and the Java client library. If you are using Log4J to log, please update to the latest version due to a vulnerability found in the logging utility. Regards, Matt Google Ads API Team Reminder: Share your feedback about the Google Ads (AdWords) API! Take the 2021 Google Ads API and AdWords API Annual Survey Matt Google Ads API Team ref:_00D1U1174p._5004Q2TkoQ6:ref -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/FZ-eI0R4XXII00nD10Gu96QOmZdl5i3FTL7Q%40sfdc.net.
Re: CONCURRENT_MODIFICATION while populate new users to audience group
Hi For example
error in 2021-12-29 05:37:49,977
first retry in 2021-12-29 05:39:31,536 retry with error success in
2021-12-29 05:44:32,084
2021-12-29 05:37:36,983 INFO
BazingaTaskQueueName_name_segment_filter_export__Worker_2#rid=segmentFilterExport/20211228T214652.579Z/r.y.alet.crm.core.googleads.GoogleAdsProvider:
Created an offline user data job with resource name:
customers/5687337496/offlineUserDataJobs/15828107152
2021-12-29 05:37:49,977 WARN
BazingaTaskQueueName_name_segment_filter_export__Worker_2#rid=segmentFilterExport/20211228T214652.579Z/r.y.alet.crm.core.googleads.GoogleAdsProvider:
Failed to perform action#1:
com.google.ads.googleads.v7.errors.GoogleAdsException: errors {
error_code {
database_error: CONCURRENT_MODIFICATION
}
message: "Multiple requests were attempting to modify the same resource at
once. Please retry the request."
}
request_id: "xU1ZDYrvYnsmJ-oTtjXOWQ"
com.google.ads.googleads.v7.errors.GoogleAdsException: errors {
error_code {
database_error: CONCURRENT_MODIFICATION
}
message: "Multiple requests were attempting to modify the same resource at
once. Please retry the request."
}
request_id: "xU1ZDYrvYnsmJ-oTtjXOWQ"
at
com.google.ads.googleads.v7.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42)
~[google-ads-12.0.0.jar:?]
at
com.google.ads.googleads.v7.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34)
~[google-ads-12.0.0.jar:?]
at
com.google.ads.googleads.lib.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105)
~[google-ads-12.0.0.jar:?]
at
com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:36)
~[google-ads-12.0.0.jar:?]
at
com.google.ads.googleads.lib.callables.ExceptionTransformingUnaryCallable$ExceptionTransformingFuture.onFailure(ExceptionTransformingUnaryCallable.java:82)
~[google-ads-12.0.0.jar:?]
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
~[api-common-1.10.1.jar:?]
at
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716)
~[guava-guava-26.0-jre.jar:?]
at
com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200)
~[gax-1.63.0.jar:1.63.0]
at
com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)
~[gax-1.63.0.jar:1.63.0]
at
com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117)
~[gax-1.63.0.jar:1.63.0]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716)
~[guava-guava-26.0-jre.jar:?]
at
com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
~[api-common-1.10.1.jar:?]
at
com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
~[api-common-1.10.1.jar:?]
at
com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
~[gax-grpc-1.63.0.jar:1.63.0]
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
~[api-common-1.10.1.jar:?]
at
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871)
~[guava-guava-26.0-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716)
~[guava-guava-26.0-jre.jar:?]
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:522)
~[grpc-grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:497)
~[grpc-grpc-stub-1.26.0.jar:1.26.0]
at
Re: CONCURRENT_MODIFICATION while populate new users to audience group
Hello,
Thank you for providing additional information, however, we need the complete
request and response logs with request-id to further investigate. If you're
using one of our client libraries, detailed logging can be enabled by
navigating to the Client libraries > Your client library (ex. Java) > Logging
documentation, which you can access from this link.
Please send the requested information via Reply privately to author option. If
this option is not available, you may send the details directly to our
googleadsapi-supp...@google.com alias instead. Also, I've deleted your previous
post as it contains private information. Below is the copy of your post with
scrubbed information.
For example... i get an error, wait about 2 minutes and get the error again
2021-12-29 05:37:49,977 WARN GoogleAdsProvider: Failed to perform action#1:
com.google.ads.googleads.v7.errors.GoogleAdsException: errors {
error_code {
database_error: CONCURRENT_MODIFICATION
}
message: "Multiple requests were attempting to modify the same resource at
once. Please retry the request."
}
request_id: "xx"
2021-12-29 05:39:31,536 WARN GoogleAdsProvider: Failed to perform action#2:
com.google.ads.googleads.v7.errors.GoogleAdsException: errors {
error_code {
database_error: CONCURRENT_MODIFICATION
}
message: "Multiple requests were attempting to modify the same resource at
once. Please retry the request."
}
request_id: "xx"
Regards,
Kevin Gil Soriano
Google Ads API Team
ref:_00D1U1174p._5004Q2TkoQ6:ref
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to adwords-api+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/adwords-api/tnYRy0R4V4XB00enzIEKtmRwyyx5j3DZfScA%40sfdc.net.
Re: CONCURRENT_MODIFICATION while populate new users to audience group
For example... i get an error, wait about 2 minutes and get the error again
*2021-12-29 05:37:49,977 WARN GoogleAdsProvider: Failed to perform
action#1: com.google.ads.googleads.v7.errors.GoogleAdsException: errors {
error_code { database_error: CONCURRENT_MODIFICATION } message: "Multiple
requests were attempting to modify the same resource at once. Please retry
the request."}request_id: "xU1ZDYrvYnsmJ-oTtjXOWQ"2021-12-29 05:39:31,536
WARN GoogleAdsProvider: Failed to perform action#2:
com.google.ads.googleads.v7.errors.GoogleAdsException: errors { error_code
{ database_error: CONCURRENT_MODIFICATION } message: "Multiple requests
were attempting to modify the same resource at once. Please retry the
request."}request_id: "8gHaQVPfYTFTi3lCV9HeIA"*
On Tuesday, December 28, 2021 at 10:53:49 PM UTC+3 adsapi wrote:
> Hello there,
>
> Thanks for replying back to us. We recommend waiting for about 30 seconds
> (not 30 mins) as suggested by another colleague of mine, then try another
> request via API to avoid the CONCURRENT_MODIFICATION
>
>
> error. This error occurs due to multiple requests attempting to modify the
> same resource at the same time.
> Please retry the request. If the issue persists, could you please provide
> us with the complete request
>
>
> and response
>
>
> logs with request-id
>
>
> via the Reply privately to the author option while sharing the details.
>
> Best regards,
> [image: Google Logo]
> Jakeia Sabrina
> Google Ads API Team
>
> ref:_00D1U1174p._5004Q2TkoQ6:ref
>
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to adwords-api+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/adwords-api/7249317a-5ed6-44d4-a940-76919e2671d9n%40googlegroups.com.
Re: CONCURRENT_MODIFICATION while populate new users to audience group
Hello there, Thanks for replying back to us. We recommend waiting for about 30 seconds (not 30 mins) as suggested by another colleague of mine, then try another request via API to avoid the CONCURRENT_MODIFICATION error. This error occurs due to multiple requests attempting to modify the same resource at the same time. Please retry the request. If the issue persists, could you please provide us with the complete request and response logs with request-id via the Reply privately to the author option while sharing the details. Best regards, Jakeia Sabrina Google Ads API Team ref:_00D1U1174p._5004Q2TkoQ6:ref -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/meeFu0R4UCLB00_JQFVDlfSlKDfjuBL3t6Pw%40sfdc.net.
Re: CONCURRENT_MODIFICATION while populate new users to audience group
Hi Thanks for your relay Yes! I find this answer already in this group Now i have to wait about from 15 to 30 minutes to next success api call? Is it normal ? On Monday, December 27, 2021 at 2:27:51 PM UTC+3 adsapi wrote: > Hi, > > Thanks for reaching out to us. > > In general, the CONCURRENT_MODIFICATION error occurs due to multiple > processes that are trying to update the same entity at the same time. This > also includes updating the entity in Google Ads UI while updating via the > API as well. We recommend waiting for about 30 seconds, then retry the > request. > > That being said, you’ll need to ensure that only one operation is > performed against an entity at each moment and slow down on the > consolidated repeated mutates against the same entity into a single request > to avoid this issue altogether. Please give this a try and let me know if > you have any additional concerns. > > Regards, > [image: Google Logo] > Yasar > Google Ads API Team > > > ref:_00D1U1174p._5004Q2TkoQ6:ref > -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/4aab57a3-08cc-4896-8b88-082fbcc632d1n%40googlegroups.com.
RE: CONCURRENT_MODIFICATION while populate new users to audience group
Hi, Thanks for reaching out to us. In general, the CONCURRENT_MODIFICATION error occurs due to multiple processes that are trying to update the same entity at the same time. This also includes updating the entity in Google Ads UI while updating via the API as well. We recommend waiting for about 30 seconds, then retry the request. That being said, you’ll need to ensure that only one operation is performed against an entity at each moment and slow down on the consolidated repeated mutates against the same entity into a single request to avoid this issue altogether. Please give this a try and let me know if you have any additional concerns. Regards, Yasar Google Ads API Team ref:_00D1U1174p._5004Q2TkoQ6:ref -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/CAKLm0R4RUI100S4CN5un9R2CYI2IltE5E-g%40sfdc.net.
