[appengine-java] Re: When is the ConcurrentModificationException thrown in GAE?

[Broc Seib]

Suppose you have two threads calling your code snippet below, both
trying to update the same entity. The first thread to call commit() in
your transaction wins. The second thread will throw a
ConcurrentModificationException upon calling commit().

I created a live demonstration to understand this behavior here:
http://gaetestjig.appspot.com/  (the "Unique Constraint" tab)
And I blogged about it here: 
http://blog.broc.seib.net/2011/06/unique-constraint-in-appengine.html

Your application should take the necessary action if another thread
has already written data to the spot you wanted. In my case, I did not
want to occupy the same spot, so I took that exception as a desired
failure to simulate a "unique constraint". Another application may
want to merge data or something crazy like that.

Broc



On Jan 3, 11:20 pm, Harshad  wrote:
> Hi,
>
> I am reading the official GAE documentation on transactions and I
> can't understand when a ConcurrentModificationException is thrown.
>
> Look at one of the examples which I am copy-pasting here:
>
> int retries = 3;
> while (true) {
>     Transaction txn = datastore.beginTransaction();
>     try {
>         Key boardKey = KeyFactory.createKey("MessageBoard",
> boardName);
>         Entity messageBoard = datastore.get(boardKey);
>
>         long count = (Long) messageBoard.getProperty("count");
>         ++count;
>         messageBoard.setProperty("count", count);
>         datastore.put(messageBoard);
>
>         txn.commit();
>         break;
>     } catch (ConcurrentModificationException e) {
>         if (retries == 0) {
>             throw e;
>         }
>         // Allow retry to occur
>         --retries;
>     } finally {
>         if (txn.isActive()) {
>             txn.rollback();
>         }
>     }
>
> }
>
> Now, all the writes to the datastore (in this example) are wrapped
> under a transaction. So why would a ConcurrentModificationException be
> thrown?
>
> Does it happen when some other code which is not wrapped in a
> transaction updates the same entity that is being modified by the
> above code? If I ensure that all code that updates an Entity is always
> wrapped in a transaction, is it guaranteed that I won't get a
> ConcurrentModificationException?
>
> thanks,
> Harshad

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: ConcurrentModificationException question

[Broc Seib]

I would like to hear Ikai or other Google folk weigh in on this point,
as I find that caveat quite unsettling myself.I would like to hear why
it is possible to get those exceptions and the transaction still
succeed.

Broc


On Dec 15, 2:24 am, coltsith  wrote:
> Hi Broc, thanks for the reply.
>
> This sounds great to me. I'm concerned about one small note on this
> page:http://code.google.com/appengine/docs/java/datastore/transactions.html
>
> "If your app receives an exception when submitting a transaction, it
> does not always mean that the transaction failed. You can receive
> DatastoreTimeoutException, ConcurrentModificationException, or
> DatastoreFailureException exceptions in cases where transactions have
> been committed and eventually will be applied successfully. Whenever
> possible, make your datastore transactions idempotent so that if you
> repeat a transaction, the end result will be the same."
>
> Doesn't that mean I can still get a ConcurrentModificationException
> even if the write was/will be successful? And if so then that example
> I outlined could still be possible?
>
> Thanks
>
> On Dec 14, 4:43 pm, Broc Seib  wrote:
>
>
>
>
>
>
>
> > I think you are mistaken about ConcurrentModificationException meaning
> > it will eventually commit.
>
> > If you get a ConcurrentModificationException, then that means the
> > entity *failed* to write (because another write has modified the
> > update timestamp on that entity group). You must catch that exception
> > and try to write again (unless you don't want to overwrite).
>
> > You can see this behavior first hand here:http://gaetestjig.appspot.com/
> > and click on the "Unique Constraint" tab. Click the "Advance Alice"
> > button four times, and click the "Advance Bobby" button four times (in
> > any order). Now click on either button a fifth time and it will write
> > the entity to the data store, and clicking the other button will
> > experience a ConcurrentModificationException.
>
> > Broc
>
> > On Dec 13, 5:48 pm, coltsith  wrote:
>
> > > I recently got a ConcurrentModificationException, and the
> > > documentation states that this will be "committed and eventually will
> > > be applied successfully." However I got to thinking of a possible
> > > outcome:
>
> > > Request A modifies entity
> > > Request B modifies entity and receives ConcurrentModificationException
> > > D
> > > Request E modifies entity
> > > ConcurrentModificationException D commits and is applied, which
> > > overwrites Request E's work.
>
> > > Can this happen? Or does Request E get its own
> > > ConcurrentModificationException since (D) hasn't been committed yet?
>
> > > Many thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: ConcurrentModificationException question

[Broc Seib]

I think you are mistaken about ConcurrentModificationException meaning
it will eventually commit.

If you get a ConcurrentModificationException, then that means the
entity *failed* to write (because another write has modified the
update timestamp on that entity group). You must catch that exception
and try to write again (unless you don't want to overwrite).

You can see this behavior first hand here: http://gaetestjig.appspot.com/
and click on the "Unique Constraint" tab. Click the "Advance Alice"
button four times, and click the "Advance Bobby" button four times (in
any order). Now click on either button a fifth time and it will write
the entity to the data store, and clicking the other button will
experience a ConcurrentModificationException.

Broc


On Dec 13, 5:48 pm, coltsith  wrote:
> I recently got a ConcurrentModificationException, and the
> documentation states that this will be "committed and eventually will
> be applied successfully." However I got to thinking of a possible
> outcome:
>
> Request A modifies entity
> Request B modifies entity and receives ConcurrentModificationException
> D
> Request E modifies entity
> ConcurrentModificationException D commits and is applied, which
> overwrites Request E's work.
>
> Can this happen? Or does Request E get its own
> ConcurrentModificationException since (D) hasn't been committed yet?
>
> Many thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Documentation Issue: TaskQueue Testing

[Broc Seib]

Erick,

A while back the API changed to use with() functions, and I'm sure
they did not catch all the docs that needed updated. You should file
an issue here for fixing the documentation and someone will eventually
get to it:  http://code.google.com/p/googleappengine/issues/list

Broc


On Nov 19, 1:49 pm, Erick Fleming  wrote:
> http://code.google.com/appengine/docs/java/tools/localunittesting.htm...
>
> TaskOptions.Builder.taskName("task29")
>
> should be
>
> TaskOptions.Builder.withTaskName("task29")
>
> PS: I didn't see any place in issues to report documentation issues

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: QR Code Generator

[Broc Seib]

You could utilize the Google Charts API: i.e. 
http://code.google.com/apis/chart/infographics/
Broc

On Nov 4, 10:17 am, David Vallejo  wrote:
> I´m looking for a QR Code Generator Library that could be integrated to
> Java App Engine, could you recomend any one?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [objectify-appengine] Re: [appengine-java] HRD - Eventual consistency

[Broc Seib]

In my unit tests for my entities using HRD, I simply switch that percentage
completely on/off around those few places in the unit test code where I
need my entity to become "fully committed" in order for the rest of the
unit test to work. Yes, it is kind of contrived, but then again so is a
unit test. I must consciously decide to flip it on/off around that bit of
code. My unit test code looks something like this:


//read this:
http://code.google.com/appengine/docs/java/tools/localunittesting.html
public class SampleUnitTest {

private static boolean SHOULD_APPLY = false; // set this flag appropriately
in the units test below.
 private static final class CustomHighRepJobPolicy implements
HighRepJobPolicy {
@Override
public boolean shouldApplyNewJob(Key arg0) {
return SHOULD_APPLY;
}
@Override
public boolean shouldRollForwardExistingJob(Key arg0) {
return SHOULD_APPLY;
}
}

private final LocalServiceTestHelper helper = new
LocalServiceTestHelper(new
LocalDatastoreServiceTestConfig().setAlternateHighRepJobPolicyClass(CustomHighRepJobPolicy.class));

@Before
public void setUp() {
helper.setUp();
}

@After
public void tearDown() {
helper.tearDown();
}


@Test
public void test1() throws Exception {
String orgName = "The Organization, Inc.";
SHOULD_APPLY = true; // temporarily allow things to be written to
datastore for real
final OOrganization org = ODaoOrganization.create(orgName);
SHOULD_APPLY = false; // return HRD writes back to full "failure" for
remaining unit test

// now do my unit test that needed my org entity to be properly
persisted.
}
}


Broc



On Mon, Oct 31, 2011 at 3:36 PM, Jeff Schnitzer  wrote:

> There's this:
>
> https://groups.google.com/d/msg/google-appengine/NdUAY0crVjg/3fJX3Gn3cOYJ
>
> Some thoughts:
>
>  * I don't think there's really anything to save you from eventuality
> except careful consideration during the design of your data model.  Pick
> your entity groups carefully and use XG transactions sparingly to move data
> "across" when you need atomicity.
>
>  * A lot of the time eventuality is not a big deal. On the other hand, if
> you're building an accounting app, it probably is a big deal.
>
>  * There's a difference between what your users experience and what your
> tests experience.  Say you have a test for "add this product to the catalog
> and then check to see if it's in the catalog".  In the real world it
> doesn't matter if your users see the new product right away.  But your
> tests demand it immediately.  The problem is with the test.  I've "fixed"
> the tests by doing this in my fixture setup:
>
> newLocalDatastoreServiceTestConfig().setDefaultHighRepJobPolicyUnappliedJobPercentage(0.1f)
>
> Unfortunately a good argument can be made that this diminishes the value
> of the tests. Really some tests should be run with eventuality and others
> without.  Anyone know if setDefaultHighRepJobPolicyUnappliedJobPercentage()
> can be called repeatedly after setup to switch back and forth?
> Jeff
>
> On Mon, Oct 31, 2011 at 6:31 AM, Aswath Satrasala <
> aswath.satras...@gmail.com> wrote:
>
>> Hi,
>> I am trying HRD on local dev server.  I have bunch of webdriver tests for
>> different parts of my application that create entities and then doing a
>> list of all the entities.
>> My tests are failing in several places.  It is due to the eventual
>> consistency of the HRD.  The tests run fine on M/S and on both
>> production/local environment.
>> I have 20-30 KINDS, and entities are created for 1-or more KINDs
>> depending on the business logic, all in a transaction.   Immediately,
>> various reports are done.  They fail.
>>
>> My question: What pattern needs to be followed?  My application has grown
>> very large during last 1.5 years or so.  Doing re-factoring at every place
>> in the application for eventual consistency will be a very large effort for
>> a small team. I don't even know what extra work needs to be put to get this
>> fixed.
>>
>> Am I better of sticking with M/S.  So far, I have managed and designed
>> entity relationships to work within one entity group.
>>
>> Are there any patterns coming out from the Objectify team to handle
>> Eventual consistency issue of the HRD?
>>
>> -Aswath
>> www.AccountingGuru.in
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine for Java" group.
>> To post to this group, send email to
>> google-appengine-java@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine-java+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine-java?hl=en.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@goog

[appengine-java] Re: Merging older queries with ancestor queries.

[Broc Seib]

You will not have the delay if you query for your data with an
ancestor query. This page explains: 
http://code.google.com/appengine/docs/java/datastore/hr/overview.html

"Ancestor queries work because entity groups are a unit of
consistency: all operations are applied to the entire group. Ancestor
queries won't return data until the entire entity group is up to date.
Thus, the data returned from ancestor queries on entity groups is
strongly consistent.

If your application relies on strongly consistent results for certain
queries, you may have to change the way your application stores
entities."

And at the bottom of that page it adds...

"Remember, if you do a get(), put(), or a transaction, you will always
see the most recently written data."

Note that a query in a transaction must be an ancestor query.

Broc



On Oct 27, 12:34 pm, markabrucey  wrote:
> Hi,
>
> I have been doing a lot of reading on Ancestor queries because we are
> currently trying to transfer from Master / Slave to the High Replication
> datastore.
>
> We have noticed while testing on the HRDS, that sometimes when adding an
> entity and redirecting the view to show that entity (or maybe a list
> containing others as well), it can take a while for it to update because it
> is eventually consistent. (which is sometimes inconvenient)
>
> I understand that you can not put an entity into an entity group once
> already created, the process of creating that entity so it is in a group
> has to be done the first time round.
>
> I was just wondering that if we kept our older entities and started doing
> ancestor queries to insert new ones, can we merge the results to produce
> and output? I think if we can, it will have to be done in 2 queries, and if
> it can, will we no longer see a delay in data being updated?
>
> Thanks,
> Mark

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Object does not persist [JDO, HR]

[Broc Seib]

There's this thread: 
http://groups.google.com/group/google-appengine-java/browse_thread/thread/8c8d3191d287553f

Might I add, I wrestled with using JDO for several months before I
realized it was just not a good fit for AppEngine's DataStore. I
switched to Objectify and life has been much better since. Now I don't
have to try to envision things from Data Nucleus point of view, but
rather just think like datastore to begin with.

Broc

On Oct 24, 5:33 am, Paul  wrote:
> Hmm... no ideas anyone? I have tried to google for hours and no
> results. I really don't understand why that problem exists.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: multiple GAE apps under one Google Apps domain

[Broc Seib]

Hi Eliot,

Short answer: When asked for what domain, I think you need to enter
"domain.com", not "dev.domain.com". The next page is where you get to
specify the hostname.

Broc


For the longer answer, I thought I'd share some of the other subtle
things I learned by trial and error when going down these roads,
including the use of domain "aliases". I am regurgitating the
following (edited) notes from my personal tech journal, if they might
be useful too.

First, let's assume you have the domain 'domain.com' as your main
domain that you have signed up with Google Apps.

If you are using another domain name other than your primary domain
name, the you must setup domain aliases from your Google Apps control
panel for "Domain Settings", i.e.,  
https://www.google.com/a/cpanel/domain.com/DomainSettingsDomains

When adding a domain to your Google Apps account, you must choose
between adding it as a "fully independent" domain, or as a "domain
alias". I found by trial and error that you must use a domain alias in
order for it to be used as a domain name for an app engine
application. Setting this up is just a matter of setting a CNAME
record in your DNS that points back to google per their instructions.
You must wait for the "verified" status to come through such that
google recognizes your DNS change.**

Once you have a verified domain alias (or you are just using your
primary domain) you may visit the control panel of your app engine
app, i.e.,
  https://appengine.google.com/dashboard?&app_id=X

1) Click the "Application Settings" link on the left panel.
2) Click the "Add Domain..." button (toward the bottom of the page).
3) In the text box labelled "Domain Name", you must supply your Google
Apps domain (domain.com) -- NOT the desired domain you want
(prod.domain.com).
4) Click the "Add Domain..." button. (takes you to a page in your
Google Apps control panel)
5) Click "Add new URL" link.
6) Supply the desired hostname after the "http://";, and choose a
domain from a fixed set of choices, which include your primary domain,
and your domain aliases, but not any "full citizen" domains you have
added to your Google Apps.
7) Click "Add".


** When I setup my DNS CNAME, it was stuck in "verifying..." mode for
several days, past the 48 hours advertised for verification. I had to
email Google Apps support and ask them to fix it. They did, quickly,
citing something about their "propagation process" gone awry. I
thought nothing of it, except that all of this happened again a second
time when I added a second alias. I hope it is no longer an issue.
Just a heads up.



On Oct 18, 7:30 am, Eliot Stock <1...@eliotstock.com> wrote:
> Hi there,
>
> I'd like to create subdomains for various environments under my domain.com
> as follows:
>
> dev.domain.com would map to the app ID dev-domain-com
> test.domain.com -> app ID: test-domain-com
> prod.domain.com -> app ID: prod-domain-comwww.domain.com-> app ID: 
> prod-domain-com
>
> Is this possible? When adding the URL dev.domain.com to an existing GAE app,
> the redirect into the Google Apps control panel gives me the following
> error:
>
> "Sorry, you've reached a login page for a domain that isn't using Google
> Apps"
>
> This is not true: domain.com is definitely a domain using Google Apps. It's
> being confused by dev.domain.com though.
>
> Any ideas?
>
> Cheers,
>
> Eliot.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Crons Optimization

[Broc Seib]

It may require a bit of typing (one time), but may I suggest just
explicitly scheduling each of the 14 cron jobs, staggered
appropriately. Use the following variety of syntax to specify the
schedule:

  every N (hours|mins|minutes) ["from" (time) "to" (time)]
  (from 
http://code.google.com/appengine/docs/java/config/cron.html#The_Schedule_Format)

E.g.,



  
/foo/task1
every 30 minutes from 00:00 to 23:29
  
  
/foo/task2
every 30 minutes from 00:02 to 23:31
  
  
/foo/task3
every 30 minutes from 00:04 to 23:33
  
  
  
/foo/task14
every 30 minutes from 00:28 to 23:57
  






On Oct 12, 5:27 am, Hakim  wrote:
> Hi everybody,
>
> I've an application that runs 14 crons that have the same schedule
> configuration: every 30 mins synchronized
>
> I'm looking for a technique to desynchronise the execution of these crons, I
> don't want them to execute at the same time in order to reduce the number of
> instances used to serve them each 30 minutes
>
> Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: [objectify-appengine] white labeling

[Broc Seib]

Here are some thoughts:

With respect to segregating your data for each brand, you can read a bit
about DataStore and Multitenancy here:
http://code.google.com/appengine/docs/java/multitenancy/overview.html

I also know that when your are a paying Google Apps customer, you can add
"domain aliases" in your Google Apps from you Domain Names settings page
here:
https://www.google.com/a/cpanel/yourprimarydomainname.com/DomainSettingsDomains
 then from AppEngine's control panel, under "Application Settings", you can
click the "Add Domain.." button, then enter "yourprimarydomainname.com",
then you'll be able to supply your "otherbrand.com" to serve your app from
AppEngine. Now otherbrand.com will load your website on appengine too. Your
customers would also have to perform the Google domain verification steps
per your request.

These things will get you close, but then you have to actually make the site
appear different when someone uses that domain name to visit your
site. There is no quick-and-dirty "mod-rewrite" on AppEngine, but the
hostname "otherbrand.com" will be in the HTTP request headers, and you
should be able to influence what happens to incoming requests, i.e. steer
different images and maybe some style sheets. For example, you might have a
servlet that acts as a (caching?) proxy to the customer's external branding
resources (you will specify to them exactly what they must supply). E.g. a
URL on your site for a header image might look like "/some/path/
otherbrand.com/header.png" where you would inject the "otherbrand.com" into
that URL when you put that image into the DOM.

Broc



On Fri, Aug 19, 2011 at 1:31 AM, Aswath Satrasala <
aswath.satras...@gmail.com> wrote:

> Hi,
> I have appengine application which is shown interest by few resellers. They
> are taking about white labeling.
> Can anyone provide details on how to do the white labeling for appengine
> application.
> What options are available?
> One possibility is that each reseller be given a different app-id with
> permissions.  In this case, I have to monitor/upgrade all the app-id's.
>
> Any pointers on this subject?
>
> Regards
> -Aswath
> www.AccountingGuru.in
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Slow JAXB context initialization

[Broc Seib]

Aurel,

JAXB on AppEngine has been slow for me as well, and causes me pain. In my 
case the Google Checkout SDK uses JAXB, and it causes problems when using 
Google Checkout with App Engine (yikes -- two Google products not getting 
along!). I described my problem 
here
.

I reported the issue, and you can vote for it here:
  http://code.google.com/p/googleappengine/issues/detail?id=4000

-broc

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: How can I front GAE's Jetty with Apache/mod_jk?

[Broc Seib]

Hi all,

I wrote up a solution which uses mod_proxy rather than mod_jk. It is 
here: http://blog.broc.seib.net/2011/03/fronting-eclipse-jetty-server-with.html

Broc

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.